Neil Ang (programming)

How to create a spell checking web spider

Sat, 24 Oct 2009 10:18:00 GMT

Recently I found out my employer was paying a third-party company to regularly check the spelling on their website and send them a monthly report. So I thought I would write a simple spider that could to do the same thing and save them money.

The setup

My solution uses ruby (plus some gems) and the awesome Aspell (an open source spell checker).

First of all you will need to download and install Aspell, and an Aspell dictionary for the language you want to use. In this example I will be using the English dictionary.

To install Aspell on OS X, download and unpack the latest version, and then using terminal:

cd path/to/aspell ./configure make sudo make install

To install a dictionary, repeat the steps above with a downloaded dictionary.

You can verify that Aspell installed correctly and that the dictionaries are loaded by typing these commands in terminal:

aspell -v aspell dicts

Next you will need to install raspell, which is a gem used to interact with Aspell. You will also need another gem called spider, which will take care of the web crawling work for us. And finally, hpricot or nokogiri to handle the HTML.

sudo gem install raspell sudo gem install spider sudo gem install hpricot

The script

With that done, you can use the spider gem to crawl all the html pages on a website, then use hpricot to extract the relevant words from the page and check each one with raspell. E.g.

#!/usr/bin/env ruby require 'rubygems' require 'spider' require 'raspell' require 'hpricot' domain = 'http://www.example.com/' speller = Aspell.new('en_GB') Spider.start_at(domain) do |s| s.add_url_check do |a_url| a_url.match("^#{domain}") end s.on :success do |a_url, resp, prior_url| unless resp['content-type'].match('text/html') puts "Skipping #{a_url}" next end puts "On page #{a_url}" document = Hpricot(resp.body) document.search('head').remove document.search('script').remove document.search('link').remove document.search('meta').remove document.search('style').remove words = document.inner_text.gsub(/\s+/, ' ').strip.split(/\s/) speller.list_misspelled(words).each do |mistake| puts " * Found mistake \"#{mistake}\" perhaps you meant \"#{speller.suggest(mistake).first}\"" end end end

A few things to note about this script:

I have set the spider to use the "en_GB" dictionary (as this is what we use in Australia), but you can set it to any dictionary you installed (e.g. "en_US").
If you wanted to use nokogiri instead of hpricot, simply replace the Hpricot declaration with document = Nokogiri(resp.body) and require it at the start of the script.
Hpricot is used to strip out typically non-visible sections of the page, so that we only spellcheck the displayed words.
The spider won't search outside of the set domain.
The spider is also capable of performing a link check as it crawls.

As you can see this is only a very basic implementation, but demonstrates how easily you can create your own spelling spider!

Please post a comment

Validate an ISSN using Perl or JavaScript

Fri, 12 Jun 2009 07:57:00 GMT

Last week I wrote about validating an ISBN, so this week I thought I would post the code to validate an ISSN.

There are only a few minor differences between validating an ISSN and ISBN.

Validate ISSN in Perl

sub valid_issn { 
   my $issn = $_[0]; 
      $issn =~ s/[^\dX]//gi; 
   return if length($issn) != 8; 
   my $sum = 0; 
   my @chars = split('', $issn); 
   $chars[7] = 10 if uc($chars[7]) eq 'X'; 
   for (my $char = 0; $char < @chars; $char++) { 
     $sum += (8-$char) * $chars[$char]; 
   } 
   return (($sum % 11) == 0); 
 }

Validate ISSN in JavaScript

function isValidISSN (issn) { 
   issn = issn.replace(/[^\dX]/gi, ''); 
   if(issn.length != 8){ 
     return false; 
   } 
   var chars = issn.split(''); 
   if(chars[7].toUpperCase() == 'X'){ 
     chars[7] = 10; 
   } 
   var sum = 0; 
   for (var i = 0; i < chars.length; i++) { 
     sum += ((8-i) * parseInt(chars[i])); 
   }; 
   return ((sum % 11) == 0); 
 }

Please post a comment

How to check if an ISBN is valid in Perl or JavaScript

Thu, 04 Jun 2009 22:22:00 GMT

The cool thing about an ISBN is that its last digit is a "check digit", which validates the rest of the number. How does it work? "It is calculated on a modulus 11 with weights 10-2, using X in lieu of 10 where ten would occur as a check digit." More info.

Validate an ISBN in Perl

Here is a simple subroutine I wrote to check if a number entered is valid:

sub valid_isbn { my $isbn = shift; $isbn =~ s/[^\dX]//gi; return if length($isbn) != 10; my $sum = 0; my @chars = split('', $isbn); $chars[9] = 10 if uc($chars[9]) eq 'X'; for (my $char = 0; $char < @chars; $char++) { $sum += (10-$char) * $chars[$char]; } return (($sum % 11) == 0); }

Code would not be complete without tests!

use Test::More tests => 12; is(valid_isbn("ISBN 0843610727"), 1); is(valid_isbn("99921-58-10-7"), 1); is(valid_isbn("9971 5 0210 0"), 1); is(valid_isbn("0-8044-2957-X"), 1); is(valid_isbn("0943396042"), 1); is(valid_isbn("0-9752298-0-X"), 1); isnt(valid_isbn("ISBN 0843610723"), 1); isnt(valid_isbn("91921-58-10-7"), 1); isnt(valid_isbn("9971 5 0214 0"), 1); isnt(valid_isbn("0-8044-2957-2"), 1); isnt(valid_isbn("0943394042"), 1); isnt(valid_isbn("0-9757297-0-X"), 1);

Validate an ISBN in JavaScript

And here it is written as a JavaScript function:

<script type="text/javascript" charset="utf-8"> function isValidISBN (isbn) { isbn = isbn.replace(/[^\dX]/gi, ''); if(isbn.length != 10){ return false; } var chars = isbn.split(''); if(chars[9].toUpperCase() == 'X'){ chars[9] = 10; } var sum = 0; for (var i = 0; i < chars.length; i++) { sum += ((10-i) * parseInt(chars[i])); }; return ((sum % 11) == 0); } </script>

Please post a comment

Generating a calendar using XSLT

Sun, 16 Nov 2008 07:43:00 GMT

Recently I was tasked to build an interactive calendar in XSLT. I wanted the calendar to always start on a Sunday as well as have clean markup. I couldn't find any good examples on the web while I was building it, so I decided to post the base of my solution.

Insert the following variables and templates into your stylesheet:

<xsl:variable name="DisplayDate" select="date:date()"/> 
 <xsl:variable name="Year" select="date:year($DisplayDate)"/> 
 <xsl:variable name="Month" select="date:month-in-year($DisplayDate)"/> 
 <xsl:variable name="MonthName" select="date:month-name($DisplayDate)" /> 
 <xsl:variable name="NumberOfDaysInMonth"> 
   <xsl:call-template name="DaysInMonth"> 
     <xsl:with-param name="month" select="$Month" /> 
     <xsl:with-param name="year" select="$Year" /> 
   </xsl:call-template> 
 </xsl:variable> 
 <xsl:variable name="FirstDayInWeekForMonth">

  <xsl:choose>

    <xsl:when test="$Month &lt; 10">

      <xsl:value-of select="date:day-in-week(date:date(concat($Year,'-0', $Month, '-01')))" />

    </xsl:when>

    <xsl:otherwise>

      <xsl:value-of select="date:day-in-week(date:date(concat($Year,'-', $Month, '-01')))" />

    </xsl:otherwise>

  </xsl:choose>

</xsl:variable> 
 <xsl:variable name="WeeksInMonth"><xsl:value-of select="($NumberOfDaysInMonth + $FirstDayInWeekForMonth - 1) div 7" /></xsl:variable> 
 
 <xsl:template name="DaysInMonth"> 
   <xsl:param name="month"><xsl:value-of select="$Month" /></xsl:param> 
   <xsl:param name="year"><xsl:value-of select="$Year" /></xsl:param> 
   <xsl:choose> 
     <xsl:when test="$month = 1 or $month = 3 or $month = 5 or $month = 7 or $month = 8 or $month = 10 or $month = 12">31</xsl:when> 
     <xsl:when test="$month=2"> 
       <xsl:choose> 
         <xsl:when test="$year mod 4 = 0">29</xsl:when> 
         <xsl:otherwise>28</xsl:otherwise> 
       </xsl:choose> 
     </xsl:when> 
     <xsl:otherwise>30</xsl:otherwise> 
   </xsl:choose> 
 </xsl:template> 
 
 <xsl:template name="Calendar"> 
   <table summary="Monthly calendar"> 
     <caption> 
       <xsl:value-of select="$MonthName" /> 
       <xsl:text> </xsl:text> 
       <xsl:value-of select="$Year" /> 
     </caption> 
     <tr> 
       <th abbr="Sunday">Sun</th> 
       <th abbr="Monday">Mon</th> 
       <th abbr="Tuesday">Tue</th> 
       <th abbr="Wednesday">Wed</th> 
       <th abbr="Thursday">Thu</th> 
       <th abbr="Friday">Fri</th> 
       <th abbr="Saturday">Sat</th> 
     </tr> 
     <xsl:call-template name="CalendarWeek"/> 
   </table> 
 </xsl:template> 
 
 <xsl:template name="CalendarWeek"> 
   <xsl:param name="week">1</xsl:param> 
   <xsl:param name="day">1</xsl:param> 
   <tr> 
     <xsl:call-template name="CalendarDay"> 
       <xsl:with-param name="day" select="$day" /> 
     </xsl:call-template> 
   </tr> 
   <xsl:if test="$WeeksInMonth &gt; $week"> 
     <xsl:call-template name="CalendarWeek"> 
       <xsl:with-param name="week" select="$week + 1" /> 
       <xsl:with-param name="day" select="$week * 7 - ($FirstDayInWeekForMonth - 2)" /> 
     </xsl:call-template> 
   </xsl:if> 
 </xsl:template> 
 
 <xsl:template name="CalendarDay"> 
   <xsl:param name="count">1</xsl:param> 
   <xsl:param name="day" /> 
   <xsl:choose> 
     <xsl:when test="($day = 1 and $count != $FirstDayInWeekForMonth) or $day &gt; $NumberOfDaysInMonth"> 
       <td class="empty"><xsl:text disable-output-escaping="yes">&amp;nbsp;</xsl:text></td> 
       <xsl:if test="$count &lt; 7"> 
         <xsl:call-template name="CalendarDay"> 
           <xsl:with-param name="count" select="$count + 1" /> 
           <xsl:with-param name="day" select="$day" /> 
         </xsl:call-template> 
       </xsl:if> 
     </xsl:when> 
     <xsl:otherwise> 
       <td> 
         <xsl:value-of select="$day" /> 
       </td> 
       <xsl:if test="$count &lt; 7"> 
         <xsl:call-template name="CalendarDay"> 
           <xsl:with-param name="count" select="$count + 1" /> 
           <xsl:with-param name="day" select="$day + 1" /> 
         </xsl:call-template> 
       </xsl:if> 
     </xsl:otherwise> 
   </xsl:choose> 
 </xsl:template>

For this code to work, you need to be using exslt.

<xsl:stylesheet version="1.0" 
   xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
   xmlns:exslt="http://exslt.org/common" 
   xmlns:date="http://exslt.org/dates-and-times">

Once you have added the templates you can call the calendar like this:

<xsl:call-template name="Calendar" />

Finally add some CSS to the page so your calendar looks more presentable:

<style type="text/css" media="screen"> 
   table{ 
     border:1px solid #CCC; 
     border-collapse: collapse; 
   } 
   caption{ 
     font-weight:bold; 
     font-size:120%; 
     margin-bottom:8px; 
   } 
    
   td,th{ 
     text-align:center; 
     vertical-align:middle; 
     width:36px; 
     height:34px; 
     border:1px solid #CCC; 
     padding:0; 
   } 
    
   th{ 
     height:inherit; 
   } 
    
   .empty{ 
     background-color:#EEE; 
   } 
 </style>

This is only the base of the calendar I built, and you can expand this to make it browse-able. But I will leave that for another post.

Please post a comment

Converting seconds into a readable format in Perl

Wed, 18 Jun 2008 09:49:00 GMT

Here is a short and easy Perl function to convert the difference between two time stamps, into something that is more human readable. There are Perl modules that can do this as well, however I thought I would share what I wrote incase someone finds it useful.

#!/usr/bin/env perl -w 
 
 sub convert_time { 
   my $time = shift; 
   my $days = int($time / 86400); 
    $time -= ($days * 86400); 
   my $hours = int($time / 3600); 
    $time -= ($hours * 3600); 
   my $minutes = int($time / 60); 
   my $seconds = $time % 60; 
 
   $days = $days < 1 ? '' : $days .'d '; 
   $hours = $hours < 1 ? '' : $hours .'h '; 
   $minutes = $minutes < 1 ? '' : $minutes . 'm '; 
   $time = $days . $hours . $minutes . $seconds . 's'; 
   return $time; 
 }

When using this function, these are the outputs you should see:

print convert_time(1); # result: 1s 
 print convert_time(61); # result: 1m 1s 
 print convert_time(3661); # result: 1h 1m 1s 
 print convert_time(90061); # result: 1d 1h 1m 1s

This code can easily be modified to output other formats as well. Let me know if you come up with something different.

Please post a comment

Title case names in perl

Tue, 17 Jun 2008 09:58:00 GMT

While I was at work today I came across a script I wrote a couple of years ago to title case names in perl. This script will transform names to a nice, readable title case equivalent. E.g. "JOHN SMITH" to "John Smith", "JOHN O'BRIAN" to "John O'Brian" and "JOHN MCCOOL" to "John McCool" etc.

#!/usr/bin/env perl 
 
 use strict; 
 
 sub title_case_name { 
   my $person = shift; 
   my ($word, $subWord); 
    
   $person =~ s/\s+/ /g; 
   $person =~ s/^\s+//; 
   $person =~ s/\s+$//; 
 
   my @namesArray = split(/ /, $person); 
   $person = ''; 
   foreach $word (@namesArray){ 
     $word = title_case($word); 
     if($word =~ /-/){ 
       my @subWordsArray = split(/-/, $word); 
       $word = ''; 
       foreach $subWord (@subWordsArray){ 
         $subWord = title_case($subWord); 
         $word .= $subWord.'-'; 
       } 
       chop($word); 
     } 
     $person .= $word.' '; 
   } 
   chop($person); 
 
   return $person; 
 } 
 
 sub title_case { 
   my $name = shift; 
   my $temp_str1 = ''; 
   my $temp_str2 = ''; 
   $name = lc($name); 
   $name = ucfirst($name); 
   $temp_str1 = substr($name, 0, 2); 
   $temp_str2 = substr($name, 2); 
   if($temp_str1 eq 'Mc' || $temp_str1 eq "O'") { 
     $temp_str2 = ucfirst($temp_str2); 
     $name = $temp_str1 . $temp_str2; 
   } 
 
   return $name; 
 } 
 
 print title_case_name('JOHN SMITH') . "\n"; 
 print title_case_name('JOHN O\'BRIAN') . "\n"; 
 print title_case_name('JOHN MCCOOL') . "\n";

Download this script.

Please post a comment

Dynamically set network location in Mac OS X

Wed, 24 Jan 2007 10:10:00 GMT

I love macs, but sometimes things could have been implemented better. In particular, I hate having to set the network location on my laptop when I go between work and home every day. I have put together a small shell script which can do this. I run this script on startup and it just checks what day and time it is so it can switch between two profiles (work or home).

#!/bin/bash day=`date +"%a"` hour=`date +"%H"` if [ $day = "Sat" -o $day = "Sun" ]; then `scselect Home` else if [ $hour -gt 8 -a $hour -lt 16 ]; then `scselect Work` else `scselect Home` fi fi exit 0

Please post a comment

PHP: Explore your server through a PHP file browser

Thu, 11 Jan 2007 10:09:00 GMT

In the past I've had to delete files from my server where the file permissions were set to apache and I could not override them because the host did not alllow terminal or shell access. So I quickly put together a PHP script that allowed me to navigate through the file system and unlink (or delete) files as the apache user. I haven't had to use the script for a while but I still think its pretty nifty to use when you want to explore a servers file system.

In the code below I've stripped out the unlink code (as it can be dangerous to have a page like that on your server) and just made it into a file explorer. Copy the code into a page and give it a go. If you're like me and have set your localhost as your homepage you can use it to quickly access files you have been working on. If anyone is interested in the version that allows you to unlink files, just send me an email and I will post it on the site.

<html> <head> <title>PHP File Browser</title> <style type="text/css"> .error{ color:red; font-weight:bold; } </style> </head> <body> <?php // Explore the files via a web interface. $script = basename(__FILE__); // the name of this script $path = !empty($_REQUEST['path']) ? $_REQUEST['path'] : dirname(__FILE__); // the path the script should access echo "<h1>Directory Browser</h1>"; echo "<p>Browsing Location: {$path}</p>"; $directories = array(); $files = array(); // Check we are focused on a dir if (is_dir($path)) { chdir($path); // Focus on the dir if ($handle = opendir('.')) { while (($item = readdir($handle)) !== false) { // Loop through current directory and divide files and directorys if(is_dir($item)){ array_push($directories, realpath($item)); } else { array_push($files, ($item)); } } closedir($handle); // Close the directory handle } else { echo "<p class=\"error\">Directory handle could not be obtained.</p>"; } } else { echo "<p class=\"error\">Path is not a directory</p>"; } // There are now two arrays that contians the contents of the path. // List the directories as browsable navigation echo "<h2>Navigation</h2>"; echo "<ul>"; foreach( $directories as $directory ){ echo ($directory != $path) ? "<li><a href=\"{$script}?path={$directory}\">{$directory}</a></li>" : ""; } echo "</ul>"; echo "<h2>Files</h2>"; echo "<ul>"; foreach( $files as $file ){ // Comment the next line out if you wish see hidden files while browsing if(preg_match("/^\./", $file) || $file == $script): continue; endif; // This line will hide all invisible files. echo '<li><a href="' . basename($file) . '" target="_blank">' . $file . '</a></li>'; } echo "</ul>"; ?> </body> </html>

Please post a comment

Sockets problem with MySQL/PHP and Mac OS X

Thu, 21 Dec 2006 10:08:00 GMT

After installing MySQL 5 on OS X 10.4, you may have noticed that the install package has placed 'mysql.sock' in '/tmp/mysql.sock' instead of '/var/mysql/mysql.sock'. This will cause the following error when you attempt to access MySQL through PHP:

Can't connect to local MySQL server through socket '/var/mysql/mysql.sock' (2)

A simple solution is to place a symbolic link between the expected folder and the install directory. This can be achieved with the following terminal commands:

sudo mkdir /var/mysql sudo ln -s /tmp/mysql.sock /var/mysql/mysql.sock

To test a database connection between PHP and MySQL, you can use the following script:

<?php $dbServer='localhost'; $dbUser='root'; $dbPass='root'; $dbName='test'; $link = mysql_connect("$dbServer", "$dbUser", "$dbPass") or die ('Could not connect.'); echo 'Connected successfully<br />'; mysql_select_db("$dbName") or die('Could not select database.'); echo 'Database selected successfully<br />'; mysql_close($link); ?>

Please post a comment

PHP: Sending HTML Emails

Tue, 19 Dec 2006 10:06:00 GMT

Just like the general nature of PHP, the PHP mailing functions are simple to use as well. The easiest way to send an email is to use the mail function built into PHP 3, PHP 4 and PHP 5. However, to send HTML based emails (or any other content-type for that matter), you will need to alter the header values of the email. The following is a short example on how to do this with PHP.

Configure PHP.ini

To send an email, you must first check your php.ini mail settings are correct (if these setting are correct, skip down below to the code). If you are using Windows, you will need access to a SMTP server (your web host should provide you with one, otherwise you can setup your own), otherwise if your using a Mac or Linux, you will need the sendmail mail transfer agent (this is usually preinstalled with the OS).

For Windows users, locate and open your php.ini file and find the following lines:

[mail function] ; For Win32 only. ;SMTP = ;smtp_port =

; For Win32 only. ;sendmail_from =

Add the correct settings and remove the line comment marks (semi-colons) so the configuration looks similar to this:

[mail function] ; For Win32 only. SMTP = localhost smtp_port = 25

; For Win32 only. sendmail_from = me@example.com

For non-Windows users locate the following lines in the php.ini:

; For Unix only. You may supply arguments as well (default: "sendmail -t -i"). ;sendmail_path =

Uncomment and add the full path to sendmail like this:

; For Unix only. You may supply arguments as well (default: "sendmail -t -i"). sendmail_path = /usr/sbin/sendmail

Save and close php.ini. If you are using PHP through a web server, it might require a restart for changes to take effect.

The code

The following PHP code when executed will create a HTML message and send it to a recipient (or multiple recipients by separating each email address in the '$to' variable with a comma):

<?php $to = 'anywhere@example.com'; $subject = 'My Test HTML'; $message =<<<EOM <html> <body><h1>My HTML Message</h1> <p>This is text</p> </body> </html> EOM; $headers = 'From: no-reply@example.com' . "\r\n" . 'content-type: text/html;'. "\r\n"; if(mail($to, $subject, $message, $headers)){ echo 'Mail sent successfully!'; } else { echo 'Mail could not be sent, please check your settings.'; } ?>

If the SMTP server you are using is on a protected network that won't accept emails from your host, you can use PHP to create the email through the SMTP host.

The following code is an example of how this could be done:

<?php $to = 'anywhere@example.com'; $subject = 'My Test HTML'; $message =<<<EOM <html> <body><h1>My HTML Message</h1> <p>This is text</p> </body> </html> EOM; // Enter the SMTP server and port here $conn = fsockopen("smtp.example.com", 25); if($conn){ fwrite($conn, "HELO\r\n"); fwrite($conn, "MAIL FROM: $from\r\n"); fwrite($conn, "RCPT TO: $to\r\n"); fwrite($conn, "DATA\r\n"); fwrite($conn, "Subject: $subject\r\n"); fwrite($conn, "content-type: text/html\r\n"); fwrite($conn, "\r\n"); fwrite($conn, "$email \r\n"); fwrite($conn, "\r\n"); fwrite($conn, ".\r\n"); fwrite($conn, "QUIT"); echo "Finished Sending Email"; } else{ echo "Couldn't connect!\n"; } ?>

Please post a comment