At this year's
No Fluff Just Stuff conference, presenter
Matthew McCullough,
the managing partner of Ambient Ideas, LLC, gave an interesting talk on
Open Source Debugging Tools, which touched on usage of freely-available tools such as
cURL. cURL is a command-line tool which
supports file transfer operations for a variety of protocols (HTTP, HTTPS, FTP, SFTP, SCP, etc.)
and can be used for a variety of useful tasks, such as retrieving the real source page of a web site:
curl -i -H Accept:text/xml http://www.phurnace.com
You can also determine your public IP (useful for those running behind a firewall/router) with the command:
curl -q http://checkip.dyndns.org
This command will return an HTML page with your public IP address embedded.
Okay, here's an example of how you can use cURL to take advantage of Google's mail feed to obtain
information about your unread e-mail, without ever using the web browser. In this example, I will show how
to take the output from the mail feed, parse it with a Groovy script, and then send a text message to your cell phone number.
First the script:
#!/bin/sh
rm -f /tmp/unread.txt
export PHONE_NUMBER=5555555555
export JAVA_HOME=/usr/lib/jvm/java-6-sun
curl -K .pwd https://mail.google.com/mail/feed/atom/unread > /tmp/unread.xml
groovy -classpath /home/admin/jars/postgresql-8.4-701.jdbc4.jar \
/home/admin/bin/Feed.groovy /tmp/unread.xml
if [ -e /tmp/unread.txt ]; then
echo "Sending notification of unread mail " `date`
cat /tmp/unread.txt | mail -s "Unread" $PHONE_NUMBER@txt.att.net
fi
This shell script uses a script written in Groovy to parse the curl output. There is nothing particularly
fancy about it. It does depend on the -K option to curl to pass in the username and password credentials.
This is a more secure method than passing them directly on the command line, or worse, including them
in the script itself. If the output file from the Groovy script exists after its execution,
the contents are emailed to the cell phone number's text message address. Ideally, this script will
run as a cron job, checking the mail feed at a specified interval.
Before you run the script, set up a database called 'mail', with one table 'processed_mail'.
The table schema is:
Column | Type | Modifiers
---------+------------------------+-----------
issued | character varying(100) | not null
author | character varying(100) |
summary | character varying(255) |
Indexes:
"processed_mail_pkey" PRIMARY KEY, btree (issued)
The Groovy script is documented, but in a nutshell, here's what it does:
- Connect to the database. I am using PostgreSQL in this case,
but any database will do.
- Set up a namespace for accessing the XML elements
- Create an output file
- Process each entry the XML file that is passed in as the first argument
- For each entry, check to see if it has been processed, i.e. it is in the database
- If it has not been processed, add an entry to the database, and output to the file
- After processing the unread entries, delete any entries in the database which
were not in the unread list
- Delete the output file if it is empty
import groovy.util.*
import groovy.sql.*;
//
// Setup a database connection
// using Postgres
//
def DB='jdbc:postgresql://localhost/mail'
def USER='postgres'
def PASSWORD='mypassword'
def DRIVER='org.postgresql.Driver'
Class.forName(DRIVER)
def sql = Sql.newInstance(DB,USER,PASSWORD)
assert sql
//
// The XML that comes back needs a namespace
//
def ns = new groovy.xml.Namespace("http://purl.org/atom/ns#")
//
// Process each unread email
// If the unread email has not been processed then
// add an entry to the database
//
def processEntry = { it, outputFile ->
//
// Look for an aleady processed entry
// If found then get out
//
def found = false
def issued = it[ns.issued].text()
sql.eachRow("select * from processed_mail where issued=${issued}") {
found = true
}
if (found) {
return
}
//
// Add the entry and output for the email text
//
println "Issued on " + issued
def author = it[ns.author][ns.name].text()
def summary = it[ns.summary].text()
println "Date: " + issued
println "From: " + author
println summary
sql.execute(
"insert into processed_mail (issued, author, summary) values(${issued}, ${author}, ${summary})")
outputFile.append("From: " + author + "\n")
outputFile.append("Date: " + issued + "\n")
outputFile.append("Summary: " + summary + "\n")
}
//
// Search in the list of entries for the
// issued date passed in
//
def searchUnread = { dbIssued, entries ->
def itFound = false
entries.each {
def xmlIssued = it[ns.issued].text()
if (dbIssued.equals(xmlIssued)) {
itFound = true
return
}
}
itFound;
}
//
// Process each row in the processed table
// If there is no entry in the list of unread email
// then delete the database row to prevent further processing
//
def cleanupDatabase = { dbIt, entries ->
def dbIssued = dbIt.issued
def entryFound = searchUnread(dbIssued, entries)
if (! entryFound) {
println "Deleting entry for " + dbIssued
sql.execute("delete from processed_mail where issued=${dbIssued}")
}
}
//
// Main body of code
//
def newFile = new File(args[0])
if (newFile.length() == 0) {
println "Empty new file"
System.exit(0)
}
//
// Create the output file
//
def output=new File("/tmp/unread.txt")
//
// Parse the input file
//
def parser = new XmlParser()
def feed = parser.parse(newFile)
//
// Gather the entries
//
def xmlEntries = feed[ns.entry]
//
// Process each entry and output if
// it has not already been processed
//
xmlEntries.each { processEntry(it, output) }
//
// Clean up the database
//
sql.eachRow("select * from processed_mail") { cleanupDatabase(it, xmlEntries) }
//
// If the output file is empty then delete
//
if (output.length() == 0) {
output.delete()
println "Deleted empty file"
}
This mail feed utility illustrates the usefulness of the combination of a common
system utility and a modern script language. This script could be modified to
include finer-grained decisions about the destinations of the mail notifications.
Both cURL and Groovy are available on Windows, Linux, and other platforms, so
this example can be tried out in almost any environment.
In Untagged
Comment (0)
Read More...
year there were a lot more technical sessions to choose from and overall the presentations seemed more professional than in the previous years. I also liked that the Monday tutorials were included in the overall conference registration fee, rather than an extra charge. 