Re: [9fans] webscript

[Dimitry Golubovsky <dimitry@golubovsky.org>]

Russ Cox wrote:
> I would like to be able to write scripts like this:
> 
> 	load "http://apc-reset/outlets.htm"
> 	find "yoshimi"
> 	nearest option, set "Immediate Reboot"
> 	submit
> 
> or like this:
> 
> 	load "http://www.fedex.com/Tracking"
> 	find form
> 	enter "792544024753"
> 	submit
> 	
> 	if (find "No information") {
> 	   select enclosing td
> 	   print
> 	} else if (find "Ship date") {
> 	   select enclosing table
> 	   select enclosing table
> 	   print
> 	} else {
> 	   print ">>> Unexpected Results\n"
> 	   print
> 	}
> 
> Does anyone know of programs/languages that let you
> script web sessions like that?  Searching around finds lots
> of mentions of web scraping but no actual programs.
> 

Well, Haskell has several HTML/XML parser packages[0] that parse a 
sequence of tags and return some tree-like structures, and then various 
queries may be made over that tree, like extracting tags with given 
properties, or building new document trees/extracting/transforming 
subtrees. There are some facilities to connect to web servers and 
retrieve HTTP responses, and I believe to submit forms, too (although I 
never tried the latter practically, I only worked with the GET method).

Then, with Haskell you may create sort of your own domain specific 
language (DSL) to perform your tasks. In fact, your example seems like 
you indeed want some DSL to analyze web pages.

A very simple example of this may be found in my "cabalfind"[1] program 
which parses a search engine (eg Google) response in order to find links 
with required properties (pointing to files with ".cabal" [2] suffix). 
Although cabalfind does not introduce any DSLs.

However I see two issues here: someone (you?) has to learn Haskell, and, 
if thinking of Plan9, there is no Haskell implementation except an old 
Hugs which would be too slow for this task (I still have some plans to 
port GHC or NHC, but cannot yet find my own resources even to start 
porting).

-------
[0] but there must be a lot of that for Java; why don't you want to use 
that?

[1] http://www golubovsky.org/repos/cabalfind,
     http://www.haskell.org/hawiki/CabalFind

[2] cabal is Haskell software package management system.