Announce: nnwarchive

Announce: nnwarchive

[Shenghuo ZHU]

I am working on nnwarchive, a backend for web archives of
mailing-lists.  Now, it only works with egroups.com. More web archives
such as onelist.com, are supposed to be implemented. 

Should I put it in CVS after my today's pre-alpha test?

-- 
Shenghuo ZHU

Re: Announce: nnwarchive

[Eric Marsden]

There are now a number of packages which do washing of HTML pages:
nnweb.el, nnslashdot.el, nnwarchive.el, and my babel.el and watson.el
(which wash various search engines and translation services; see
X-URL), I'm probably forgetting some.

They all operate basically the same way: download the page, extract
the information, return it formatted. They face the same challenges:
conveniently and securely providing updates to users relatively often,
as web sites undergo facelifts.

So I am wondering if there is scope for a generic "wash.el" which
would take as an argument an URL (which is dynamically generated in
certain cases), a parser function which returns matches, operating on
the raw HTML, and provides automatic or semi-automatic update services
which connect to some trusted web site where washing-authors can put
updates.

-- 
Eric Marsden
It's elephants all the way down

Re: Announce: nnwarchive

[Shenghuo ZHU]

>>>>> "ecm" == Eric Marsden <emarsden@mail.dotcom.fr> writes:

[...]

ecm> So I am wondering if there is scope for a generic "wash.el" which
ecm> would take as an argument an URL (which is dynamically generated
ecm> in certain cases), a parser function which returns matches,
ecm> operating on the raw HTML, and provides automatic or
ecm> semi-automatic update services which connect to some trusted web
ecm> site where washing-authors can put updates.

A generic "wash.el" is not easy. Different pages need different wash.
Some pages need rebuild headers, some others need dissect a page into
several articles. A generic "wash.el" would have to use callback
functions or so. However, a "wash-util.el" may be possible.

-- 
Shenghuo ZHU

Re: Announce: nnwarchive

[Eric Marsden]

>>>>> "ZSH" == Shenghuo ZHU <zsh@cs.rochester.edu> writes:
>>>>> "ecm" == Eric Marsden <emarsden@mail.dotcom.fr> writes:

  ecm> So I am wondering if there is scope for a generic "wash.el"
  ecm> which would take as an argument an URL (which is dynamically
  ecm> generated in certain cases), a parser function which returns
  ecm> matches, operating on the raw HTML, and provides automatic or
  ecm> semi-automatic update services which connect to some trusted
  ecm> web site where washing-authors can put updates.

  ZSH> A generic "wash.el" is not easy. Different pages need different
  ZSH> wash. Some pages need rebuild headers, some others need dissect
  ZSH> a page into several articles. A generic "wash.el" would have to
  ZSH> use callback functions or so. However, a "wash-util.el" may be
  ZSH> possible.

I was thinking of an interface something like

  (wash url-generator washer)

where URL-GENERATOR is a function which takes `&rest args', and WASHER
is a function which operates on the current buffer and returns a list
of items (of arbitrary type, to be interpreted by the caller).

But the area where there's the most scope for reuse IMO is in the
semi-automatic updates. The current state of nnweb.el shows that there
is a need for auto-updates, and I think it would be easier for users
to trust a single module (which connects to a trusted server) to
download new code to be run in their emacs.
 
-- 
Eric Marsden
It's elephants all the way down

Re: Announce: nnwarchive

[Kai Großjohann]

Eric Marsden <emarsden@mail.dotcom.fr> writes:

> So I am wondering if there is scope for a generic "wash.el" which
> would take as an argument an URL (which is dynamically generated in
> certain cases), a parser function which returns matches, operating on
> the raw HTML, and provides automatic or semi-automatic update services
> which connect to some trusted web site where washing-authors can put
> updates.

How about xql.el?  XQL seems to be a fairly simple way to get stuff
from XML documents, and one would `only' have to normalize the HTML so
that it is valid HTML.

kai
-- 
This gubblick contains many nonsklarkish English flutzpahs,
but the overall pluggandisp can be glorked from context. -- David Moser

Re: Announce: nnwarchive

[Eric Marsden]

>>>>> "kg" == Kai Großjohann <Kai.Grossjohann@CS.Uni-Dortmund.DE> writes:

  kg> How about xql.el? XQL seems to be a fairly simple way to get
  kg> stuff from XML documents, and one would `only' have to normalize
  kg> the HTML so that it is valid HTML.

surely you jest :-) Have you seen the HTML that some of the search
engines return?

  <URL:http://www.w3.org/TandS/QL/QL98/pp/xql.html>

-- 
Eric Marsden
It's elephants all the way down

Re: Announce: nnwarchive

[Shenghuo ZHU]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

>>>>> "ZSH" == Shenghuo ZHU <zsh@cs.rochester.edu> writes:

ZSH> I am working on nnwarchive, a backend for web archives of
ZSH> mailing-lists.  Now, it only works with egroups.com. More web
ZSH> archives such as onelist.com, are supposed to be implemented.

I attached the code.

To setup, use the following code.

(defun nnwarchive-load ()
  "Setup nnwarchive."
  (interactive)
  (require 'nnwarchive)
  (setq nnwarchive-login "your@email.address.at.egroups") 
  (setq nnwarchive-passwd "the passwd") 
  (push '("nnwarchive" none)  gnus-valid-select-methods))

Maybe a small bug in w3 0.44. (load-library "url") to make sure
`url-maybe-relative' is using the copy from url.el.

-- 
Shenghuo ZHU


[-- Attachment #2: nnwarchive.el --]
[-- Type: application/emacs-lisp, Size: 15766 bytes --]

Re: Announce: nnwarchive

[Kai Großjohann]

Eric Marsden <emarsden@mail.dotcom.fr> writes:

> surely you jest :-) Have you seen the HTML that some of the search
> engines return?

No, err.  Which one is particularly bad?  And what is the problem with
it?  That the HTML is so broken that it is hard to convert it to XML?
Or that the HTML structure is convoluted?

kai
-- 
This gubblick contains many nonsklarkish English flutzpahs,
but the overall pluggandisp can be glorked from context. -- David Moser

Re: Announce: nnwarchive

[Eric Marsden]

>>>>> "kg" == Kai Großjohann <Kai.Grossjohann@CS.Uni-Dortmund.DE> writes:

  ecm> surely you jest :-) Have you seen the HTML that some of the search
  ecm> engines return?

  kg> No, err. Which one is particularly bad? And what is the problem
  kg> with it? That the HTML is so broken that it is hard to convert
  kg> it to XML? Or that the HTML structure is convoluted?

<URL:http://validator.w3.org/check?uri=http://www.google.com/>
<URL:http://validator.w3.org/check?uri=http://www.google.com/search?q=bug>
<URL:http://validator.w3.org/check?uri=http://www.altavista.com/  >
<URL:http://validator.w3.org/check?uri=http://www.snap.com/>
<URL:http://validator.w3.org/check?uri=http://www.yahoo.com/>  

-- 
Eric Marsden
It's elephants all the way down

Re: Announce: nnwarchive

[Lars Magne Ingebrigtsen]

Shenghuo ZHU <zsh@cs.rochester.edu> writes:

> I am working on nnwarchive, a backend for web archives of
> mailing-lists.  Now, it only works with egroups.com. More web archives
> such as onelist.com, are supposed to be implemented. 
> 
> Should I put it in CVS after my today's pre-alpha test?

Yes; please do.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Announce: nnwarchive

[Lars Magne Ingebrigtsen]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> How about xql.el?  XQL seems to be a fairly simple way to get stuff
> from XML documents, and one would `only' have to normalize the HTML so
> that it is valid HTML.

w3 parses HTML very well; almost no matter how broken, w3 will give
out reasonable results.

The problem with HTML is that the elements are basically meaningless.
Finding out what's what ("Is this number the number of articles in the 
topic or the number of authors?") is what is difficult.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Announce: nnwarchive

[Lars Magne Ingebrigtsen]

Eric Marsden <emarsden@mail.dotcom.fr> writes:

> They all operate basically the same way: download the page, extract
> the information, return it formatted. They face the same challenges:
> conveniently and securely providing updates to users relatively often,
> as web sites undergo facelifts.

The layout parsing is one thing, but w3 can deal with that better than 
the current regexp-based things in nnweb, I think.

> So I am wondering if there is scope for a generic "wash.el" which
> would take as an argument an URL (which is dynamically generated in
> certain cases), a parser function which returns matches, operating on
> the raw HTML, and provides automatic or semi-automatic update services
> which connect to some trusted web site where washing-authors can put
> updates.

It's an interesting idea.  It doesn't even have to be code on the
trusted web site -- it could just be something that would say how the
w3 parse tree should be interpreted.  Like -- "this node in the tree
is the author name, and this node is the body of the message".

It sounds like an interesting project.  "How to extract information
from HTML pages."  But it'll be difficult.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Announce: nnwarchive

[Eric Marsden]

>>>>> "lmi" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

  lmi> It's an interesting idea.  It doesn't even have to be code on the
  lmi> trusted web site -- it could just be something that would say how the
  lmi> w3 parse tree should be interpreted.  Like -- "this node in the tree
  lmi> is the author name, and this node is the body of the message".
  lmi> 
  lmi> It sounds like an interesting project.  "How to extract information
  lmi> from HTML pages."  But it'll be difficult.

hmm, that's not where I was headed, but it's interesting. Current
research in the area are WebL [1] which talks about a "markup algebra
[which] extracts structured and unstructured values from pages for
computation, and is based on algebraic operations on sets of markup
elements.", and sgrep [2], a tool for searching structured text, which
"implements an algebra of unrestricted text fragments called regions.
The algebra allows the retrieval of document components, represented
as regions, based on conditions on their relative containment and
ordering.".


[1] <URL:http://gatekeeper.dec.com/pub/DEC/SRC/technical-notes/SRC-1997-029-html/> 
[2] <URL:http://www.cs.helsinki.fi/TR/C-1996/83/>

-- 
Eric Marsden

Re: Announce: nnwarchive

[Kai Großjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> The problem with HTML is that the elements are basically meaningless.
> Finding out what's what ("Is this number the number of articles in the 
> topic or the number of authors?") is what is difficult.

Yes, one would have to address them by position in the (HTML parse)
tree, I guess.  Or maybe by looking at some relatives.

In XQL you could say that the number of articles is in an <em> element
which has a <b> cousin, whereas the number of authors is in an <em>
element which is the first one inside the third <ul>.  And stuff.

kai
-- 
This gubblick contains many nonsklarkish English flutzpahs,
but the overall pluggandisp can be glorked from context. -- David Moser

Re: Announce: nnwarchive

[Lars Magne Ingebrigtsen]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> In XQL you could say that the number of articles is in an <em> element
> which has a <b> cousin, whereas the number of authors is in an <em>
> element which is the first one inside the third <ul>.  And stuff.

Yes, that would help.  But can you say "the article number is the last 
<td> in the <tr> that contains only numerical characters"?  That's
what I had to do for nnultimate, for instance.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Announce: nnwarchive

[Lars Magne Ingebrigtsen]

Eric Marsden <emarsden@mail.dotcom.fr> writes:

> hmm, that's not where I was headed, but it's interesting. Current
> research in the area are WebL [1] which talks about a "markup algebra
> [which] extracts structured and unstructured values from pages for
> computation, and is based on algebraic operations on sets of markup
> elements.", 

Yeah.  We need to get some master's student interested in implementing 
this kind of stuff for Gnus.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Announce: nnwarchive

[Kai Großjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> 
> > In XQL you could say that the number of articles is in an <em> element
> > which has a <b> cousin, whereas the number of authors is in an <em>
> > element which is the first one inside the third <ul>.  And stuff.
> 
> Yes, that would help.  But can you say "the article number is the last 
> <td> in the <tr> that contains only numerical characters"?  That's
> what I had to do for nnultimate, for instance.

Well, you may need to add regexp matching to XQL, but what does that
mean?  It doesn't mean that doing xql.el is bad, it just means that it
may be useful to implement some extensions to the XQL standard, as
well.

(For example, the Perl module XML::XQL implements such extensions.)

BillP has posted that he doesn't want to maintain W3 anymore (I
think), and that he wishes that somebody write an XML display engine.
This requires an XSL implementation and XQL is (pretty much) a module
which is needed in XSL.  So xql.el would be useful in that context, as
well.

kai
-- 
This gubblick contains many nonsklarkish English flutzpahs,
but the overall pluggandisp can be glorked from context. -- David Moser

Re: Announce: nnwarchive

[Karl EICHWALDER]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

|   BillP has posted that he doesn't want to maintain W3 anymore (I
|   think),

Reading the Emacs/W3 mailinglist, he's involved again (thanks William!).

|   and that he wishes that somebody write an XML display engine.
|   This requires an XSL implementation

or DSSSL!

-- 
work    : ke@suse.de                          |
        : http://www.suse.de/~ke/             |       ------    ,__o
personal: ke@gnu.franken.de                   |      ------   _-\_<,
        : http://www.franken.de/users/gnu/ke/ |     ------   (*)/'(*)

Re: Announce: nnwarchive

[Kai Großjohann]

Karl EICHWALDER <ke@gnu.franken.de> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> 
> |   and that he wishes that somebody write an XML display engine.
> |   This requires an XSL implementation
> 
> or DSSSL!

Oh, yes, I completely forgot about that.  In the context of Emacs, it
might be better to do DSSSL rather than XSL.  Then, of course, XQL
doesn't make much sense anymore.  Hm.

kai
-- 
This gubblick contains many nonsklarkish English flutzpahs,
but the overall pluggandisp can be glorked from context. -- David Moser

Re: Announce: nnwarchive

[Steinar Bang]

>>>>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann):

> Karl EICHWALDER <ke@gnu.franken.de> writes:
>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>> 
>> |   and that he wishes that somebody write an XML display engine.
>> |   This requires an XSL implementation
>> 
>> or DSSSL!

> Oh, yes, I completely forgot about that.  In the context of Emacs,
> it might be better to do DSSSL rather than XSL.

Can't you do both?  Aren't DSSSL and XSL pretty much the same thing
with different syntax?

Re: Announce: nnwarchive

[Kai Großjohann]

Steinar Bang <sb@metis.no> writes:

> Can't you do both?  Aren't DSSSL and XSL pretty much the same thing
> with different syntax?

They used to (and in fact there was a program to convert from XSL to
DSSSL which Jade could grok), but then the XSL people completely
revamped XSL, and now I'm not sure anymore.

kai
-- 
This gubblick contains many nonsklarkish English flutzpahs,
but the overall pluggandisp can be glorked from context. -- David Moser