Re: nnir/freeWAIS-sf

[Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann)]

On 20 Jul 2000, Harry Putnam wrote:

>   bsd > waissearch -d mail 'global resounding and silence' 
> 
>    Search Response:
>     NumberOfRecordsReturned: 1
>      1: Score: 2113, lines:  54 '2177   /home/reader/Mail/ding2/'
> 
> Seems to work... but wait.. that message contains neither
> `resounding' nor silence.

Whee.  Hm.  Maybe fwsf implements `and' in a fuzzy way.  This is
useful for people who issue queries like `term1 and term2 and ... and
term10'.  If there are no documents with all ten terms, chances are
that people will be happy with a document containing only nine of
them.

I'm not sure about this one, though.

> Message-ID: <vxku2fqhjin.fsf@mesquite.charcoal.com> (on ding)
> 
> Whats worse
> 
> `grep -rl 'resounding.*silence' ~Mail'  easily finds 5 that actually
> contain the strings.
> 
> /home/reader/Mail/ding2/2771 <m2vgybe81z.fsf@reader.ptw.com>
> /home/reader/Mail/ding2/2790
> <14703.1528.800308.229816@klortho.stepstone.ie>
> /home/reader/Mail/bbdb/460 <m2vgybe81z.fsf@reader.ptw.com>
> /home/reader/Mail/bbdb/463
> <14703.1528.800308.229816@klortho.stepstone.ie>
> /home/reader/Mail/bbdb/472 <m2n1jkgpsp.fsf@reader.ptw.com>
> 
> Its looking more and more like freewais is just not a sturdy tool
> like glimpse.  It has to be molycoddled every inch of the way, every
> phase is as painful as pulling teeth.  Then the end result is flaky
> and not dependable.  It lacks precision in searching.  And on the
> command line fails to show the hits in some fashion.  Only full
> documents.

Yes, it appears so.  FWIW, I'm quite interested in reading all this.
>From an Information Retrieval point of view, fwsf is doing the right
things.  Yet it is obviously not easy to use at all!  Quite amazing.

But thanks a lot for persevering, this sure helps me to learn things,
and I can only hope that my feeble attempts at getting some of this
into DesIRe (the successor of fwsf) will bear some fruits.

> The little awk based search tool I made, is much sturdier and can be
> used on any unix like platform.  It is excruciatingly slow, but
> because it is fully regexp based it finds strings with great
> precision.  Needs some handy way to insert the search string, in an
> easy one step manor very badly too.

Maybe it could be integrated into nnir.el.  Hm.  Do you think that in
principle the idea of producing a summary buffer containing the search
results is a good idea?  If so, it might be worth it to try to
integrate the two.

Basically, nnir.el needs a list of article identifiers as a result.
The article identifier needs to contain the group name (in some form)
and the article number.

So if your tool just spits out the file names, this should be
sufficient for searching nnml groups.

> Sometimes `ranking'or `heuristics' of some kind aren't what is
> needed.

:-)

> I'm thinking of how to intetgrate that into gnus.  It can report hte
> group and file name, message id or whatever.  I wasn't able to see
> in nnir how the lisp code grabs that info from glimpse or wais.  But
> surely if the tool can pass the article number, filename, message
> ID, then gnus can assemble the hits.

Yes.  Hm.  You may wish to have a look at the nnir-run-glimpse
function.  This function contains two parts.  The first part invokes
glimpse with the right options.  The second part expects glimpse to
produce a list of file names, which is then massaged in an appropriate
way.

It seems that you can reuse the second part (if your tool just prints
file names), but have to change the first part a bit.

When you have written your nnir-run-harrys-tool function, you can hook
it into nnir.el by adding an entry into nnir-engines, like this:

(add-to-list 'nnir-engines '(harrys-tool nnir-run-harrys-tool nil))

And then you (setq nnir-search-engine 'harrys-tool), and that's it!

(You might need a couple of variables, for example a variable for your
home dir, so that you can cut off the right prefix from the file
names.  Can you understand the code in nnir-run-glimpse that does this?)

kai
-- 
I like BOTH kinds of music.