Fledgling project nnsearch.el

Fledgling project nnsearch.el

[Kai Großjohann]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

There is nnir.el but it has a few problems.  So I'm starting a
rewrite.  I have something which does more than a no-op, but it's
nowhere complete, yet.  But as the saying goes, release early, release
often...

What are the problems with nnir?

* It includes code from people who may not have submitted the
  paperwork, so it can never be part of Gnus.  I think it would be
  useful to have searching be part of Gnus.

* It has obfuscated configuration that's prone to bugs, too.

What are the aims for nnsearch?

* Make it easy to set up.

* Allow for searching multiple search engines and multiple select
  methods in one go.  For example, you might index your nnml stuff
  with Namazu but also want to use IMAP searching for nnimap groups.

What does nnsearch currently do?

* It can use Namazu to query your nnml backend, if it is named
  "nnml:".

How to set it up?

* Let's say you have this server spec:
      (nnml "" ...other.parms...)
  Change it to read like so:
      (nnml ""
       (nnsearch-root-directory "~/Mail")
       (nnsearch-namazu-index-directory "~/.namazu")
       ...other.parms...)

* Index your mail with Namazu 2.0.x:

    mkdir ~/.namazu
    cd ~/.namazu
    mknmz -h ~/Mail

* Load nnsearch.el.

* Invoke M-x gnus-group-search RET from the group buffer.
-- 
Two cafe au lait please, but without milk.


[-- Attachment #2: nnsearch.el --]
[-- Type: application/emacs-lisp, Size: 12885 bytes --]

Re: Fledgling project nnsearch.el

[Kai Großjohann]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> What does nnsearch currently do?
>
> * It can use Namazu to query your nnml backend, if it is named
>   "nnml:".
>
> How to set it up?
>
> * Let's say you have this server spec:
>       (nnml "" ...other.parms...)
>   Change it to read like so:
>       (nnml ""
>        (nnsearch-root-directory "~/Mail")
>        (nnsearch-namazu-index-directory "~/.namazu")
>        ...other.parms...)

Now you can use it on multiple nnml backends, regardless of their
name.  The logic to choose the search engine is in place.  (Though
only Namazu is currently supported.)

Now the server spec should read like this:

    (nnml ""
     (nnsearch-function nnsearch-namazu-query)
     (nnsearch-root-directory "~/Mail")
     (nnsearch-namazu-index-directory "~/.namazu")
     ...other.parms...)

What's the scoop on the current feature-freeze?  Does this mean
nnsearch.el can't go in at the moment?  (It can't go in until it gets
much better anyway.)
-- 
Two cafe au lait please, but without milk.


[-- Attachment #2: nnsearch.el --]
[-- Type: application/emacs-lisp, Size: 13295 bytes --]

Re: Fledgling project nnsearch.el

[Kai Großjohann]

It seems I'm too stupid to implement a simple parser.  I thought it
would be good for nnsearch.el to accept a standard query syntax, and
to transmogrify these queries into things that the search engines
grok.

I looked at the Namazu query syntax, and I like it.  It goes like
this:

    Query ::= Term
            | Term "and" Query
            | Term "or" Query
            | Term "not" Query
            | "(" Query ")"

    Term  ::= Word
            | Field ":" Word

    Word  ::= { a string of letters }

    Field ::= { name of mail header }

This appears to be simple enough.  But my parser isn't working.  I
tried a recursive descent, and I'm getting weird results.

So does somebody have a good design pattern that could be used to
construct that kind of parser?

It seems to be clear that I need a function that reads a token.
Tokens are: and, or, not, open paren, close paren, colon, words and
fields.  (I guess that word and field can't be distinguished by the
lexer.)

Then it also seems to be clear that I need a function for each
nonterminal in the grammar.  So there will be a parse-query, a
parse-term, a parse-field, and a parse-word.

What should be the argument of each such function and what should be
the return value?

I put the string into a buffer and used point as an indication what
to read next.  For the lookahead I used save-excursion.  The parse-*
functions didn't have arguments, and the return values were (partial)
parse trees.

Help?
-- 
Two cafe au lait please, but without milk.

Re: Fledgling project nnsearch.el

[Kai Großjohann]

[-- Attachment #1: Type: text/plain, Size: 20 bytes --]

I think I have it:


[-- Attachment #2: Type: application/emacs-lisp, Size: 2402 bytes --]

[-- Attachment #3: Type: text/plain, Size: 47 bytes --]

-- 
Two cafe au lait please, but without milk.

Re: Fledgling project nnsearch.el

[Juha Autero]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> It seems I'm too stupid to implement a simple parser.  I thought it
> would be good for nnsearch.el to accept a standard query syntax, and
> to transmogrify these queries into things that the search engines
> grok.

What is wrong with lisp syntax? It is simple and you already have
tools to parse and process it. 
-- 
Juha Autero
http://www.iki.fi/jautero/
Eschew obscurity!

Re: Fledgling project nnsearch.el

[Kai Großjohann]

Juha Autero <Juha.Autero@iki.fi> writes:

> kai.grossjohann@gmx.net (Kai Großjohann) writes:
>
>> It seems I'm too stupid to implement a simple parser.  I thought it
>> would be good for nnsearch.el to accept a standard query syntax, and
>> to transmogrify these queries into things that the search engines
>> grok.
>
> What is wrong with lisp syntax? It is simple and you already have
> tools to parse and process it. 

I'm afraid it might be more inconvenient to type.  You know, I'm
thinking about a query language for interactive use by the end users.

It's very easy to type (+ 3 4) into the *scratch* buffer or ielm or
eshell, but still, people invoke calculator.el and type 3 + 4 instead.
-- 
Two cafe au lait please, but without milk.

Re: Fledgling project nnsearch.el

[Christoph Conrad]

Hi Kai,

    > * It includes code from people who may not have submitted the

If you ask, you get an answer. I have submitted the necessary papers.
All elisp code i have written, write and will write in the future is
assigned to the FSF.

Herzliche Grüße,
  Christoph
-- 
"Lieber ein kleiner Teil von etwas ganz Großem als ein großer Teil von
einem Haufen Scheiße." (Franziska van Almsick)

Re: Fledgling project nnsearch.el

[Kai Großjohann]

Christoph Conrad <christoph.conrad@gmx.de> writes:

> If you ask, you get an answer. I have submitted the necessary papers.
> All elisp code i have written, write and will write in the future is
> assigned to the FSF.

Argh.  Silly me.  I remember that there were some contributions, but
I didn't realize you were *so* active that you actually did all of
them (except one, done by Simon Josefsson)!  Whee.

[time passes]

Ah, the contributions from our student are not covered by a copyright
assignment.  Hm.
-- 
Two cafe au lait please, but without milk.

Re: Fledgling project nnsearch.el

[Kai Großjohann]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Ah, the contributions from our student are not covered by a copyright
> assignment.  Hm.

Maybe a good approach is to redo the infrastructure stuff (the
student was working on that) and then to plug your search engine
interfaces back in.

WDYT?
-- 
Two cafe au lait please, but without milk.

Re: Fledgling project nnsearch.el

[Christoph Conrad]

Hi Kai,

    > Maybe a good approach is to redo the infrastructure stuff (the
    > student was working on that) and then to plug your search engine
    > interfaces back in.

    > WDYT?

Exactly what i thought.

Herzliche Grüße,
  Christoph

gnus-namazu.el (was: Fledgling project nnsearch.el)

[TSUCHIYA Masatoshi]

[-- Attachment #1: Type: text/plain, Size: 1075 bytes --]

Hi,

>> On Sun, 24 Aug 2003 16:29:17 +0200
>> kai.grossjohann@gmx.net (Kai Großjohann) said as follows:

>There is nnir.el but it has a few problems.  So I'm starting a
>rewrite.

I have developed gnus-namazu.el, which is an another search interface
using Namazu.  It has some advantages and some disadvantages than
nnir.el.  Advantages are:

(1) Persistent articles and agentized articles: gnus-namazu can handle
    them.

(2) Virtual group: gnus-namazu generate a virtual group that consists
    of all groups including all discovered articles.  Therefore,
    gnus-summary-refer-parent-article() is available in the summary
    buffer of the virtual group, even if a parent article is not
    included in discovered articles.

(3) Simple configuration: gnus-namazu is simplified for Namazu, and
    only four steps are required to start using gnus-namazu.

The biggest disadvantage is that gnus-namazu.el supports only Namazu,
but do not support other search engines.

I am inviting any comments about this program.

-- 
TSUCHIYA Masatoshi

[-- Attachment #2: gnus-namazu.el.gz --]
[-- Type: application/octet-stream, Size: 8423 bytes --]

Re: gnus-namazu.el

[TSUCHIYA Masatoshi]

Hi,

>> On Tue, 26 Aug 2003 19:54:53 +0200
>> bouvin@daimi.au.dk (Niels Olof Bouvin) said as follows:

>However, I cannot get it to give me cached articles as hits (and it
>certainly indexed my cached articles).  What it does is slightly different:
>if I as an example search for the term 'Shannara' which I happen to know is
>present only in ~/News/cache/dk/kultur/sciencefiction/9537, it does not give
>me this article, but instead opens the dk.kultur.sciencefiction newsgroup
>with a large number of articles.

Could you evaluate the following expression?

  (gnus-namazu/search nil "Shannara")

When gnus-namazu works correctly, it will return pairs of groups and
articles, as follows:

  (("dk.kultur.sciencefiction" . 9537))

-- 
TSUCHIYA Masatoshi

Re: gnus-namazu.el

[TSUCHIYA Masatoshi]

>> On Wed, 10 Sep 2003 13:57:13 +0200
>> bouvin@daimi.au.dk (Niels Olof Bouvin) said as follows:

>> Could you evaluate the following expression?
>> 
>>   (gnus-namazu/search nil "Shannara")

>(gnus-namazu/search nil "Shannara")
>(("nnml:mail.archive.2003-08" . 77) ("dk.kultur.sciencefiction" . 9537))

Well, it looks fine.

>> When gnus-namazu works correctly, it will return pairs of groups and
>> articles, as follows:
>> 
>>   (("dk.kultur.sciencefiction" . 9537))

>The first hit is my original mail to you.  When I search for
>'Shannara' now (in Gnus), I get only this mail - not the d.k.sf one.
>However, I am a bit suspicious about the lack of prefixes to the
>d.k.sf hit - how does gnus-namazu know that this is a cached article?

If I know what you mean, d.k.sf is not correct as a newsgroup name,
and lacks a prefix such as `nntp:', right?

In this case, there is a bug in gnus-namazu/make-directory-table()
that creates a table of all possible cache directories from
`gnus-newsrc-hashtb'.  Gnus-namazu decides whether articles found by
Namazu are cached or not based on the table.

symbol-name() is used to get a group name from a symbol interned in
`gnus-newsrc-hashtb', as follows:

  (mapatoms
   (lambda (group)
     (unless (gnus-ephemeral-group-p (setq group (symbol-name group)))
     ...))
   gnus-newsrc-hashtb)

Could you tell me a right way to get all group names?

-- 
TSUCHIYA Masatoshi

gnus-namazu.el in gnus/contrib?

[TSUCHIYA Masatoshi]

Hi,

I developed gnus-namazu.el, which is a search interface using Namazu,
like nnir.el.  It has some advantages and some disadvantages than
nnir.el.  Advantages are:

(1) Persistent articles and agentized articles: gnus-namazu can handle
    them.

(2) Virtual group: gnus-namazu generate a virtual group that consists
    of all groups including all discovered articles.  Therefore,
    gnus-summary-refer-parent-article() is available in the summary
    buffer of the virtual group, even if a parent article is not
    included in discovered articles.

(3) Simple configuration: gnus-namazu is simplified for Namazu, and
    only four steps are required to start using gnus-namazu.

The biggest disadvantage is that gnus-namazu.el supports only Namazu,
but does not support other search engines.

gnus-namazu.el was originally developed for T-gnus, which is a
localized version of Gnus, and became quite popular among T-gnus
users.  I ported it for Oort Gnus, and posted it to this group eight
month ago.

And now, I'd like to contribute to Gnus CVS.  Can I put it in
gnus/contrib?

-- 
TSUCHIYA Masatoshi

Re: gnus-namazu.el in gnus/contrib?

[Kai Grossjohann]

TSUCHIYA Masatoshi <tsuchiya@namazu.org> writes:

> And now, I'd like to contribute to Gnus CVS.  Can I put it in
> gnus/contrib?

Way cool!  Thanks.

Kai

Re: gnus-namazu.el in gnus/contrib?

[Stefan Reichör]

H!

> And now, I'd like to contribute to Gnus CVS.  Can I put it in
> gnus/contrib?

Yes please!
I am a happy user of gnus-namazu.el


Stefan.

Re: gnus-namazu.el in gnus/contrib?

[TSUCHIYA Masatoshi]

>> On Sat, 01 May 2004 11:43:01 +0900
>> tsuchiya@namazu.org (TSUCHIYA Masatoshi) said as follows:

> Can I put it in gnus/contrib?

I have put gnus-namazu.el in gnus/contrib with two approvals.

-- 
TSUCHIYA Masatoshi

Re: gnus-namazu.el in gnus/contrib?

[Reiner Steib]

On Tue, May 11 2004, TSUCHIYA Masatoshi wrote:

> I have put gnus-namazu.el in gnus/contrib with two approvals.

Could you please add a brief description about it in contrib/README?

BTW: nnir.el and starttls.el are not mentioned there neither (and some
files are just mentioned without a description).

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: gnus-namazu.el in gnus/contrib?

[TSUCHIYA Masatoshi]

>> On Tue, 11 May 2004 10:51:54 +0200
>> 4.uce.03.r.s@nurfuerspam.de (Reiner Steib) said as follows:

> Could you please add a brief description about it in contrib/README?

Done.

-- 
TSUCHIYA Masatoshi