Re: Abandoning the concept of groups as a storage medium?

[Ted Zlatanov <tzz@lifelogs.com>]

On Mon, 04 May 2009 13:07:20 +0200 David Engster <deng@randomsample.de> wrote: 

DE> I think the registry should save all important headers of a message, and
DE> maybe also some MIME information, like attachment names. Of course, as
DE> Ted also said, this information can't be saved anymore in plain text
DE> files like gnus.registry.eld, but needs some kind of external database
DE> back end.

You're probably thinking of saving the keywords too, yes?  Generally any
key-val map should be storeable; this is ridiculously easy with proper
hashtable serialization, which is coming in Emacs CVS once the pretest
is over.  I have a patch ready to go.  With that serialization, we can
finally write out the registry whole or piece by piece in one simple
command, and read it back just as easily.

With an IMAP storage backend for the registry, this could result in
registry entries stored in messages like this:

Subject: [entry subject]
Keywords: [entry keywords]
Message-ID: [entry ID].gnus.registry (so it doesn't collide with the original)
... other headers ...

[whole serialized entry as the body]

so then we could do IMAP SEARCH by keyword, for example.

>> nnregistry?
>> group list: dynamic based on tags (labels) defined by user
>> article list in group: generated on entry with a tag search
>> article retrieve: uses the original article backend

DE> Exactly. What makes this stuff difficult is that article numbers in Gnus
DE> have to be unique, just like in IMAP. nnmairix "solves" this problem by
DE> adding an offset to the article numbers each time a group is re-created.

I wonder if it would be possible to use message IDs as article numbers.

DE> However, full text search is another matter entirely. This simply cannot
DE> be done in Emacs Lisp.
>> 
>> The index necessary for good search performance would be huge, but easy
>> to store in a database (on a server, on IMAP, whatever).  It's easy to
>> parallelize these searches (especially with IMAP as the backend).  So
>> there's some hope.

DE> Maybe. But just feeding such a database through Emacs Lisp would already
DE> be a very time consuming task, considering the huge amount of mails many
DE> people have (I have about 1GB, and I think this is pretty average).

I think it's possible, as long as we keep the indexing and searching on
the server, and Emacs Lisp is only the query and display agent.
Displaying large amounts of data is not so bad if you can get it 50
results at a time...

Ted