Re: Abandoning the concept of groups as a storage medium?

[Ted Zlatanov <tzz@lifelogs.com>]

On Tue, 28 Apr 2009 10:56:39 +0200 David Engster <deng@randomsample.de> wrote: 

DE> Jan Rychter <jan@rychter.com> writes:
>> I want to be able to set flags and tags on *any* E-mail message
>> *anywhere*, not just in the "real group it belongs to". I don't want
>> my "groups" to have anything to do with the way my messages are
>> stored. E-mail should be stored in a key/value store with metadata
>> copied and indexed separately.
>> 
>> The problem I see with Gnus is that it is designed around a central
>> concept of a mail backend which exposes groups. 

DE> Yes, but I don't see this as a restriction for what you would like to
DE> have. Gnus can handle "dynamic" groups, where the contents changes all
DE> the time, although it requires a lot of work to get right. As you say,
DE> nnmairix comes pretty close, and it strictly works with the Gnus group
DE> back end API. My main problem with maintaining nnmairix is that the back
DE> ends behave differently, especially when it comes to marks and unread
DE> count.

The big problem I see is that Gnus can't build groups asynchronously.
Emacs Lisp itself is the impediment here.

>> The registry, if I understand correctly, is a workaround for some of
>> the problems people encountered with this approach.
>> 
>> Is there any way to achieve what I want with Gnus? Is anybody working on
>> something of the kind? What would be the possible approaches?

DE> We just had this discussion, and Ted raised some interesting points
DE> regarding the registry and how it could be extended, and I agree with
DE> him.

DE> I think the registry should save all important headers of a message, and
DE> maybe also some MIME information, like attachment names. Of course, as
DE> Ted also said, this information can't be saved anymore in plain text
DE> files like gnus.registry.eld, but needs some kind of external database
DE> back end.

DE> We could then add a back end which can create virtual groups based on
DE> registry information. One could extend nnir to do that, but I'd vote for
DE> creating a completely new one.

nnregistry?

group list: dynamic based on tags (labels) defined by user
article list in group: generated on entry with a tag search
article retrieve: uses the original article backend

I'm definitely not going to get to it anytime soon, but if anyone else
feels adventurous, I'll help out any way I can.

DE> However, full text search is another matter entirely. This simply cannot
DE> be done in Emacs Lisp.

The index necessary for good search performance would be huge, but easy
to store in a database (on a server, on IMAP, whatever).  It's easy to
parallelize these searches (especially with IMAP as the backend).  So
there's some hope.

Ted