Re: Abandoning the concept of groups as a storage medium?

[David Engster <deng@randomsample.de>]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Tue, 28 Apr 2009 10:56:39 +0200 David Engster <deng@randomsample.de> wrote: 
>
> DE> Jan Rychter <jan@rychter.com> writes:
>>> I want to be able to set flags and tags on *any* E-mail message
>>> *anywhere*, not just in the "real group it belongs to". I don't want
>>> my "groups" to have anything to do with the way my messages are
>>> stored. E-mail should be stored in a key/value store with metadata
>>> copied and indexed separately.
>>> 
>>> The problem I see with Gnus is that it is designed around a central
>>> concept of a mail backend which exposes groups. 
>
> DE> Yes, but I don't see this as a restriction for what you would like to
> DE> have. Gnus can handle "dynamic" groups, where the contents changes all
> DE> the time, although it requires a lot of work to get right. As you say,
> DE> nnmairix comes pretty close, and it strictly works with the Gnus group
> DE> back end API. My main problem with maintaining nnmairix is that the back
> DE> ends behave differently, especially when it comes to marks and unread
> DE> count.
>
> The big problem I see is that Gnus can't build groups asynchronously.
> Emacs Lisp itself is the impediment here.

That's why it better has to build those groups fast. ;-)

But you're right, of course. There has been some work on introducing
parallelism into Emacs Lisp on emacs-devel. Maybe Emacs 24 will have
something like that...

> DE> I think the registry should save all important headers of a message, and
> DE> maybe also some MIME information, like attachment names. Of course, as
> DE> Ted also said, this information can't be saved anymore in plain text
> DE> files like gnus.registry.eld, but needs some kind of external database
> DE> back end.
>
> DE> We could then add a back end which can create virtual groups based on
> DE> registry information. One could extend nnir to do that, but I'd vote for
> DE> creating a completely new one.
>
> nnregistry?

Makes sense. :-)

> group list: dynamic based on tags (labels) defined by user
> article list in group: generated on entry with a tag search
> article retrieve: uses the original article backend

Exactly. What makes this stuff difficult is that article numbers in Gnus
have to be unique, just like in IMAP. nnmairix "solves" this problem by
adding an offset to the article numbers each time a group is re-created.

> I'm definitely not going to get to it anytime soon, but if anyone else
> feels adventurous, I'll help out any way I can.

My guess is that most of the needed code already exists in nnir and
nnmairix. Unfortunately, I won't have time to do it either, at least not
in the coming months.

> DE> However, full text search is another matter entirely. This simply cannot
> DE> be done in Emacs Lisp.
>
> The index necessary for good search performance would be huge, but easy
> to store in a database (on a server, on IMAP, whatever).  It's easy to
> parallelize these searches (especially with IMAP as the backend).  So
> there's some hope.

Maybe. But just feeding such a database through Emacs Lisp would already
be a very time consuming task, considering the huge amount of mails many
people have (I have about 1GB, and I think this is pretty average).

-David