Re: mail group by author?

[Ted Zlatanov <tzz@lifelogs.com>]

On Fri, 17 Apr 2009 11:00:16 +0200 David Engster <deng@randomsample.de> wrote: 

DE> Ted Zlatanov <tzz@lifelogs.com> writes:
>> The registry could periodically visit all groups to see new messages.
>> But that's messy. It could also read a "dribble" file with the message
>> IDs and groups it needs to refresh.  The gnus.registry.eld file could
>> live on the server and get rsynced to the clients or opened remotely
>> through Tramp.

At least for IMAP, it's possible to do this kind of update without
Gnus-level calls.  It would probably be more efficient.  Hmm, maybe it
makes sense to open a second IMAP session just for registry and search
updates, and refresh it periodically.  That would be much better than
updating things synchronously in the main IMAP session.

DE> Well, having one general database for all search related things would
DE> surely be a nice thing to have. But searching is just one feature; what
DE> I really wanted to have in Gnus is something resembling "Filters" in
DE> Opera Mail. I remember a message from Kai Großjohann years ago where he
DE> praised Opera for this feature. It's an impressive mail reader, and
DE> using filters instead of splitting immediately made sense to me. And
DE> most importantly, filtering is not restricted to the mail headers. You
DE> can filter with a full text search, including MIME parsing. For example,
DE> Opera directly shows you groups containing all attachments you have
DE> received, filtered according to type.

DE> To do something like that in Gnus, you need an external indexer. Mairix
DE> is just one possibility, but it's well suited for Gnus because it
DE> directly creates the resulting mailbox, so you don't have to parse large
DE> amounts of output in Emacs Lisp. Alternatively, you could surely also
DE> use search engines like Lucene, Xapian, Sphinx, etc., but those would
DE> need much more work to interface them with Gnus.

I prefer splitting, but perhaps I'm old-fashioned.  Splitting works
locally and remotely, whereas indexers need to be run and maintained.

Also, splitting costs are paid once and splitting rules can be
complicated.  By contrast, filtering rules must be cheap, so they often
drop down to simple tags or a domain-specific query language (as with
nnir.el).

I don't know the right solution now, but I'll think about it (ETA 2020 :)

On Fri, 17 Apr 2009 12:07:49 +0200 Vegard Vesterheim <vegard.vesterheim@uninett.no> wrote: 

VV> I have great hopes for dbmail (dbmail.org). dbmail has an IMAP-backend
VV> for retrieving messages. Storing headers (and body) for mail messages
VV> into a database, opens possibilities for flexible and efficient
VV> queries and retrieval of email messages.

Considering the large amounts of people using IMAP on third-party
servers nowadays, I think it's best to find an approach that works with
any IMAP server.  I know this is unfair to solutions like dbmail, but
otherwise we end up with very limited solutions.  I think David agrees
with me from his followup.

On Fri, 17 Apr 2009 13:50:54 +0200 Vegard Vesterheim <vegard.vesterheim@uninett.no> wrote: 

VV> I typically use several different computers, so I prefer a
VV> server-side solution. One of my biggest gripes about Gnus is that
VV> some meta-information (.newsrc.eld etc.) is stored on the client
VV> instead of the server. 

I would also like that, and in fact that's where we'll end up if we
develop the facilities to have a global Gnus registry (since the newsrc
file can be serialized and synchronized exactly like the registry).

VV> This means that the count of unread messages typically is wrong when
VV> I move from one computer to another.

This actually could be avoided with IMAP, because it supports flags for
exactly this purpose.  I don't remember why the flags are not used in
preference to the local read/unread lists in newsrc.eld.

On Fri, 17 Apr 2009 15:27:36 +0200 David Engster <deng@randomsample.de> wrote: 

DE> One day there might even be IMAP storage in Tramp. :-)

Unfortunately, plain file storage would not be a good solution for
storing large data sets.  This is a problem with any
rsync/SSH/etc. file-level synchronization: speed, reliability, and
accessibility all suffer.

I think we need a way to serialize a hashtable like the Gnus registry to
an IMAP mailbox and make it searchable.  At least for the registry this
is easy, since every piece is self-sufficient in a single hashtable.
The Gnus newsrc.eld would be a bit harder to serialize.  This approach
essentially makes the IMAP server a weakly organized database server and
each mailbox a table.  I like it.

Magnus, have you considered starting work on the IMAP-based file
storage?  Does any of the above seem interesting or are you looking for
simple file persistence and nothing more?

Ted