mail group by author?

mail group by author?

[Stephen Berman]

Is it possible to make a temporary group consisting of mails by a
specific person whose mail articles may be scattered across many
different groups?  I would like to do this to this to be able to see all
mail by given people, regardless of which groups the individual mails
are stored in.

Steve Berman

Re: mail group by author?

[David Engster]

Stephen Berman <stephen.berman@gmx.net> writes:
> Is it possible to make a temporary group consisting of mails by a
> specific person whose mail articles may be scattered across many
> different groups?  I would like to do this to this to be able to see all
> mail by given people, regardless of which groups the individual mails
> are stored in.

nnkiboze should be able to do this, but I for one never got it to work
with my mail groups. It is also pretty slow.

If you store your mails locally, you can set up mairix to index them and
use the nnmairix back end to create search groups like you describe.

There is also the nnir search back end which works with several
different search engines, including the IMAP search features, so if
you're using nnimap this might be the easiest solution (nnmairix can be
used via ssh, but that's a bit cumbersome to set up).

Maybe the Gnus registry could be used for creating such temporary
groups, but as far as I know there's nothing implemented yet that does
what you want right away.

-David

Re: mail group by author?

[Tassilo Horn]

David Engster <deng@randomsample.de> writes:

Hi David,

> There is also the nnir search back end which works with several
> different search engines, including the IMAP search features, so if
> you're using nnimap this might be the easiest solution (nnmairix can
> be used via ssh, but that's a bit cumbersome to set up).

nnir seaches only in the group point is on.

Bye,
Tassilo
-- 
Whenever Richard Stallman points at a Windows computer, it segfaults.

Re: mail group by author?

[David Engster]

Tassilo Horn <tassilo@member.fsf.org> writes:
> David Engster <deng@randomsample.de> writes:
>> There is also the nnir search back end which works with several
>> different search engines, including the IMAP search features, so if
>> you're using nnimap this might be the easiest solution (nnmairix can
>> be used via ssh, but that's a bit cumbersome to set up).
>
> nnir seaches only in the group point is on.

Oh. Thanks for the correction. Do you happen to know if this is a
principal restriction in nnir, or only when using the IMAP search back
end?

-David

Re: mail group by author?

[Tassilo Horn]

David Engster <deng@randomsample.de> writes:

Hi David,

>>> There is also the nnir search back end which works with several
>>> different search engines, including the IMAP search features, so if
>>> you're using nnimap this might be the easiest solution (nnmairix can
>>> be used via ssh, but that's a bit cumbersome to set up).
>>
>> nnir seaches only in the group point is on.
>
> Oh. Thanks for the correction. Do you happen to know if this is a
> principal restriction in nnir, or only when using the IMAP search back
> end?

I don't know, I use it only for searching nnimap groups.

But I think about switching to nnmairix anyway:

  - It's much faster (the imap search on my local dovecot server seems
    to do simple grepping, no indexing involved)

  - I get strange results for `G G Tassilo Horn RET'.  It seems to
    find all mails in the group, although most of them don't contain my
    name, neither in any header nor in the body.

    If I restrict the search to some field using `C-u G G <query> RET
    <field>' it seems to work, though...

Bye,
Tassilo
-- 
My opinions may have changed, but not the fact that I am right.

Re: mail group by author?

[Andreas Schwab]

Tassilo Horn <tassilo@member.fsf.org> writes:

> David Engster <deng@randomsample.de> writes:
>
> Hi David,
>
>> There is also the nnir search back end which works with several
>> different search engines, including the IMAP search features, so if
>> you're using nnimap this might be the easiest solution (nnmairix can
>> be used via ssh, but that's a bit cumbersome to set up).
>
> nnir seaches only in the group point is on.

The last time I used it (with namazu) it searched whatever the search
index covers.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: mail group by author?

[Tassilo Horn]

Andreas Schwab <schwab@linux-m68k.org> writes:

Hi Andreas,

>>> There is also the nnir search back end which works with several
>>> different search engines, including the IMAP search features, so if
>>> you're using nnimap this might be the easiest solution (nnmairix can
>>> be used via ssh, but that's a bit cumbersome to set up).
>>
>> nnir seaches only in the group point is on.
>
> The last time I used it (with namazu) it searched whatever the search
> index covers.

Yes, this may be different for indexing search engines.  But for nnimap
it seems to be local for a group.  (But maybe this can be changed with
some configuration bits.)

Bye,
Tassilo

Re: mail group by author?

[David Engster]

Tassilo Horn <tassilo@member.fsf.org> writes:
> Andreas Schwab <schwab@linux-m68k.org> writes:
>
> Hi Andreas,
>
>>>> There is also the nnir search back end which works with several
>>>> different search engines, including the IMAP search features, so if
>>>> you're using nnimap this might be the easiest solution (nnmairix can
>>>> be used via ssh, but that's a bit cumbersome to set up).
>>>
>>> nnir seaches only in the group point is on.
>>
>> The last time I used it (with namazu) it searched whatever the search
>> index covers.
>
> Yes, this may be different for indexing search engines.  But for nnimap
> it seems to be local for a group.  (But maybe this can be changed with
> some configuration bits.)

It seems according to IMAPrev4 the SEARCH command only works in the
currently selected mailbox, so this is primarily a restriction in the
IMAP protocol. Of course, one could extend nnir to select several
mailboxes, do a SEARCH in each of them and then combine the results.

BTW, for people using Dovecot: version 1.2 (still in beta) has a new
"virtual mailbox" extension which looks pretty interesting for this kind
of stuff:

http://wiki.dovecot.org/Plugins/Virtual

-David

Dovecot 1.2 virtual mailboxes (was: mail group by author?)

[Tassilo Horn]

David Engster <deng@randomsample.de> writes:

Hi David,

> BTW, for people using Dovecot: version 1.2 (still in beta) has a new
> "virtual mailbox" extension which looks pretty interesting for this
> kind of stuff:
>
> http://wiki.dovecot.org/Plugins/Virtual

Looks quite similar to kiboze groups, right?

I imagine that I could benefit from that feature by creating virtual
groups for my most important contacts, but I guess it would conflict
with things like `gcc-self', i.e. when I reply to a mail in a virtual
group I want to gcc the original group, not the virtual one.

But probably the registry could jump in here (at least if I've read the
message in the original group before and omit virtual groups from
registration).

Bye,
Tassilo

Re: Dovecot 1.2 virtual mailboxes

[David Engster]

Tassilo Horn <tassilo@member.fsf.org> writes:
> David Engster <deng@randomsample.de> writes:
>> BTW, for people using Dovecot: version 1.2 (still in beta) has a new
>> "virtual mailbox" extension which looks pretty interesting for this
>> kind of stuff:
>>
>> http://wiki.dovecot.org/Plugins/Virtual
>
> Looks quite similar to kiboze groups, right?

It seems so. I didn't try it yet, though.

> I imagine that I could benefit from that feature by creating virtual
> groups for my most important contacts, but I guess it would conflict
> with things like `gcc-self', i.e. when I reply to a mail in a virtual
> group I want to gcc the original group, not the virtual one.
>
> But probably the registry could jump in here (at least if I've read the
> message in the original group before and omit virtual groups from
> registration).

Yes, I guess this could be scripted somehow, since the virtual groups
are in their own namespace anyway. But indeed, the registry would have
to see the original message in the original group beforehand. I already
struggled with that problem with nnmairix, and don't yet know how to
solve that efficiently, especially since I'm splitting with procmail on
the server and not with Gnus.

-David

Re: Dovecot 1.2 virtual mailboxes

[Tassilo Horn]

David Engster <deng@randomsample.de> writes:

Hi David,

>> But probably the registry could jump in here (at least if I've read
>> the message in the original group before and omit virtual groups from
>> registration).
>
> Yes, I guess this could be scripted somehow, since the virtual groups
> are in their own namespace anyway. But indeed, the registry would have
> to see the original message in the original group beforehand. I
> already struggled with that problem with nnmairix, and don't yet know
> how to solve that efficiently, especially since I'm splitting with
> procmail on the server and not with Gnus.

Yes, I do so with SIEVE, too + some very restricted local fancy
splitting with gnus-registry-split-fancy-with-parent.  Therefore it
would be nice to do the registration of messages not on reading them but
somewhere after fetching and splitting.

Bye,
Tassilo

Re: mail group by author?

[Vegard Vesterheim]

On Tue, 31 Mar 2009 08:42:59 +0200 Tassilo Horn <tassilo@member.fsf.org> wrote:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
> Hi Andreas,
>
>>>> There is also the nnir search back end which works with several
>>>> different search engines, including the IMAP search features, so if
>>>> you're using nnimap this might be the easiest solution (nnmairix can
>>>> be used via ssh, but that's a bit cumbersome to set up).
>>>
>>> nnir seaches only in the group point is on.
>>
>> The last time I used it (with namazu) it searched whatever the search
>> index covers.
>
> Yes, this may be different for indexing search engines.  But for nnimap
> it seems to be local for a group.  (But maybe this can be changed with
> some configuration bits.)

It is possible to mark (gnus-topic-mark-topic) several groups, and issue a
nnimap search (gnus-group-make-nnir-group) across all marked groups.

 - Vegard V -

Re: mail group by author?

[Tassilo Horn]

Vegard Vesterheim <vegard.vesterheim@uninett.no> writes:

Hi Vegard,

>>>> nnir seaches only in the group point is on.
>>>
>>> The last time I used it (with namazu) it searched whatever the
>>> search index covers.
>>
>> Yes, this may be different for indexing search engines.  But for
>> nnimap it seems to be local for a group.  (But maybe this can be
>> changed with some configuration bits.)
>
> It is possible to mark (gnus-topic-mark-topic) several groups, and
> issue a nnimap search (gnus-group-make-nnir-group) across all marked
> groups.

Hey, excellent!  Thanks for the tip.

Bye,
Tassilo
-- 
     My software never has bugs. It just develops random features.

Re: mail group by author?

[Ted Zlatanov]

On Mon, 30 Mar 2009 13:57:02 +0200 David Engster <deng@randomsample.de> wrote: 

DE> Stephen Berman <stephen.berman@gmx.net> writes:
>> Is it possible to make a temporary group consisting of mails by a
>> specific person whose mail articles may be scattered across many
>> different groups?  I would like to do this to this to be able to see all
>> mail by given people, regardless of which groups the individual mails
>> are stored in.

DE> Maybe the Gnus registry could be used for creating such temporary
DE> groups, but as far as I know there's nothing implemented yet that does
DE> what you want right away.

This is really easy to implement as far as locating the message IDs and
article groups+numbers.  The hard part is to make all those articles
into one group, I don't know how to do that.  But I can write the first
part if you need it.

On Tue, 31 Mar 2009 13:35:10 +0200 David Engster <deng@randomsample.de> wrote: 

DE> Tassilo Horn <tassilo@member.fsf.org> writes:
>> But probably the registry could jump in here (at least if I've read the
>> message in the original group before and omit virtual groups from
>> registration).

DE> Yes, I guess this could be scripted somehow, since the virtual
DE> groups are in their own namespace anyway. But indeed, the registry
DE> would have to see the original message in the original group
DE> beforehand. I already struggled with that problem with nnmairix, and
DE> don't yet know how to solve that efficiently, especially since I'm
DE> splitting with procmail on the server and not with Gnus.

The registry could periodically visit all groups to see new messages.
But that's messy.  It could also read a "dribble" file with the message
IDs and groups it needs to refresh.  The gnus.registry.eld file could
live on the server and get rsynced to the clients or opened remotely
through Tramp.

Since the registry is pure data storage, it is not hard to serialize it
to and from a database.  Registry queries currently assume the data
structure is local, but that can be easily modified to query the remote.

A remote DB may in fact be the best option, either something relaxed
like CouchDB or SimpleDB, or something more constrained like a standard
RDBMS schema.  A benefit of CouchDB or SimpleDB is that users could make
their registry publically readable, so I'm in favor of that.  Tying
mairix and other search backends into the same DB is also a good idea.

On Tue, 31 Mar 2009 14:23:19 +0200 Tassilo Horn <tassilo@member.fsf.org> wrote: 

TH> Yes, I do so with SIEVE, too + some very restricted local fancy
TH> splitting with gnus-registry-split-fancy-with-parent.  Therefore it
TH> would be nice to do the registration of messages not on reading them but
TH> somewhere after fetching and splitting.

You mean after SIEVE splitting?  I don't know much about SIEVE on the
server, can you hook scripts to a step?  If yes it may be possible to
tie it into the above database discussion.

Ted

Re: mail group by author?

[David Engster]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Mon, 30 Mar 2009 13:57:02 +0200 David Engster <deng@randomsample.de> wrote: 
>
> DE> Stephen Berman <stephen.berman@gmx.net> writes:
>>> Is it possible to make a temporary group consisting of mails by a
>>> specific person whose mail articles may be scattered across many
>>> different groups?  I would like to do this to this to be able to see all
>>> mail by given people, regardless of which groups the individual mails
>>> are stored in.
>
> DE> Maybe the Gnus registry could be used for creating such temporary
> DE> groups, but as far as I know there's nothing implemented yet that does
> DE> what you want right away.
>
> This is really easy to implement as far as locating the message IDs and
> article groups+numbers. 

Yes, I tested that some time ago using maphash, and it was pretty
fast. My idea was to extend nnmairix to also build groups containing all
messages which have a certain label. I still plan on doing that when the
feature freeze is over.

> The hard part is to make all those articles into one group, I don't
> know how to do that.  But I can write the first part if you need it.

Making these kinds of "virtual" groups needs a back end which implements
the Gnus API functions. This is already implemented in nnir, so I think
extending nnir to also be able to query the registry shouldn't be
difficult to do.

> On Tue, 31 Mar 2009 13:35:10 +0200 David Engster <deng@randomsample.de> wrote: 
> DE> Tassilo Horn <tassilo@member.fsf.org> writes:
>>> But probably the registry could jump in here (at least if I've read the
>>> message in the original group before and omit virtual groups from
>>> registration).
>
> DE> Yes, I guess this could be scripted somehow, since the virtual
> DE> groups are in their own namespace anyway. But indeed, the registry
> DE> would have to see the original message in the original group
> DE> beforehand. I already struggled with that problem with nnmairix, and
> DE> don't yet know how to solve that efficiently, especially since I'm
> DE> splitting with procmail on the server and not with Gnus.
>
> The registry could periodically visit all groups to see new messages.
> But that's messy. It could also read a "dribble" file with the message
> IDs and groups it needs to refresh.  The gnus.registry.eld file could
> live on the server and get rsynced to the clients or opened remotely
> through Tramp.

Visiting all groups for new messages surely isn't nice, but I'd still
prefer it because it's self-contained. One could call this in the
gnus-after-getting-new-news-hook.

Using external dribble files or scripts is probably faster and more
elegant, but also more difficult to set up.

> Since the registry is pure data storage, it is not hard to serialize it
> to and from a database.  Registry queries currently assume the data
> structure is local, but that can be easily modified to query the remote.
>
> A remote DB may in fact be the best option, either something relaxed
> like CouchDB or SimpleDB, or something more constrained like a standard
> RDBMS schema.  A benefit of CouchDB or SimpleDB is that users could make
> their registry publically readable, so I'm in favor of that.  Tying
> mairix and other search backends into the same DB is also a good idea.

Well, having one general database for all search related things would
surely be a nice thing to have. But searching is just one feature; what
I really wanted to have in Gnus is something resembling "Filters" in
Opera Mail. I remember a message from Kai Großjohann years ago where he
praised Opera for this feature. It's an impressive mail reader, and
using filters instead of splitting immediately made sense to me. And
most importantly, filtering is not restricted to the mail headers. You
can filter with a full text search, including MIME parsing. For example,
Opera directly shows you groups containing all attachments you have
received, filtered according to type.

To do something like that in Gnus, you need an external indexer. Mairix
is just one possibility, but it's well suited for Gnus because it
directly creates the resulting mailbox, so you don't have to parse large
amounts of output in Emacs Lisp. Alternatively, you could surely also
use search engines like Lucene, Xapian, Sphinx, etc., but those would
need much more work to interface them with Gnus.

Regards,
David

Re: mail group by author?

[Vegard Vesterheim]

On Fri, 17 Apr 2009 11:00:16 +0200 David Engster <deng@randomsample.de> wrote:

> To do something like that in Gnus, you need an external indexer. Mairix
> is just one possibility, but it's well suited for Gnus because it
> directly creates the resulting mailbox, so you don't have to parse large
> amounts of output in Emacs Lisp. Alternatively, you could surely also
> use search engines like Lucene, Xapian, Sphinx, etc., but those would
> need much more work to interface them with Gnus.

I have great hopes for dbmail (dbmail.org). dbmail has an IMAP-backend
for retrieving messages. Storing headers (and body) for mail messages
into a database, opens possibilities for flexible and efficient
queries and retrieval of email messages.

One of the missing parts seems to be support for Virtual Folders:
http://dbmail.org/dokuwiki/doku.php?id=v-folders
http://tools.ietf.org/id/draft-ietf-lemonade-vfolder-01.txt

Has anyone tried the combination dbmail/GNUS?

 - Vegard V -

 

Re: mail group by author?

[David Engster]

Vegard Vesterheim <vegard.vesterheim@uninett.no> writes:
> On Fri, 17 Apr 2009 11:00:16 +0200 David Engster <deng@randomsample.de> wrote:
>
>> To do something like that in Gnus, you need an external indexer. Mairix
>> is just one possibility, but it's well suited for Gnus because it
>> directly creates the resulting mailbox, so you don't have to parse large
>> amounts of output in Emacs Lisp. Alternatively, you could surely also
>> use search engines like Lucene, Xapian, Sphinx, etc., but those would
>> need much more work to interface them with Gnus.
>
> I have great hopes for dbmail (dbmail.org). dbmail has an IMAP-backend
> for retrieving messages. Storing headers (and body) for mail messages
> into a database, opens possibilities for flexible and efficient
> queries and retrieval of email messages.

Thanks for that link, I didn't know of dbmail. This looks pretty similar
to what archiveopteryx does (www.archiveopteryx.org), which however has
support for virtual folders (they call it "views").

Both seem a bit oversized for a single user though, and I guess you
would need your own server to easily access your mail from different
machines? Also, I'm really happy with Dovecot, and there would have to
be some serious advantages to make me switch to something else. And
there are many people who happily use commercial IMAP services
(Fastmail, Google, whatever), which should also be able to have fast
full text search.

What I'd really love to see is a client-only solution, e.g. a program
which creates a *local* index from a *remote* IMAP server.

-David

Re: mail group by author?

[Vegard Vesterheim]

On Fri, 17 Apr 2009 12:46:09 +0200 David Engster <deng@randomsample.de> wrote:

> Vegard Vesterheim <vegard.vesterheim@uninett.no> writes:

>> I have great hopes for dbmail (dbmail.org). dbmail has an IMAP-backend
>> for retrieving messages. Storing headers (and body) for mail messages
>> into a database, opens possibilities for flexible and efficient
>> queries and retrieval of email messages.
>
> Thanks for that link, I didn't know of dbmail. This looks pretty similar
> to what archiveopteryx does (www.archiveopteryx.org), which however has
> support for virtual folders (they call it "views").

Heh, thanks for that link, I didn't know of archiveopteryx ;-)

It does indeed look similar to dbmail, in fact there is a special FAQ
entry that compares the two:
  http://www.archiveopteryx.org/faq/mailstore#dbmail

I'm intrigued about the possibility for virtual folders in
archiveopteryx, I hope to get a chance to try this soon.

> Both seem a bit oversized for a single user though, and I guess you
> would need your own server to easily access your mail from different
> machines? Also, I'm really happy with Dovecot, and there would have to
> be some serious advantages to make me switch to something else. And
> there are many people who happily use commercial IMAP services
> (Fastmail, Google, whatever), which should also be able to have fast
> full text search.
>
> What I'd really love to see is a client-only solution, e.g. a program
> which creates a *local* index from a *remote* IMAP server.

I typically use several different computers, so I prefer a server-side
solution. One of my biggest gripes about Gnus is that some
meta-information (.newsrc.eld etc.) is stored on the client instead of
the server. This means that the count of unread messages typically is
wrong when I move from one computer to another.

Yes, I know about solutions like this:
http://www.emacswiki.org/emacs/GnusSync

 - Vegard V -

Syncing Gnus (was: mail group by author?)

[David Engster]

Vegard Vesterheim <vegard.vesterheim@uninett.no> writes:
> I'm intrigued about the possibility for virtual folders in
> archiveopteryx, I hope to get a chance to try this soon.

Please let me know how this works out for you.

> I typically use several different computers, so I prefer a server-side
> solution. One of my biggest gripes about Gnus is that some
> meta-information (.newsrc.eld etc.) is stored on the client instead of
> the server. This means that the count of unread messages typically is
> wrong when I move from one computer to another.
>
> Yes, I know about solutions like this:
> http://www.emacswiki.org/emacs/GnusSync

Maybe the following is helpful:

- Tramp:

In your .emacs set:

(require 'gnus-load)
(require 'message)
(setq gnus-startup-file "/scpx:login@server:.newsrc")
(setq message-directory "/scpx:login@server:Mail/")
(setq gnus-directory "/scpx:login@server:News/")

Disclaimer: I've only tested this very shortly, so I'm not sure if this
is working correctly.

To speed this up, set up a ssh master connection beforehand (e.g. do
"ssh -f -N -M login@server" in the shell and specify ControlMaster for
the host in your .ssh/config). Otherwise, it will not only be dead slow,
but might also raise some administrator's eyebrows.

One day there might even be IMAP storage in Tramp. :-)

- sshfs:

http://fuse.sourceforge.net/sshfs.html

Mount single directory using sshfs and again set gnus-startup-file,
message-directory and gnus-directory accordingly. Never tried this, but
I don't see why this shouldn't just work.

Needless to say, both these methods are slower than just using rsync,
which is why I prefer the latter.

-David

Re: mail group by author?

[Ted Zlatanov]

On Fri, 17 Apr 2009 11:00:16 +0200 David Engster <deng@randomsample.de> wrote: 

DE> Ted Zlatanov <tzz@lifelogs.com> writes:
>> The registry could periodically visit all groups to see new messages.
>> But that's messy. It could also read a "dribble" file with the message
>> IDs and groups it needs to refresh.  The gnus.registry.eld file could
>> live on the server and get rsynced to the clients or opened remotely
>> through Tramp.

At least for IMAP, it's possible to do this kind of update without
Gnus-level calls.  It would probably be more efficient.  Hmm, maybe it
makes sense to open a second IMAP session just for registry and search
updates, and refresh it periodically.  That would be much better than
updating things synchronously in the main IMAP session.

DE> Well, having one general database for all search related things would
DE> surely be a nice thing to have. But searching is just one feature; what
DE> I really wanted to have in Gnus is something resembling "Filters" in
DE> Opera Mail. I remember a message from Kai Großjohann years ago where he
DE> praised Opera for this feature. It's an impressive mail reader, and
DE> using filters instead of splitting immediately made sense to me. And
DE> most importantly, filtering is not restricted to the mail headers. You
DE> can filter with a full text search, including MIME parsing. For example,
DE> Opera directly shows you groups containing all attachments you have
DE> received, filtered according to type.

DE> To do something like that in Gnus, you need an external indexer. Mairix
DE> is just one possibility, but it's well suited for Gnus because it
DE> directly creates the resulting mailbox, so you don't have to parse large
DE> amounts of output in Emacs Lisp. Alternatively, you could surely also
DE> use search engines like Lucene, Xapian, Sphinx, etc., but those would
DE> need much more work to interface them with Gnus.

I prefer splitting, but perhaps I'm old-fashioned.  Splitting works
locally and remotely, whereas indexers need to be run and maintained.

Also, splitting costs are paid once and splitting rules can be
complicated.  By contrast, filtering rules must be cheap, so they often
drop down to simple tags or a domain-specific query language (as with
nnir.el).

I don't know the right solution now, but I'll think about it (ETA 2020 :)

On Fri, 17 Apr 2009 12:07:49 +0200 Vegard Vesterheim <vegard.vesterheim@uninett.no> wrote: 

VV> I have great hopes for dbmail (dbmail.org). dbmail has an IMAP-backend
VV> for retrieving messages. Storing headers (and body) for mail messages
VV> into a database, opens possibilities for flexible and efficient
VV> queries and retrieval of email messages.

Considering the large amounts of people using IMAP on third-party
servers nowadays, I think it's best to find an approach that works with
any IMAP server.  I know this is unfair to solutions like dbmail, but
otherwise we end up with very limited solutions.  I think David agrees
with me from his followup.

On Fri, 17 Apr 2009 13:50:54 +0200 Vegard Vesterheim <vegard.vesterheim@uninett.no> wrote: 

VV> I typically use several different computers, so I prefer a
VV> server-side solution. One of my biggest gripes about Gnus is that
VV> some meta-information (.newsrc.eld etc.) is stored on the client
VV> instead of the server. 

I would also like that, and in fact that's where we'll end up if we
develop the facilities to have a global Gnus registry (since the newsrc
file can be serialized and synchronized exactly like the registry).

VV> This means that the count of unread messages typically is wrong when
VV> I move from one computer to another.

This actually could be avoided with IMAP, because it supports flags for
exactly this purpose.  I don't remember why the flags are not used in
preference to the local read/unread lists in newsrc.eld.

On Fri, 17 Apr 2009 15:27:36 +0200 David Engster <deng@randomsample.de> wrote: 

DE> One day there might even be IMAP storage in Tramp. :-)

Unfortunately, plain file storage would not be a good solution for
storing large data sets.  This is a problem with any
rsync/SSH/etc. file-level synchronization: speed, reliability, and
accessibility all suffer.

I think we need a way to serialize a hashtable like the Gnus registry to
an IMAP mailbox and make it searchable.  At least for the registry this
is easy, since every piece is self-sufficient in a single hashtable.
The Gnus newsrc.eld would be a bit harder to serialize.  This approach
essentially makes the IMAP server a weakly organized database server and
each mailbox a table.  I like it.

Magnus, have you considered starting work on the IMAP-based file
storage?  Does any of the above seem interesting or are you looking for
simple file persistence and nothing more?

Ted

Re: mail group by author?

[Vegard Vesterheim]

On Fri, 17 Apr 2009 11:53:01 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote:

> On Fri, 17 Apr 2009 12:07:49 +0200 Vegard Vesterheim <vegard.vesterheim@uninett.no> wrote: 
>
> VV> I have great hopes for dbmail (dbmail.org). dbmail has an IMAP-backend
> VV> for retrieving messages. Storing headers (and body) for mail messages
> VV> into a database, opens possibilities for flexible and efficient
> VV> queries and retrieval of email messages.
>
> Considering the large amounts of people using IMAP on third-party
> servers nowadays, I think it's best to find an approach that works with
> any IMAP server.  I know this is unfair to solutions like dbmail, but
> otherwise we end up with very limited solutions.  I think David agrees
> with me from his followup.

But dbmail *is* a IMAP server, I do not necessarily suggest any
special support for dbmail in Gnus. Having a proper database as
storage with search indexes for headers, instead of relying on the
filesystem, will hopefully result in a IMAP server with more flexible
and efficienct query and retrieval functionality. Support for virtual
folders (predefined searches), is more easily accomplished with an
infrastructure like this.

Presorting of mail into statically defined folders is IMHO the wrong
approach. I have wasted much time searching for a specific email, only
to discover it was filtered into another folder than I had
thought. With virtual folders, the same message can be found in
several folders, but will still be stored only once, since the folder
is really a query on the IMAP server. Sitewide de-duplication of
messages would be an added benefit of this approach.

For instance, I would like to have a folder called 'People' with
automatically generated subfolders for each person having sent, or
recieved, an email to/from me. I would also like to have the
possibilities to define folders with names (and content) like:
'Messages_recieved_this_month', 'Messages_I_sent_yesterday' etc.

 - Vegard V -

Re: mail group by author?

[Ted Zlatanov]

On Fri, 17 Apr 2009 22:15:33 +0200 Vegard Vesterheim <vegard.vesterheim@uninett.no> wrote: 

VV> On Fri, 17 Apr 2009 11:53:01 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote:
>> On Fri, 17 Apr 2009 12:07:49 +0200 Vegard Vesterheim <vegard.vesterheim@uninett.no> wrote: 
>> 
VV> I have great hopes for dbmail (dbmail.org). dbmail has an IMAP-backend
VV> for retrieving messages. Storing headers (and body) for mail messages
VV> into a database, opens possibilities for flexible and efficient
VV> queries and retrieval of email messages.
>> 
>> Considering the large amounts of people using IMAP on third-party
>> servers nowadays, I think it's best to find an approach that works with
>> any IMAP server.  I know this is unfair to solutions like dbmail, but
>> otherwise we end up with very limited solutions.  I think David agrees
>> with me from his followup.

VV> But dbmail *is* a IMAP server, I do not necessarily suggest any
VV> special support for dbmail in Gnus. Having a proper database as
VV> storage with search indexes for headers, instead of relying on the
VV> filesystem, will hopefully result in a IMAP server with more flexible
VV> and efficienct query and retrieval functionality. Support for virtual
VV> folders (predefined searches), is more easily accomplished with an
VV> infrastructure like this.

Sorry, I thought you were implying we should have special support for
dbmail's queries (instead of just using the regular IMAP commands).  But
the IMAP client needs at least some awareness of dbmail's abilities, no?

VV> Presorting of mail into statically defined folders is IMHO the wrong
VV> approach. I have wasted much time searching for a specific email, only
VV> to discover it was filtered into another folder than I had
VV> thought. With virtual folders, the same message can be found in
VV> several folders, but will still be stored only once, since the folder
VV> is really a query on the IMAP server. 

Managing large collections of data is expensive.  No matter how good the
database, it's slower to manage all the messages in one set as opposed
to partitioning them in some way.  In effect, "static" mailboxes are
really partitions of the full message set, and the server's storage
backend (Maildir+ or some other mechanism) handles the partitioning
transparently.

VV> Sitewide de-duplication of messages would be an added benefit of
VV> this approach.

I don't think the savings there are significant.

VV> For instance, I would like to have a folder called 'People' with
VV> automatically generated subfolders for each person having sent, or
VV> recieved, an email to/from me. I would also like to have the
VV> possibilities to define folders with names (and content) like:
VV> 'Messages_recieved_this_month', 'Messages_I_sent_yesterday' etc.

This can be done with the Gnus registry currently.  What I'm slowly
realizing is that the registry needs to be integrated deeper with Gnus
as a sort of "mail crawler" instead of an opportunistic voyeur that can
miss mail.  In addition, the registry data needs to be storeable
somewhere outside the local filesystem.

Ted