Re-imagining Gnus as a mail reader

Re-imagining Gnus as a mail reader

[Lars Magne Ingebrigtsen]

It's so quiet here tonight.  I guess everybody is out partying.

Yesterday I chased down a performance regression that turned out to be
our IMAP server taking five seconds to re-index my spam group every time
I APPEND something to it.  It just took two hours.  So naturally I just
wanted to delete all the articles, but entering the group and deleting
all the millions of spam messages would have taken hours, so I just
wrote `M-x gnus-group-delete-articles' instead.

But now I'm kinda wondering how Gnus would have looked if it had
modelled itself on IMAP instead of NNTP.  IMAP is (sort of) a superset
of NNTP, but not quite.  Perhaps the main difference is that IMAP is
slightly more explicit about the trifurcation between
unread/read/non-existent, while NNTP only (sort of) has "some articles
exist in this range".

Hm...

But, I mean, also about the marks storage and subscription storage,
which would perhaps be more usefully modelled around IMAP and then
scaled down for other backends like NNTP.

Ok, back to the cooking wine.

Sent from my Emacs

Re: Re-imagining Gnus as a mail reader

[Ted Zlatanov]

On Sat, 18 Dec 2010 03:34:07 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> But now I'm kinda wondering how Gnus would have looked if it had
LMI> modelled itself on IMAP instead of NNTP.  IMAP is (sort of) a superset
LMI> of NNTP, but not quite.  Perhaps the main difference is that IMAP is
LMI> slightly more explicit about the trifurcation between
LMI> unread/read/non-existent, while NNTP only (sort of) has "some articles
LMI> exist in this range".

LMI> But, I mean, also about the marks storage and subscription storage,
LMI> which would perhaps be more usefully modelled around IMAP and then
LMI> scaled down for other backends like NNTP.

I think you mean you'd like gnus-sync.el to be more than data storage: a
search/modify facility that Gnus would rely on to set and get article
counts, marks, etc.  It would act like a general IMAP backend between
the specific backends like nnimap and nnml and the general Gnus
facilities like getting the list of articles and marks.  Yes, it
certainly could do that, and then it would be much more integrated with
Gnus than I was thinking.  You'd have to do a lot of work.

Ted

Re: Re-imagining Gnus as a mail reader

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 813 bytes --]

On Sat, Dec 18 2010, Ted Zlatanov wrote:

> LMI> But, I mean, also about the marks storage and subscription storage,
> LMI> which would perhaps be more usefully modelled around IMAP and then
> LMI> scaled down for other backends like NNTP.

Oh, yes!

> I think you mean you'd like gnus-sync.el to be more than data storage: a
> search/modify facility that Gnus would rely on to set and get article
> counts, marks, etc.  It would act like a general IMAP backend between
> the specific backends like nnimap and nnml and the general Gnus
> facilities like getting the list of articles and marks.  Yes, it
> certainly could do that, and then it would be much more integrated with
> Gnus than I was thinking.  You'd have to do a lot of work.

No.

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Re-imagining Gnus as a mail reader

[Ted Zlatanov]

On Sat, 18 Dec 2010 16:18:32 +0100 Julien Danjou <julien@danjou.info> wrote: 

JD> On Sat, Dec 18 2010, Ted Zlatanov wrote:
LMI> But, I mean, also about the marks storage and subscription storage,
LMI> which would perhaps be more usefully modelled around IMAP and then
LMI> scaled down for other backends like NNTP.

JD> Oh, yes!

>> I think you mean you'd like gnus-sync.el to be more than data storage: a
>> search/modify facility that Gnus would rely on to set and get article
>> counts, marks, etc.  It would act like a general IMAP backend between
>> the specific backends like nnimap and nnml and the general Gnus
>> facilities like getting the list of articles and marks.  Yes, it
>> certainly could do that, and then it would be much more integrated with
>> Gnus than I was thinking.  You'd have to do a lot of work.

JD> No.

I appreciate the depth and thoroughness of your response.

Ted

Re: Re-imagining Gnus as a mail reader

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I think you mean you'd like gnus-sync.el to be more than data storage: a
> search/modify facility that Gnus would rely on to set and get article
> counts, marks, etc.

Well, I wasn't really thinking about gnus-sync.el at all...  just
wondering whether a change of base model (NNTP->IMAP) for Gnus would
entail anything interesting...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Re-imagining Gnus as a mail reader

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

On Sat, Dec 18 2010, Ted Zlatanov wrote:

>>> I think you mean you'd like gnus-sync.el to be more than data storage: a
>>> search/modify facility that Gnus would rely on to set and get article
>>> counts, marks, etc.  It would act like a general IMAP backend between
>>> the specific backends like nnimap and nnml and the general Gnus
>>> facilities like getting the list of articles and marks.  Yes, it
>>> certainly could do that, and then it would be much more integrated with
>>> Gnus than I was thinking.  You'd have to do a lot of work.
>
> JD> No.
>
> I appreciate the depth and thoroughness of your response.

I'm sorry, but I misread you[1], and after re-reading your message a dozen
of times I just understand what you really meant.

So actually, I think your idea could be very good if that's done
correctly. `gnus-sync' would be a no-op in the case of nnimap + a good
and usual IMAP server[2], but would be a file storage facility for other
back-ends where the server does not provide marks/subscriptions/whatever
storage.

(Not sure gnus-sync is a good name since I don't think it would be the
sync part itself like it is currently.)

Once again, my apologies for being a bit rude, Ted. :)

[1]  Not because you wrote badly, but probably because I'm not a native
     English speaker and your sentence sounded another way in my head,
     plus that I merged that with what you wrote previously about
     gnus-sync.
[2]  It could still be useful for IMAP anonymous access, à la NNTP.

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Re-imagining Gnus as a mail reader

[Ted Zlatanov]

On Sat, 18 Dec 2010 19:47:47 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I think you mean you'd like gnus-sync.el to be more than data storage: a
>> search/modify facility that Gnus would rely on to set and get article
>> counts, marks, etc.

LMI> Well, I wasn't really thinking about gnus-sync.el at all...  just
LMI> wondering whether a change of base model (NNTP->IMAP) for Gnus would
LMI> entail anything interesting...

(I'm trying to use the "state" vs. "configuration" distinction we've all
been making, consciously or not.)

Probably it wouldn't be called gnus-sync.el anyhow.  Yes, I think it's
both interesting and useful.  The Gnus UI layer would talk to a generic
IMAP-like layer which would map commands directly for nnimap, but would
use gnus-sync.el or something like it to map state for nntp and similar
less capable backends.

So... gnus-state.el?  And then each nn* instance would declare "I
implement my own state management for X" or "I need help managing my
state for X."  X is any one of: splitting rules, marks, subscriptions,
etc.

The gnus-sync.el code would then be a way to merge the Gnus *state*
between machines.  The data-sync.el code would be for generic data
merging through a remote or local backend.  The user would choose to use
data-sync.el to synchronize their *configuration* but Gnus shouldn't
care.

It's all starting to make sense to me.  Which should worry everyone else.

On Sat, 18 Dec 2010 22:38:34 +0100 Julien Danjou <julien@danjou.info> wrote: 

JD> I'm sorry, but I misread you[1], and after re-reading your message a dozen
JD> of times I just understand what you really meant.

JD> So actually, I think your idea could be very good if that's done
JD> correctly. `gnus-sync' would be a no-op in the case of nnimap + a good
JD> and usual IMAP server[2], but would be a file storage facility for other
JD> back-ends where the server does not provide marks/subscriptions/whatever
JD> storage.

Yes.  The name was misleading; the goal is not to synchronize data but
to store it in a neutral way.  But even IMAP can't handle everything
Gnus needs, so each nn* backend *instance* has to declare what state it
needs stored externally and what it can store internally.  I say
instance because, as you mentioned, anonymous IMAP is less capable so
such a server would need more help storing state.

Marks are an obvious example IMAP can handle.  But splitting rules and
subscriptions can't be stored by IMAP (although it could be interesting
to synchronize some of the splitting rules with SIEVE).

JD> (Not sure gnus-sync is a good name since I don't think it would be the
JD> sync part itself like it is currently.)

JD> Once again, my apologies for being a bit rude, Ted. :)

Sorry for confusing the issue, and I didn't think you were rude.  Just
too brief :)

Ted

Re: Re-imagining Gnus as a mail reader

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> But now I'm kinda wondering how Gnus would have looked if it had
> modelled itself on IMAP instead of NNTP.  IMAP is (sort of) a superset
> of NNTP, but not quite.  Perhaps the main difference is that IMAP is
> slightly more explicit about the trifurcation between
> unread/read/non-existent, while NNTP only (sort of) has "some articles
> exist in this range".

One more thing to think about if re-imagining the backend interface
around IMAP: MIME awareness (specifically accessing parts of a message)

And a reimaginging should probably consider agent-like functionality
from the start, rather than as an afterthought.

Desirable agent-ish functionality:
 - Offline reading and responding/forwarding/resend (setting marks
   "upstream" when going offline)
 - Offline delete/copy/move (copy and move should work across servers,
   but it should be able to do them cheaply when eg. moving on an IMAP
   server)
 - Offline expiry
 - Agent synced with upstream changes when going online (could be an
   awful lot of talking when the agent has many articles, the server has
   many articles etc.)
 - MIME awareness (partial dowloads)
 - Minimize network traffic (ideally each message part should only cross
   the wire once, cross server copies/moves excepted)

Another point of consideration: should the agent be another backend like
the others?  Could being an agent be a property of the backend
interface?  Ie. any backend could be an agent for any other (and you
could eg. use an IMAP server as an agent for NNTP if that's your fancy)?

If so, the agent could just be good old nnml, or whatever is the most
efficient local mail backend these days...?

Possible ways of wiring the backend and the agent together:
 - Letting the actual backend own the agent backend and use it for local
   storage 
 - Letting the agent be in front of the actual backend.  Ie. the user
   will always be talking to the agent

I think if I was implementing it, I would prefer the latter alternative,
based on nothing more scentific than "it feels right".