Re: gnus-sync.el v2

[Ted Zlatanov <tzz@lifelogs.com>]

On Sun, 02 Dec 2012 11:17:10 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> I do (really really) want to rewrite gnus-sync.el to use native IMAP
TZ> commands instead of an external CouchDB installation, and at that time
TZ> will settle on a new, more robust format for flag and general data
TZ> storage.  I was supposed to get to that last month but my work
TZ> commitment was extended by a few weeks; the rewrite is at the top of my
TZ> TODO list.

TZ> I also asked Lars for help with the IMAP side but he hasn't been
TZ> available.  If you or anyone else wants to help, I need generic
TZ> functions that, given an IMAP mailbox, a string key, and an alist, can
TZ> store the alist under the key or delete the key (so basically I want a
TZ> hashtable implemented in an IMAP mailbox).  Writing those functions will
TZ> probably be 80-90% of the time I spend on the rewrite, so any help is
TZ> appreciated.  I used to have some understanding of imap.el, but the new
TZ> nnimap.el is a super wonderful mystery to me.

I spent time prototyping and thinking about this.

I still think the right approach is to use IMAP and nnimap.el for
recording the data, as Lars suggested originally.

I have realized that the whole gnus-sync.el structure is unnecessary.
Gnus already has synchronization built-in.  The dribble file is perfect
for synchronizing the data, and all the Gnus internals already write
group info and topics to the dribble file.

To implement a new IMAP-synchronized mode, we need to make every Gnus
instance behave like a slave Gnus, always consulting the dribble IMAP
group and saving back to it.

The configuration will be to simply set the dribble filename to an IMAP
folder and to set some gnus-sync variable to indicate perpetual slavery.

This has to be done in profiles, so you can have multiple connectivity
profiles but still share the marks and other information.

The format for the IMAP dribble group could be a series of messages in
the following formats:

1. set group info
subject = "gnus-group-set-info GROUPNAME"
from = PROFILE
body = (gnus-group-set-info ...)

2. set topology
subject = "setq gnus-topic-topology"
from = PROFILE
body = (setq gnus-topic-topology ...)

3. set group/topic parameters
subject = "gnus-topic-set-parameters GROUP-OR-TOPIC"
from = PROFILE
body = (gnus-topic-set-parameters ...)

4. set server parameters
subject = "PROFILE gnus-server-set-info SERVER"
from = PROFILE
body = (gnus-server-set-info ...)

All of the above are already entered into the dribble file at the right
time, with the right data, except for the profiles.

The sequence for saving each of the types above, given SUBJECT and BODY
and synchronization folder SYNC-FOLDER, and local connectivity profile
PROFILE:

1. create a new message with SUBJECT and BODY in SYNC-FOLDER

2. mark as 'read' all older messages with the same subject in FOLDER

Note that (2) must wait for (1) to succeed, but operates on all the
messages with SUBJECT except the newest one, so it's idempotent and will
not break if multiple clients do it at the same time.  Also, (2) can
fail and should be sent without waiting for a confirmation back.  It's
intended to speed up the read operations, but doesn't matter for the
save operations.

The sequence to synchronize:

1. find all the unread messages in SYNC-FOLDER.  Remember them in an
alist with cell format (SUBJECT . MESSAGE-ID).  This will be a small
alist.  For each distinct SUBJECT, only remember and load the latest
message.

2. ask the IMAP server to notify us when SYNC-FOLDER's UID changes and
do the usual dribble loading of new messages, skipping any message IDs
we have already remembered.  Are these notifications possible?

We should NOT synchronize the currently entered group; I think to avoid
user confusion we should simply ignore changes to it with a warning, and
save it on exit in the usual way.

Note that all the operations above for reading and saving are fast on
IMAP and are idempotent.

In synchronized mode, the newsrc file can still be saved and read, but
it will be overridden by whatever is in the SYNC-FOLDER.  So it's more
of a backup.

Let me know what you think about this proposal.  I think it's the right
solution, technically and practically.  Removing code (in this case
gnus-sync.el) in favor of the existing facilities is IMHO a good way to
simplify the user experience, reduce the maintenance burden, and improve
Gnus as a whole.

Ted