gnus-sync ideas (was: Separating marks from other group configuration)

[Ted Zlatanov <tzz@lifelogs.com>]

On Mon, 13 Dec 2010 19:49:00 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:
>> Wouldn't it be nice -- if you had an IMAP server in your setup, then
>> everything would be stored there?  (I mean, in addition to the
>> .newsrc.eld file, which would be used if you couldn't access the IMAP
>> server.)

SB> Ted has already had some thoughts in this line, about extending his
SB> gnus-sync.el to cover stuff other than tick marks.

OK, let's throw some gnus-sync specific ideas around.  Assume the
current implementation is just a prototype and can be entirely thrown
away.  Also assume I will work on this soon-ish but am happy to let
others contribute up to 100%.  Finally, let's say one of the goals is to
eliminate newsrc.eld and other Gnus config files entirely or at least
make them a gnus-sync storage backend.

First of all, how should it operate in Gnus?  IMO it should be immediate
and not triggered as it is now; a sync should happen every time you
enter or exit a group (and can also be triggered explicitly).  Here we
can use the hooks I added recently: `gnus-after-set-mark-hook' and
`gnus-before-update-mark-hook'.

Second, how should the data be stored?  IMO it should be stored in a
free-form key-value format but the storage backend should be able to
understand the rtree format; the ELisp list, vector, and plist formats;
and maybe JSON and merge such data (with parameters that control how the
merge will happen).  I think a custom network server would actually be
the best storage backend solution here; imap-hash.el could also work
with some love; the file storage backend is also sensible to duplicate
the current newsrc.eld functionality; and, of course, people will come
up with a Git-based and RDBMS-based storage backend Because They Can.

Third, what data are we storing?  I think marks should only be synced
for news-p backends.  But we should generally be able to save, in order
of importance IMHO:

- marks for news-p servers (medium difficulty)

- splitting rules (trivial but needs a view/edit interface)

- BBDB (hard but very important, and useful beyond Gnus)

- topic hierarchy (medium)

- subscriptions (but the user can pick subsets as the "work" and "home"
  setup for instance) (hard)

- the Gnus registry (medium)

So because this is such diverse data I think the storage backend should
just be a key-value store.  The rtree/ELisp data merge support is really
the only thing special about it.  And maybe it's not a Gnus-specific
thing but could be useful for Emacs in general, in which case it should
be called data-sync.el and the Gnus glue is gnus-sync.el.  I'm sure the
org-mode users, for instance, would like this kind of functionality too,
and maybe they already have it and we can just steal it :)

Fourth, for general accessibility, data-sync.el will have a simple
view/edit interface to load a specific key in a buffer, edit it in Emacs
Lisp, rtree, or JSON mode, and commit it back if it validates (no merge,
just overwrite).  This functionality could also connect with revisions
if the storage backend supports them, so you could look at older
versions of your data.

Fifth, the network server storage backend should be easy to set up and
run.  Maybe a HTTP-backed implementation would make the most sense
because those are so easy to set up and are not firewalled.

Finally, what other parts of Emacs besides BBDB and org-mode could
benefit from a general data-sync facility?

WDYT?

Ted