Re: gnus-sync ideas

[Steinar Bang <sb@dod.no>]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

[snip!]
> First of all, how should it operate in Gnus?  IMO it should be
> immediate and not triggered as it is now; a sync should happen every
> time you enter or exit a group (and can also be triggered explicitly).

Definitely.  The way it is today, where I have to remember to stop
gnusen every time I leave them, or minimum remember to save, is a
nuisance. 

[snip!]
> Second, how should the data be stored?  IMO it should be stored in a
> free-form key-value format

Um... do you mean that the values should be blobs in elisp or somesuch
format?  (if so, what is the keys?  Not obvious to me) Or do you mean a
fine-grained "everything as a key/value pair" storage? (hm.. neither the
keys, nor the values are obvious to me here either... :-) )

> but the storage backend should be able to understand the rtree format;
> the ELisp list, vector, and plist formats; and maybe JSON and merge
> such data (with parameters that control how the merge will happen).

Hm... why so many formats?  We're not going for interoperability with
anything else (ie. why use JSON?).  And what's the difference between
elisp lists and the rtree format?  (FWIW I think the format should be
s-expressions/elisp lists, because it is simple and low cost on the
emacs side)

> I think a custom network server would actually be the best storage
> backend solution here; imap-hash.el could also work with some love;
> the file storage backend is also sensible to duplicate the current
> newsrc.eld functionality; and, of course, people will come up with a
> Git-based and RDBMS-based storage backend Because They Can.

Heh... the server could be ssh-ing to a central computer and starting an
emacs remotely to interpret the sync data, and consolidate with the lisp
list data stored on the server... "emacs in the cloud" :-)

(not serious: relying on ssh is too limiting, because it is blocked by
too many corporate firewalls)

Actually I think there only needs to be a single protocol http/https
with PUT or POST for updates (which could also return updated sync data
uploaded by other clients), and GET for initial sync on start, or
explicit syncs.

Wrt. round trip times: GET can be pipelined.  PUT and POST can't.  But
if you don't care about piggybacking updated sync info on the response
(only read upstream sync data with GET) you could just fire them off in
the background with the thread thing Lars talked about in a different
thread. 

> Third, what data are we storing?  I think marks should only be synced
> for news-p backends.  But we should generally be able to save, in order
> of importance IMHO:

> - marks for news-p servers (medium difficulty)

That I definitely want.
+1

> - splitting rules (trivial but needs a view/edit interface)

+0 (I don't use it, but don't let me stop you...:-) )

> - BBDB (hard but very important, and useful beyond Gnus)

Currently version controlled.  As you've pointed out, merge is not
really possible on the file level.
+1

> - topic hierarchy (medium)

0 (neutral, for the same reason as the next one)

> - subscriptions (but the user can pick subsets as the "work" and "home"
>   setup for instance) (hard)

-1 I like to set up those manually.  But I would like a newly subscribed
   group to get its read state from the sync server, if it is read in a
   different gnus

> - the Gnus registry (medium)

0 (neutral, because I don't use it)

- Score rules

+1 (because I use it)

> So because this is such diverse data I think the storage backend
> should just be a key-value store.  The rtree/ELisp data merge support
> is really the only thing special about it.  And maybe it's not a
> Gnus-specific thing but could be useful for Emacs in general, in which
> case it should be called data-sync.el and the Gnus glue is
> gnus-sync.el.  I'm sure the org-mode users, for instance, would like
> this kind of functionality too, and maybe they already have it and we
> can just steal it :)

Hm... I still don't have a clear idea how this maps to a key value
store.  These are nested constructs, aren't they?  Will you construct a
key by combining identificators at different level?

> Fourth, for general accessibility, data-sync.el will have a simple
> view/edit interface to load a specific key in a buffer, edit it in
> Emacs Lisp, rtree, or JSON mode, and commit it back if it validates
> (no merge, just overwrite).  This functionality could also connect
> with revisions if the storage backend supports them, so you could look
> at older versions of your data.

Why involve JSON?  Why not just lisp list structures?

> Fifth, the network server storage backend should be easy to set up and
> run.  Maybe a HTTP-backed implementation would make the most sense
> because those are so easy to set up and are not firewalled.

Yep.  It's what I think as well.  See comments above.

> Finally, what other parts of Emacs besides BBDB and org-mode could
> benefit from a general data-sync facility?

I'm sure others will think of something once it is in place.  The diary,
perhaps?  Though that has other sync possibilities (iCalendar).