Re: gnus-sync ideas

[Ted Zlatanov <tzz@lifelogs.com>]

On Sun, 19 Dec 2010 21:08:21 +0100 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:

>> Second, how should the data be stored?  IMO it should be stored in a
>> free-form key-value format

SB> Um... do you mean that the values should be blobs in elisp or somesuch
SB> format?  (if so, what is the keys?  Not obvious to me) Or do you mean a
SB> fine-grained "everything as a key/value pair" storage? (hm.. neither the
SB> keys, nor the values are obvious to me here either... :-) )

The key is going to be consistent, e.g. "gnus.marks.GROUPNAME" but the
storage (data-sync.el) backend may prefix the key with something.  For a
shared networked storage backend, for instance, the prefix would be
something unique and unguessable to prevent accidental or malicious
collisions (there will also be a password).

The values will be data blobs with only one property: they can be
merged.  Thus many data formats can be supported.

>> but the storage backend should be able to understand the rtree format;
>> the ELisp list, vector, and plist formats; and maybe JSON and merge
>> such data (with parameters that control how the merge will happen).

SB> Hm... why so many formats?  We're not going for interoperability with
SB> anything else (ie. why use JSON?).  And what's the difference between
SB> elisp lists and the rtree format?  (FWIW I think the format should be
SB> s-expressions/elisp lists, because it is simple and low cost on the
SB> emacs side)

rtree is much more compact that simple sexps for marks IIUC.  So it
makes sense to use it for marks, but it's not appropriate for many other
formats.

The JSON support would be there for other uses.  I may actually convert
between JSON, rtree, and sexps on the fly but haven't decided that yet.
If I do, other clients besides ELisp-based could access the data sync
functionality.

>> I think a custom network server would actually be the best storage
>> backend solution here; imap-hash.el could also work with some love;
>> the file storage backend is also sensible to duplicate the current
>> newsrc.eld functionality; and, of course, people will come up with a
>> Git-based and RDBMS-based storage backend Because They Can.

SB> Heh... the server could be ssh-ing to a central computer and starting an
SB> emacs remotely to interpret the sync data, and consolidate with the lisp
SB> list data stored on the server... "emacs in the cloud" :-)

SB> (not serious: relying on ssh is too limiting, because it is blocked by
SB> too many corporate firewalls)

Yeah.  Also Emacs as a server is extremely limited--I thought of this.
It has to use HTTP as the transport and some kind of web service as the
RPC mechanism for a networked storage backend.  But for a local storage
backend Emacs itself can do the job just fine.

SB> Actually I think there only needs to be a single protocol http/https
SB> with PUT or POST for updates (which could also return updated sync data
SB> uploaded by other clients), and GET for initial sync on start, or
SB> explicit syncs.

SB> Wrt. round trip times: GET can be pipelined.  PUT and POST can't.  But
SB> if you don't care about piggybacking updated sync info on the response
SB> (only read upstream sync data with GET) you could just fire them off in
SB> the background with the thread thing Lars talked about in a different
SB> thread. 

That's easy to do as an optimization.  I'll try to get the basics
working first.  The HTTP GET/POST semantics should be sufficient so PUT
and DELETE won't be needed.

>> So because this is such diverse data I think the storage backend
>> should just be a key-value store.  The rtree/ELisp data merge support
>> is really the only thing special about it.  And maybe it's not a
>> Gnus-specific thing but could be useful for Emacs in general, in which
>> case it should be called data-sync.el and the Gnus glue is
>> gnus-sync.el.  I'm sure the org-mode users, for instance, would like
>> this kind of functionality too, and maybe they already have it and we
>> can just steal it :)

SB> Hm... I still don't have a clear idea how this maps to a key value
SB> store.  These are nested constructs, aren't they?  Will you construct a
SB> key by combining identificators at different level?

gnus-sync.el will do the mapping between the "Gnus state" and the
"synchronized state."  Eventually the two will converge, I hope, and
Gnus will only have a "synchronized state" which happens to be
synchronized to local disk by default.  But at first there will be a
conversion process to avoid rewriting Gnus too radically and to get the
data synchronization working properly.

Ted