Re: gnus-sync ideas

[Ted Zlatanov <tzz@lifelogs.com>]

On Wed, 15 Dec 2010 20:18:50 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> And maybe it's not a Gnus-specific thing but could be useful for Emacs
>> in general, in which case it should be called data-sync.el and the
>> Gnus glue is gnus-sync.el.

LMI> That's an intriguing idea.  I mean, Emacs-in-the-cloud, sort of.

Yeah, the key is that the server can *merge* ELisp data structures and
possibly take partial updates.  Without that functionality, it's no
better than a Tramp remote file.

LMI> So when I start up my Emacs on my MeeGo phone (or something; my
LMI> phone ain't smart enough yet), then it'd pull down all the, er,
LMI> relevant bits to allow me to read news/mail/etc with the least
LMI> amount of stored data locally.

Right.  So you'd rely on data-sync.el to do the data management while
gnus-sync.el would do the Gnus-specific control, verification,
and conversion.

LMI> However, this has to work unplugged too, and it should sync (with the
LMI> least amount of data transferred) when it manages to connect to the
LMI> network.  So Emacs should have a way of asking "what's happened since
LMI> <TIMESTAMP>" and get all the changes...

LMI> That'd be cool.

LMI> The challenge (in addition to this being generally a challenging problem
LMI> area :-) would be to make this fast enough that it'd be appealing...

I can do a lot of the server-side work required here.  Something with a
HTTP front-end that takes POST requests, linked to a Membase or SQLite
database, would be ridiculously fast (probably faster than the
newsrc/registry save mechanism, heh).  In fact I imagine you or me will
end up running some version of this as a service for Emacs users.

But I want to figure out the server/data-sync.el requirements before
writing even one line of the implementation because adding more features
later will be a pain.

First of all, the data types will be: ELisp rtree, list, plist, alist,
vector, hashtable; JSON array and map; string.

So, data-sync.el would let you:

1) choose a storage backend, which can be many things but will at least
have the choice to be a file (which, of course, can use Tramp).

2) query a key and find out the latest value, data type, CAS key, and
timestamp (you'd use the CAS key in the CAS ops below)

3) set a key with a data type (unconditionally or CAS)

4) delete a key (unconditionally or CAS)

5) merge data into a key with a data type and a policy (overwrite,
prepend, append, set-if-present, set-if-absent) (unconditionally or CAS)

6) do all of the above as a batch

7) synchronize a bunch of keys and data according to a given policy and
return which ones succeeded, which ones failed to merge, and which ones
failed due to a CAS key mismatch (so you need to retry them).  This
would be your basic API.

Does that sound reasonable?  What else would you need?

Ted