From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/75070 Path: news.gmane.org!not-for-mail From: Ted Zlatanov Newsgroups: gmane.emacs.gnus.general Subject: Re: gnus-sync ideas Date: Wed, 15 Dec 2010 15:26:53 -0600 Organization: =?utf-8?B?0KLQtdC+0LTQvtGAINCX0LvQsNGC0LDQvdC+0LI=?= @ Cienfuegos Message-ID: <871v5iviaq.fsf@lifelogs.com> References: <87zks9ecf7.fsf@dod.no> <87r5djxahm.fsf_-_@lifelogs.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: dough.gmane.org 1292448618 27213 80.91.229.12 (15 Dec 2010 21:30:18 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Wed, 15 Dec 2010 21:30:18 +0000 (UTC) To: ding@gnus.org Original-X-From: ding-owner+M23425@lists.math.uh.edu Wed Dec 15 22:30:14 2010 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1PSyvN-0007Wl-2h for ding-account@gmane.org; Wed, 15 Dec 2010 22:30:13 +0100 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1PSyvK-0005Ih-Hr; Wed, 15 Dec 2010 15:30:10 -0600 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1PSyvJ-0005IW-A6 for ding@lists.math.uh.edu; Wed, 15 Dec 2010 15:30:09 -0600 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtp (Exim 4.72) (envelope-from ) id 1PSyvF-0005Eh-5V for ding@lists.math.uh.edu; Wed, 15 Dec 2010 15:30:09 -0600 Original-Received: from lo.gmane.org ([80.91.229.12]) by quimby.gnus.org with esmtp (Exim 4.72) (envelope-from ) id 1PSyvE-0000QJ-Ds for ding@gnus.org; Wed, 15 Dec 2010 22:30:04 +0100 Original-Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1PSyvD-0007QD-Kq for ding@gnus.org; Wed, 15 Dec 2010 22:30:03 +0100 Original-Received: from 38.98.147.130 ([38.98.147.130]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 15 Dec 2010 22:30:03 +0100 Original-Received: from tzz by 38.98.147.130 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 15 Dec 2010 22:30:03 +0100 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 70 Original-X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: 38.98.147.130 X-Face: bd.DQ~'29fIs`T_%O%C\g%6jW)yi[zuz6;d4V0`@y-~$#3P_Ng{@m+e4o<4P'#(_GJQ%TT= D}[Ep*b!\e,fBZ'j_+#"Ps?s2!4H2-Y"sx" User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/24.0.50 (gnu/linux) Cancel-Lock: sha1:T9qoIkmzpyTqdoxXGwqEnnuecDQ= X-Spam-Score: -0.7 (/) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:75070 Archived-At: On Wed, 15 Dec 2010 20:18:50 +0100 Lars Magne Ingebrigtsen wrote: LMI> Ted Zlatanov writes: >> And maybe it's not a Gnus-specific thing but could be useful for Emacs >> in general, in which case it should be called data-sync.el and the >> Gnus glue is gnus-sync.el. LMI> That's an intriguing idea. I mean, Emacs-in-the-cloud, sort of. Yeah, the key is that the server can *merge* ELisp data structures and possibly take partial updates. Without that functionality, it's no better than a Tramp remote file. LMI> So when I start up my Emacs on my MeeGo phone (or something; my LMI> phone ain't smart enough yet), then it'd pull down all the, er, LMI> relevant bits to allow me to read news/mail/etc with the least LMI> amount of stored data locally. Right. So you'd rely on data-sync.el to do the data management while gnus-sync.el would do the Gnus-specific control, verification, and conversion. LMI> However, this has to work unplugged too, and it should sync (with the LMI> least amount of data transferred) when it manages to connect to the LMI> network. So Emacs should have a way of asking "what's happened since LMI> " and get all the changes... LMI> That'd be cool. LMI> The challenge (in addition to this being generally a challenging problem LMI> area :-) would be to make this fast enough that it'd be appealing... I can do a lot of the server-side work required here. Something with a HTTP front-end that takes POST requests, linked to a Membase or SQLite database, would be ridiculously fast (probably faster than the newsrc/registry save mechanism, heh). In fact I imagine you or me will end up running some version of this as a service for Emacs users. But I want to figure out the server/data-sync.el requirements before writing even one line of the implementation because adding more features later will be a pain. First of all, the data types will be: ELisp rtree, list, plist, alist, vector, hashtable; JSON array and map; string. So, data-sync.el would let you: 1) choose a storage backend, which can be many things but will at least have the choice to be a file (which, of course, can use Tramp). 2) query a key and find out the latest value, data type, CAS key, and timestamp (you'd use the CAS key in the CAS ops below) 3) set a key with a data type (unconditionally or CAS) 4) delete a key (unconditionally or CAS) 5) merge data into a key with a data type and a policy (overwrite, prepend, append, set-if-present, set-if-absent) (unconditionally or CAS) 6) do all of the above as a batch 7) synchronize a bunch of keys and data according to a given policy and return which ones succeeded, which ones failed to merge, and which ones failed due to a CAS key mismatch (so you need to retry them). This would be your basic API. Does that sound reasonable? What else would you need? Ted