From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/75453 Path: news.gmane.org!not-for-mail From: Ted Zlatanov Newsgroups: gmane.emacs.gnus.general Subject: Re: gnus-sync ideas Date: Mon, 27 Dec 2010 09:22:46 -0600 Organization: =?utf-8?B?0KLQtdC+0LTQvtGAINCX0LvQsNGC0LDQvdC+0LI=?= @ Cienfuegos Message-ID: <87bp472qbd.fsf@lifelogs.com> References: <87zks9ecf7.fsf@dod.no> <87r5djxahm.fsf_-_@lifelogs.com> <87fwttv83u.fsf@dod.no> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: dough.gmane.org 1293463407 6756 80.91.229.12 (27 Dec 2010 15:23:27 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Mon, 27 Dec 2010 15:23:27 +0000 (UTC) To: ding@gnus.org Original-X-From: ding-owner+M23804@lists.math.uh.edu Mon Dec 27 16:23:23 2010 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1PXEuu-0001mB-Px for ding-account@gmane.org; Mon, 27 Dec 2010 16:23:21 +0100 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1PXEub-0007aW-Pp; Mon, 27 Dec 2010 09:23:01 -0600 Original-Received: from mx2.math.uh.edu ([129.7.128.33]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1PXEuZ-0007aK-P6 for ding@lists.math.uh.edu; Mon, 27 Dec 2010 09:22:59 -0600 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx2.math.uh.edu with esmtp (Exim 4.72) (envelope-from ) id 1PXEuY-00079T-D1 for ding@lists.math.uh.edu; Mon, 27 Dec 2010 09:22:59 -0600 Original-Received: from lo.gmane.org ([80.91.229.12]) by quimby.gnus.org with esmtp (Exim 4.72) (envelope-from ) id 1PXEuX-00018L-Mw for ding@gnus.org; Mon, 27 Dec 2010 16:22:57 +0100 Original-Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1PXEuX-0001fR-FG for ding@gnus.org; Mon, 27 Dec 2010 16:22:57 +0100 Original-Received: from c-67-186-102-106.hsd1.il.comcast.net ([67.186.102.106]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 27 Dec 2010 16:22:57 +0100 Original-Received: from tzz by c-67-186-102-106.hsd1.il.comcast.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 27 Dec 2010 16:22:57 +0100 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 94 Original-X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: c-67-186-102-106.hsd1.il.comcast.net X-Face: bd.DQ~'29fIs`T_%O%C\g%6jW)yi[zuz6;d4V0`@y-~$#3P_Ng{@m+e4o<4P'#(_GJQ%TT= D}[Ep*b!\e,fBZ'j_+#"Ps?s2!4H2-Y"sx" User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/24.0.50 (gnu/linux) Cancel-Lock: sha1:8xAQLP7xywt7/UjYf1eliII4mSw= X-Spam-Score: -1.9 (-) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:75453 Archived-At: On Sun, 19 Dec 2010 21:08:21 +0100 Steinar Bang wrote: >>>>>> Ted Zlatanov : >> Second, how should the data be stored? IMO it should be stored in a >> free-form key-value format SB> Um... do you mean that the values should be blobs in elisp or somesuch SB> format? (if so, what is the keys? Not obvious to me) Or do you mean a SB> fine-grained "everything as a key/value pair" storage? (hm.. neither the SB> keys, nor the values are obvious to me here either... :-) ) The key is going to be consistent, e.g. "gnus.marks.GROUPNAME" but the storage (data-sync.el) backend may prefix the key with something. For a shared networked storage backend, for instance, the prefix would be something unique and unguessable to prevent accidental or malicious collisions (there will also be a password). The values will be data blobs with only one property: they can be merged. Thus many data formats can be supported. >> but the storage backend should be able to understand the rtree format; >> the ELisp list, vector, and plist formats; and maybe JSON and merge >> such data (with parameters that control how the merge will happen). SB> Hm... why so many formats? We're not going for interoperability with SB> anything else (ie. why use JSON?). And what's the difference between SB> elisp lists and the rtree format? (FWIW I think the format should be SB> s-expressions/elisp lists, because it is simple and low cost on the SB> emacs side) rtree is much more compact that simple sexps for marks IIUC. So it makes sense to use it for marks, but it's not appropriate for many other formats. The JSON support would be there for other uses. I may actually convert between JSON, rtree, and sexps on the fly but haven't decided that yet. If I do, other clients besides ELisp-based could access the data sync functionality. >> I think a custom network server would actually be the best storage >> backend solution here; imap-hash.el could also work with some love; >> the file storage backend is also sensible to duplicate the current >> newsrc.eld functionality; and, of course, people will come up with a >> Git-based and RDBMS-based storage backend Because They Can. SB> Heh... the server could be ssh-ing to a central computer and starting an SB> emacs remotely to interpret the sync data, and consolidate with the lisp SB> list data stored on the server... "emacs in the cloud" :-) SB> (not serious: relying on ssh is too limiting, because it is blocked by SB> too many corporate firewalls) Yeah. Also Emacs as a server is extremely limited--I thought of this. It has to use HTTP as the transport and some kind of web service as the RPC mechanism for a networked storage backend. But for a local storage backend Emacs itself can do the job just fine. SB> Actually I think there only needs to be a single protocol http/https SB> with PUT or POST for updates (which could also return updated sync data SB> uploaded by other clients), and GET for initial sync on start, or SB> explicit syncs. SB> Wrt. round trip times: GET can be pipelined. PUT and POST can't. But SB> if you don't care about piggybacking updated sync info on the response SB> (only read upstream sync data with GET) you could just fire them off in SB> the background with the thread thing Lars talked about in a different SB> thread. That's easy to do as an optimization. I'll try to get the basics working first. The HTTP GET/POST semantics should be sufficient so PUT and DELETE won't be needed. >> So because this is such diverse data I think the storage backend >> should just be a key-value store. The rtree/ELisp data merge support >> is really the only thing special about it. And maybe it's not a >> Gnus-specific thing but could be useful for Emacs in general, in which >> case it should be called data-sync.el and the Gnus glue is >> gnus-sync.el. I'm sure the org-mode users, for instance, would like >> this kind of functionality too, and maybe they already have it and we >> can just steal it :) SB> Hm... I still don't have a clear idea how this maps to a key value SB> store. These are nested constructs, aren't they? Will you construct a SB> key by combining identificators at different level? gnus-sync.el will do the mapping between the "Gnus state" and the "synchronized state." Eventually the two will converge, I hope, and Gnus will only have a "synchronized state" which happens to be synchronized to local disk by default. But at first there will be a conversion process to avoid rewriting Gnus too radically and to get the data synchronization working properly. Ted