Re: I can haz cloud idea

[Ted Zlatanov <tzz@lifelogs.com>]

On Sun, 19 Feb 2012 16:21:51 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> So the original idea would be to have a bundle of files distributed via
LI> IMAP, plus diffs to avoid down/uploading so much stuff.  When stuff
LI> changes in the bundle, Gnus updates the appropriate files.

The actual amount of data for a group, as a JSON object in LeSync, is
tiny.  I would avoid deltas if possible.  They are fragile, hard to
debug, and unshareable with other MUAs.  Put all the data a group needs
in a single place (a message in the mailbox, in this case, or a CouchDB
document in LeSync) and rewrite the whole document atomically when it
changes.  Yes, you have a versioning problem, if someone else writes
before you.  So use a revision number.  Please look at how CouchDB does
it: only save the document if the latest document's version matches the
latest version you know.  Otherwise, reconcile with the latest saved.

LI> Why rely on other files for the significant stuff?  We control the
LI> vertical and the horizontal, so why not just rethink this?

LI> The score files, for instance, would more naturally be a structure
LI> stored in .newsrc.eld.  Finding out what score files that match what
LI> group is pretty gross code-wise.  Why not just make that into a nice
LI> alist and put it in the .newsrc.eld file?

LI> The same goes for other obvious stuff like ~/.signature.

LI> Just put it all in .newsrc.eld, and then distribute that via IMAP.

I would have the gnus-sync backend store user profiles, like the posting
styles in Gnus.  The profile would point to or inline the signature
file, score files, SMTP server, and everything else the user needs, as
customizations.  Then you pick a profile on each machine you use.

I would make each profile, with all the customizations, a single
document that users can edit and understand.  Don't spread things out
just because you can.

LI> Ted Zlatanov <tzz@lifelogs.com> writes:

>> Why not add a gnus-sync backend to do this?

LI> Because I want it all to be asynchronous, opportunistic and streaming.

That's fine.  The gnus-sync backend should support that when you call
`gnus-sync-save'.  You're taking the mechanics, the way code will
operate, and using that as an argument against the architecture, the way
data and user-customizable variables are planned.

With CouchDB you can stream by opening a persistent connection to the
server.  url.el can be asynchronous, and so can you against CouchDB.
I'm not sure what "opportunistic" means exactly, but if you mean "save
when possible" then I think that would work.

Just realize that by setting the gnus-sync backend to files, you can be
backwards compatible with the current Gnus.  That way you don't have to
have a special "cloud mode" but simply "sync to local filesystem."  Your
way is a special case all the way from data layout to mechanics.

LI> When exiting a group, we'll be squirting off an APPEND command to the
LI> IMAP server (with the current delta), but we don't wait for it to
LI> return finish.  We queue them up, and if we get three group exits before
LI> the first APPEND has finished, we just skip the second one.

LI> Conversely, using IDLE on the mail box will keep the other instances
LI> updated asynchronously without anybody having to issue any commands.

LI> It'll be like using the IMAP synchronisation methodology, only for all
LI> backends and stuff.

OK.  But that's nearly impossible to share with other MUAs, unlike my
LeSync proposal, and I hope you don't use deltas as I said above.

>> The CouchDB gnus-sync backend has built-in document versioning, which I
>> think is really nice.

LI> I don't think having CouchDB as a prerequisite is totally reasonable.
LI> It should just work without having to install anything.

You're basing your side on a working, always-reachable IMAP server,
which is less reasonable in the context of the Internet and requires
significant infrastructure.  Port 80 is much more likely to be open than
other ports, especially in a corporate environment.

I didn't decide on CouchDB because it's trendy (it's SO 2009!), but
because it's a schemaless database+web server hybrid with custom views
(e.g. I can put together a XML view of the marks that NewsTap can import
directly).  You can get a free instance at http://iriscouch.com or
https://cloudant.com and get running in minutes, or set your own up even
quicker, or I can set one up for you at WHATEVER.lesync.info.

Ted