Re: I can haz cloud idea

[Ted Zlatanov <tzz@lifelogs.com>]

On Thu, 16 Feb 2012 06:03:47 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> The main problems with going all cloudy is that you have to have yet
LI> another server out there somewhere.  Whether you use Dropbox/sshfs/a yet
LI> to be determined Gnus-specific server, that's not very sexy.

LI> But how about if we just cheat and put all the information on the IMAP
LI> server?

LI> That is: Gnus would create and maintain a special group on one of your
LI> IMAP servers that would contain all the data Gnus needs.

LI> You'd set this up by setting, er, something, like:

LI> (setq gnus-cloud-server '(nnimap "imap.gmail.com"))

Why not add a gnus-sync backend to do this?

LI> This would make Gnus save all the data on that server whenever you exit
LI> Gnus.  (Or on certain intervals.)  It could gzip and encrypt the data
LI> before uploading, I guess.

LI> So when you go to your other machine, you'd say `M-x gnus-cloud', and it
LI> would prompt you for the server and password(s), download the data, and
LI> then start up more or less as normal.

LI> And it would upload the data on exit, as well, so you'd be all
LI> cloudy.

This is what `gnus-sync-read' and `gnus-sync-save' do already, for some of
the newsrc data.  There's no problem with storing more, my only issue
was to figure out how to synchronize it all cleanly.

I also want to use gnus-sync to store the gnus-registry and even BBDB
data.  So it makes sense to make this a general facility.

LI> What would be stored on the server?  Well, the .newsrc.eld file, for
LI> one.  And the SCORE files, I guess.  And...  well, whatever else you
LI> want.  It could be a simple archive "file" type, and Gnus could just
LI> decrypt, uncompress and unpack the files.  Some sanity check about
LI> overwriting newer files with older files, I guess.

The CouchDB gnus-sync backend has built-in document versioning, which I
think is really nice.  Also it's portable through JSON and HTTP (what I
call the LeSync protocol), so other newsreaders can support it too.  The
developer of NewsTap, a iOS app, is on board with this and told me it's
on top of his feature list.

LI> If this is to work smoothly (i.e., at all), then every time you hit `g'
LI> in a Gnus, it'll have to know whether something else updated the state
LI> group.  This also means that every time you exit a group or does
LI> something else that changes state, it'll have to be uploaded pretty much
LI> immediately.  This also means that there would be no explicit "upload"
LI> commands.

So basically a `gnus-sync-save' on group exit.  I would amortize it so
it's not done so frequently (currently, gnus-sync saves when the newsrc
is saved) because network transactions are expensive.

LI> Here's how I see this working:

LI> When you first start the thing, it does a total upload.  Then, every
LI> time you (say) exit a group, it only uploads the delta between the
LI> previous total upload and the current state.  So after you've read a
LI> couple of groups, the delta file will look like

LI> ("gwene.com.feedburner.cheezburger" (add (read (56 . 59))) (del (tick 58)))

LI> or something.  Gnus will continue to upload the entire delta file to the
LI> server on each group exit (and delete any previous deltas).  Until the
LI> delta gets so big that Gnus just does a total upload again, and removes
LI> all previous data.

LI> It sucks that `g' has to query the state group, though.  But it really
LI> doesn't.  If we maintain a separate cloud connection, then we can just
LI> IDLE on the state group.  Then every time a different Gnus somewhere
LI> does something, then the other Gnus instances can just download the
LI> state change, and update itself automatically.  Totally transparently
LI> and in the background.

LI> Unless the IMAP server has dropped the connection, of course.  Then `g'
LI> will have to reconnect and do the update while you wait.

With CouchDB, at least, you can do a differential poll: tell me what has
changed since change X (integer numeric sequence).  You can also open a
persistent connection and get changes live when they are made.  So this
could be, again, a gnus-sync feature per backend, and the gnus-sync
LeSync and IMAP backends would support live updates while the file
backend wouldn't.

CouchDB's document model is interesting: you have to read the current
revision to write a new one back.  So you can't accidentally overwrite a
document, you have to read it first (which gives you a chance to
merge).  So at least for CouchDB/LeSync you don't need to do delta
files.

Ted