I can haz cloud idea

I can haz cloud idea

[Lars Ingebrigtsen]

The main problems with going all cloudy is that you have to have yet
another server out there somewhere.  Whether you use Dropbox/sshfs/a yet
to be determined Gnus-specific server, that's not very sexy.

But how about if we just cheat and put all the information on the IMAP
server?

That is: Gnus would create and maintain a special group on one of your
IMAP servers that would contain all the data Gnus needs.

You'd set this up by setting, er, something, like:

(setq gnus-cloud-server '(nnimap "imap.gmail.com"))

This would make Gnus save all the data on that server whenever you exit
Gnus.  (Or on certain intervals.)  It could gzip and encrypt the data
before uploading, I guess.

So when you go to your other machine, you'd say `M-x gnus-cloud', and it
would prompt you for the server and password(s), download the data, and
then start up more or less as normal.

And it would upload the data on exit, as well, so you'd be all cloudy.

What would be stored on the server?  Well, the .newsrc.eld file, for
one.  And the SCORE files, I guess.  And...  well, whatever else you
want.  It could be a simple archive "file" type, and Gnus could just
decrypt, uncompress and unpack the files.  Some sanity check about
overwriting newer files with older files, I guess.

How much data would this be?  Here's my .newsrc.eld file:

larsi@rusty:~/mgnus$ ls -l ~/.newsrc.eld
-rw-r--r-- 1 larsi larsi 502770 2012-02-16 05:46 /home/larsi/.newsrc.eld
larsi@rusty:~/mgnus$ gzip -9 -c /home/larsi/.newsrc.eld | wc 
    441    2222  117577
larsi@rusty:~/mgnus$ xz -c /home/larsi/.newsrc.eld | wc 
    258    1440   65288

A 65K download isn't too bad.  Well.  You'd have to base64 it before
uploading, I guess.

Does Emacs have xz compression built in?  And built-in encryption would
also be nice, but that's already planned for Emacs 24.2, I guess...

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Rome

Re: I can haz cloud idea

[Lars Ingebrigtsen]

If this is to work smoothly (i.e., at all), then every time you hit `g'
in a Gnus, it'll have to know whether something else updated the state
group.  This also means that every time you exit a group or does
something else that changes state, it'll have to be uploaded pretty much
immediately.  This also means that there would be no explicit "upload"
commands.

Here's how I see this working:

When you first start the thing, it does a total upload.  Then, every
time you (say) exit a group, it only uploads the delta between the
previous total upload and the current state.  So after you've read a
couple of groups, the delta file will look like

("gwene.com.feedburner.cheezburger" (add (read (56 . 59))) (del (tick 58)))

or something.  Gnus will continue to upload the entire delta file to the
server on each group exit (and delete any previous deltas).  Until the
delta gets so big that Gnus just does a total upload again, and removes
all previous data.

It sucks that `g' has to query the state group, though.  But it really
doesn't.  If we maintain a separate cloud connection, then we can just
IDLE on the state group.  Then every time a different Gnus somewhere
does something, then the other Gnus instances can just download the
state change, and update itself automatically.  Totally transparently
and in the background.

Unless the IMAP server has dropped the connection, of course.  Then `g'
will have to reconnect and do the update while you wait.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Rome

Re: I can haz cloud idea

[Antoine Levitt]

That'd be a really cute feature !

16/02/12 07:49, Lars Ingebrigtsen
> If this is to work smoothly (i.e., at all), then every time you hit `g'
> in a Gnus, it'll have to know whether something else updated the state
> group.  This also means that every time you exit a group or does
> something else that changes state, it'll have to be uploaded pretty much
> immediately.  This also means that there would be no explicit "upload"
> commands.

Please consider making the feature customizable enough to allow the
possibility of an explicit sync.

Re: I can haz cloud idea

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:
[snip!]
> Here's how I see this working:

Have you given any thought to offline/agent operations?

Re: I can haz cloud idea

[Lars Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Have you given any thought to offline/agent operations?

No...  :-)  But thinking about it for the length of time it's taken me
to write these words, I don't think offline-ness complicates matters
much.  I mean, you need to go online to pull/push updates, but the
computations themselves would be identical.

That is, Gnus would keep a set of "baseline" data in memory.  (Basically
everything that is in .newsrc.eld.)  Whenever Gnus does a push, it
computes the difference between the baseline and the current state.  If
the difference is smallish, it then pushes a delta to the IMAP server.
If not, it pushes the complete state.

So going-online-and-then-pushing wouldn't introduce any new code.  Same
with pulling -- it should just work.

I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Rome

Re: I can haz cloud idea

[Vegard Vesterheim]

On Thu, 16 Feb 2012 06:03:47 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote:

> But how about if we just cheat and put all the information on the IMAP
> server?

yes, Yes, YES!

 - Vegard V -

Re: I can haz cloud idea

[Richard Riley]

Vegard Vesterheim <vegard.vesterheim@uninett.no> writes:

> On Thu, 16 Feb 2012 06:03:47 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
>> But how about if we just cheat and put all the information on the IMAP
>> server?
>
> yes, Yes, YES!
>
>  - Vegard V -
>

Use the existing gpg stuff as currently supported for auth files etc.

Re: I can haz cloud idea

[Lars Ingebrigtsen]

I've been gallivanting around a bit, but I've been thinking a bit more
about the cloud thing.

So the original idea would be to have a bundle of files distributed via
IMAP, plus diffs to avoid down/uploading so much stuff.  When stuff
changes in the bundle, Gnus updates the appropriate files.

But then I thought:  This is kinda like a virtual file system.  This
could be implemented along similar lines like Emacs tar support.  That
is: We have an in-memory file system "mounted" at (say) ~/.gnus-cloud,
and whenever you edit ~/.gnus-cloud/News/all.SCORE, you're really just
editing the bundle kept in-memory.  This means that if you start Gnus
with `M-x gnus-cloud', no files would need to be written to local disk.

So you'd typically have a "real" Gnus somewhere that writes stuff out to
real files.  All (or most) of your other Gnus instances would just rely
on the data from the IMAP server.  No local files!

Doesn't that have a clean feeling?

But then, today, I had another thought.

Gnus setup consists of the .newsrc.eld file, and then a bunch of other
files, like ~/.signature and ~/News/*.SCORE.  But...  why?

Why rely on other files for the significant stuff?  We control the
vertical and the horizontal, so why not just rethink this?

The score files, for instance, would more naturally be a structure
stored in .newsrc.eld.  Finding out what score files that match what
group is pretty gross code-wise.  Why not just make that into a nice
alist and put it in the .newsrc.eld file?

The same goes for other obvious stuff like ~/.signature.

Just put it all in .newsrc.eld, and then distribute that via IMAP.

Admittedly, I've had a few Italian beers, but it all seems to make sense
if you do it that way.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Rome

Re: I can haz cloud idea

[Andy Moreton]

On Sun 19 Feb 2012, Lars Ingebrigtsen wrote:

> I've been gallivanting around a bit, but I've been thinking a bit more
> about the cloud thing.
>
> So the original idea would be to have a bundle of files distributed via
> IMAP, plus diffs to avoid down/uploading so much stuff.  When stuff
> changes in the bundle, Gnus updates the appropriate files.
>
> But then I thought:  This is kinda like a virtual file system.  This
> could be implemented along similar lines like Emacs tar support.  That
> is: We have an in-memory file system "mounted" at (say) ~/.gnus-cloud,
> and whenever you edit ~/.gnus-cloud/News/all.SCORE, you're really just
> editing the bundle kept in-memory.  This means that if you start Gnus
> with `M-x gnus-cloud', no files would need to be written to local disk.

Surely this is really a lot more like a distributed version control
system. After all, you do want to be able to recover the previous state
after errors...

So keep everything in simple self consistent local files, then use DVCS
mechanisms to propagate versions.

> So you'd typically have a "real" Gnus somewhere that writes stuff out to
> real files.  All (or most) of your other Gnus instances would just rely
> on the data from the IMAP server.  No local files!
>
> Doesn't that have a clean feeling?

Shudder... sounds like "single point of failure" to me. I'd like a local
copy of everything, so that when the cloudy stuff isn't working I can
ignore it and carry one regardless.

> But then, today, I had another thought.
>
> Gnus setup consists of the .newsrc.eld file, and then a bunch of other
> files, like ~/.signature and ~/News/*.SCORE.  But...  why?
>
> Why rely on other files for the significant stuff?  We control the
> vertical and the horizontal, so why not just rethink this?
>
> The score files, for instance, would more naturally be a structure
> stored in .newsrc.eld.  Finding out what score files that match what
> group is pretty gross code-wise.  Why not just make that into a nice
> alist and put it in the .newsrc.eld file?

Please, please do *not* add more stuff to .newsrc.eld. That file is
already over-complex, and trying to do too many things. Keep things
simple and unixy - a single responsibility for each file.

> The same goes for other obvious stuff like ~/.signature.
>
> Just put it all in .newsrc.eld, and then distribute that via IMAP.
>
> Admittedly, I've had a few Italian beers, but it all seems to make sense
> if you do it that way.

I hope you feel differently when you've sobered up :-)

    AndyM

Re: I can haz cloud idea

[Dan Christensen]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> Please, please do *not* add more stuff to .newsrc.eld. 

I agree.

> That file is already over-complex, and trying to do too many
> things. Keep things simple and unixy - a single responsibility for
> each file.

Moreover, as soon as you have multiple clients sharing data, you are
bound to get into a situation where some merging is required.  (E.g.
I go to work and learn that my home machine hasn't synced;  I've got
to get work done, so I've got to make changes at work as well.)
Merging is generally easier when things are stored in logically
separate containers.

In addition, updating single "records" should be fast, and if everything
is stored in the .newsrc.eld on an abstract filesystem, that's hard, and
you have to resort to storing diffs, etc, to be efficient.

Even more, it's doubtful that I'm going to want to sync the full 
.newsrc.eld, since on some machines I'll access different servers.
Sync/cloud should be configured separately for each backend, if
only for efficiency.

I'm also not sure that a DCVS is a magic bullet here, since we're
going to need problem specific merging and updating.  For example,
Gnus should remember the last state it saw on the server, and 
propagate changes back and forth, rather than copying over the
new data (in case it has also been changed by another client).

I haven't looked into it closely, but doesn't Ted's gnus-sync framework
seem like a good basis for continued development?  This really seems to
me to fit into the database point of view.

I also have to say that I always liked the idea of having the backends
store marks.  It allows naive methods of syncing and backup to just
work, and it makes the backends behave more like nnimap which is now
becoming one of the most used backends.

I feels to me like the syncing gnus already has to do between it's
knowledge of the marks in an nnimap group and the remote server's
knowledge is really similar to what needs to happen for other backends
in a cloud set-up.  This is also closely related to the
plugged/unplugged/agent feature.  Can all of this be merged into one
concept?

0) Gnus doesn't store any marks in the .newsrc.eld.
1) On startup, Gnus asks the backend for the marks, and displays the group.
2) As the user changes things, Gnus keeps track of the changes made
   to marks.
3) As various intervals, Gnus syncs the changes to the backend.
   Being "unplugged" which just mean deferring any syncs.

The user could configure the backend to be nnimap, which already
handles storing the marks on a server.

Or the user could configure the backend to be nnml, with marks
just stored locally.

Or the user could configure the backend to be nnml, with marks
stored on a server (e.g. Ted's LeSync idea, or some other remote
storage).

It really seems like getting all the marks out of .newsrc.eld
and putting them in the backends would unify many ideas in Gnus.
(And by default, servers could just store marks in a file
like .newsrc.server-name.eld, using nnoo common code, so there
wouldn't need to be code duplication in the backends.  Even
the default syncing code could be in common, but overridden
by servers like nnimap that have a custom way to store marks.)

Dan

Re: I can haz cloud idea

[Lars Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> I'm also not sure that a DCVS is a magic bullet here, since we're
> going to need problem specific merging and updating.  For example,
> Gnus should remember the last state it saw on the server, and 
> propagate changes back and forth, rather than copying over the
> new data (in case it has also been changed by another client).

Gnus will be uploading diffs, of course, and not the complete
.newsrc.eld file.

> It really seems like getting all the marks out of .newsrc.eld
> and putting them in the backends would unify many ideas in Gnus.

And going in the other direction, which Ma Gnus is, also unifies many
things.  :-)

As for the "merging different views" -- that's not going to happen in
any meaningful way.  If you're reading stuff offline on two different
machines (without ever going online), one client is going to win when
you're finally going online.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Rome

Re: I can haz cloud idea

[Dan Christensen]

I really think syncing an entire .newsrc.eld file is the wrong approach.

On my home machine, I read lots of nntp groups, which I never want to
read on other machines, so syncing that data is inefficient and risks
data loss if the remote machine updates the .newsrc.eld after I've
done some reading on the local machine.

Similarly, I don't want my imap groups synced, since imap already
handles that.  Syncing this too would be inefficient and risk data loss.

This definitely should be configured on a server by server basis.

Lars Ingebrigtsen <larsi@gnus.org> writes:

> As for the "merging different views" -- that's not going to happen in
> any meaningful way.  If you're reading stuff offline on two different
> machines (without ever going online), one client is going to win when
> you're finally going online.

If instead of diffs you upload actions like "(mark-read 12 (34 . 37) 44)"
then you get lots of advantages:

- very small amount of data to upload:  a diff would include the old
  data, all the marks that didn't change, and context lines 
- diffs are fragile, as they might not apply cleanly if things get
  out of sync
- this method automatically handles merging to some degree, as instead
  of uploading all marks for a group it only sends the changes

When Gnus wants to get the marks from the server, it'll download a 
baseline list of marks and all of the actions, and apply them
dynamically.  

Eventually the list of actions will get long and the baseline marks
will get rewritten, but this could happen infrequently enough that
the risks of overwriting other data is small.

But I still think using Ted's approach might be better, especially since
I think he offered to host a server.

Dan

Re: I can haz cloud idea

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> On my home machine, I read lots of nntp groups, which I never want to
> read on other machines, so syncing that data is inefficient and risks
> data loss if the remote machine updates the .newsrc.eld after I've
> done some reading on the local machine.

And the same goes for nnml/nnfolder groups, which would be useless to
sync.  So some sort of control would be necessary here to determine what
parts of the .newsrc.eld data to share.

> Similarly, I don't want my imap groups synced, since imap already
> handles that.  Syncing this too would be inefficient and risk data loss.

Well, yes and no.  The .newsrc.eld file keeps track of sequence numbers
(in the QRESYNC case) and max seen articles (in the non-QRESYNC) case.
Syncing with an IMAP server involves a lot of network traffic, and the
longer since you've synced, the more data has to be transferred.

There's a large data gain to be had by syncing the .newsrc.eld data for
nnimap groups too, I think.

>> As for the "merging different views" -- that's not going to happen in
>> any meaningful way.  If you're reading stuff offline on two different
>> machines (without ever going online), one client is going to win when
>> you're finally going online.
>
> If instead of diffs you upload actions like "(mark-read 12 (34 . 37) 44)"
> then you get lots of advantages:
>
> - very small amount of data to upload:  a diff would include the old
>   data, all the marks that didn't change, and context lines 
> - diffs are fragile, as they might not apply cleanly if things get
>   out of sync
> - this method automatically handles merging to some degree, as instead
>   of uploading all marks for a group it only sends the changes

Yes, I think that's what I meant by "diffs".  :-)  But what I meant was
that if you mark an article as unread in one Gnus, and then read in
another, without going online, one of them is going to win when you
finally go online.  For that particular article.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: I can haz cloud idea

[Lars Ingebrigtsen]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> So keep everything in simple self consistent local files, then use DVCS
> mechanisms to propagate versions.

That doesn't sound very user-friendly.

> I hope you feel differently when you've sobered up :-)

Nope.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Rome

Re: I can haz cloud idea

[Lars Ingebrigtsen]

As an example of stuff that should go into the .newsrc.eld file is user
settings like topic mode.  If the user taps `t', then topic mode should
be switched on the next time Gnus starts up without the user having to
put 

(add-hook 'gnus-group-mode-hook 'gnus-topic-mode)

into .gnus.el, which is just absurd.

So there will be a new section in the .newsrc.eld file for "saved state"
kind of things:

(setq gnus-saved-state '((topics t)
                         (offline nil)))

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Rome

Re: I can haz cloud idea

[Adam Sjøgren]

On Sun, 19 Feb 2012 16:21:51 +0100, Lars wrote:

> The same goes for other obvious stuff like ~/.signature.

I like to generate my .signature using a script that writes a file; nice
and simple.

(Surely a detail; just sayin'...)


  Best regards,

    Adam

-- 
 "My internal clock is on Tokyo time."                        Adam Sjøgren
                                                         asjo@koldfront.dk

Re: I can haz cloud idea

[Ted Zlatanov]

On Sun, 19 Feb 2012 16:21:51 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> So the original idea would be to have a bundle of files distributed via
LI> IMAP, plus diffs to avoid down/uploading so much stuff.  When stuff
LI> changes in the bundle, Gnus updates the appropriate files.

The actual amount of data for a group, as a JSON object in LeSync, is
tiny.  I would avoid deltas if possible.  They are fragile, hard to
debug, and unshareable with other MUAs.  Put all the data a group needs
in a single place (a message in the mailbox, in this case, or a CouchDB
document in LeSync) and rewrite the whole document atomically when it
changes.  Yes, you have a versioning problem, if someone else writes
before you.  So use a revision number.  Please look at how CouchDB does
it: only save the document if the latest document's version matches the
latest version you know.  Otherwise, reconcile with the latest saved.

LI> Why rely on other files for the significant stuff?  We control the
LI> vertical and the horizontal, so why not just rethink this?

LI> The score files, for instance, would more naturally be a structure
LI> stored in .newsrc.eld.  Finding out what score files that match what
LI> group is pretty gross code-wise.  Why not just make that into a nice
LI> alist and put it in the .newsrc.eld file?

LI> The same goes for other obvious stuff like ~/.signature.

LI> Just put it all in .newsrc.eld, and then distribute that via IMAP.

I would have the gnus-sync backend store user profiles, like the posting
styles in Gnus.  The profile would point to or inline the signature
file, score files, SMTP server, and everything else the user needs, as
customizations.  Then you pick a profile on each machine you use.

I would make each profile, with all the customizations, a single
document that users can edit and understand.  Don't spread things out
just because you can.

LI> Ted Zlatanov <tzz@lifelogs.com> writes:

>> Why not add a gnus-sync backend to do this?

LI> Because I want it all to be asynchronous, opportunistic and streaming.

That's fine.  The gnus-sync backend should support that when you call
`gnus-sync-save'.  You're taking the mechanics, the way code will
operate, and using that as an argument against the architecture, the way
data and user-customizable variables are planned.

With CouchDB you can stream by opening a persistent connection to the
server.  url.el can be asynchronous, and so can you against CouchDB.
I'm not sure what "opportunistic" means exactly, but if you mean "save
when possible" then I think that would work.

Just realize that by setting the gnus-sync backend to files, you can be
backwards compatible with the current Gnus.  That way you don't have to
have a special "cloud mode" but simply "sync to local filesystem."  Your
way is a special case all the way from data layout to mechanics.

LI> When exiting a group, we'll be squirting off an APPEND command to the
LI> IMAP server (with the current delta), but we don't wait for it to
LI> return finish.  We queue them up, and if we get three group exits before
LI> the first APPEND has finished, we just skip the second one.

LI> Conversely, using IDLE on the mail box will keep the other instances
LI> updated asynchronously without anybody having to issue any commands.

LI> It'll be like using the IMAP synchronisation methodology, only for all
LI> backends and stuff.

OK.  But that's nearly impossible to share with other MUAs, unlike my
LeSync proposal, and I hope you don't use deltas as I said above.

>> The CouchDB gnus-sync backend has built-in document versioning, which I
>> think is really nice.

LI> I don't think having CouchDB as a prerequisite is totally reasonable.
LI> It should just work without having to install anything.

You're basing your side on a working, always-reachable IMAP server,
which is less reasonable in the context of the Internet and requires
significant infrastructure.  Port 80 is much more likely to be open than
other ports, especially in a corporate environment.

I didn't decide on CouchDB because it's trendy (it's SO 2009!), but
because it's a schemaless database+web server hybrid with custom views
(e.g. I can put together a XML view of the marks that NewsTap can import
directly).  You can get a free instance at http://iriscouch.com or
https://cloudant.com and get running in minutes, or set your own up even
quicker, or I can set one up for you at WHATEVER.lesync.info.

Ted

Re: I can haz cloud idea

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LI> It'll be like using the IMAP synchronisation methodology, only for all
> LI> backends and stuff.
>
> OK.  But that's nearly impossible to share with other MUAs, unlike my
> LeSync proposal, and I hope you don't use deltas as I said above.

I don't think sharing this data with other MUAs should be a primary
design goal here...

> You're basing your side on a working, always-reachable IMAP server,
> which is less reasonable in the context of the Internet and requires
> significant infrastructure.

If you can't reach the IMAP server, you can't read your email, which
makes the whole thing somewhat moot.

> Port 80 is much more likely to be open than other ports, especially in
> a corporate environment.

The use case here (which I think is the typical use case, these days) is
"I read email all the time, and then I read a couple of groups from
eternal-news, and 74 groups from news.gmane.org, and I want those extra
groups to sync just like my mail".  Like magic, without having to know
or set up anything else.

> I didn't decide on CouchDB because it's trendy (it's SO 2009!),

But it has web scale!

Or was that MongoDB?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: I can haz cloud idea

[Reiner Steib]

On Sat, Mar 10 2012, Lars Magne Ingebrigtsen wrote:

> But it has web scale!
>
> Or was that MongoDB?

http://www.mongodb-is-web-scale.com/ ;-)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: I can haz cloud idea

[Ted Zlatanov]

On Sat, 10 Mar 2012 02:12:29 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
LI> It'll be like using the IMAP synchronisation methodology, only for all
LI> backends and stuff.
>> 
>> OK.  But that's nearly impossible to share with other MUAs, unlike my
>> LeSync proposal, and I hope you don't use deltas as I said above.

LMI> I don't think sharing this data with other MUAs should be a primary
LMI> design goal here...

LMI> The use case here (which I think is the typical use case, these days) is
LMI> "I read email all the time, and then I read a couple of groups from
LMI> eternal-news, and 74 groups from news.gmane.org, and I want those extra
LMI> groups to sync just like my mail".  Like magic, without having to know
LMI> or set up anything else.

OK, you're solving a different problem.  Mine is "I want to read and
sync NNTP groups from various MUAs."

I'll keep maintaining gnus-sync.el separately from you, so I can keep it
open to other MUAs.

Ted

Re: I can haz cloud idea

[Ted Zlatanov]

On Sat, 10 Mar 2012 08:06:13 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Sat, 10 Mar 2012 02:12:29 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 
LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
LI> It'll be like using the IMAP synchronisation methodology, only for all
LI> backends and stuff.
>>> 
>>> OK.  But that's nearly impossible to share with other MUAs, unlike my
>>> LeSync proposal, and I hope you don't use deltas as I said above.

LMI> I don't think sharing this data with other MUAs should be a primary
LMI> design goal here...

LMI> The use case here (which I think is the typical use case, these days) is
LMI> "I read email all the time, and then I read a couple of groups from
LMI> eternal-news, and 74 groups from news.gmane.org, and I want those extra
LMI> groups to sync just like my mail".  Like magic, without having to know
LMI> or set up anything else.

TZ> OK, you're solving a different problem.  Mine is "I want to read and
TZ> sync NNTP groups from various MUAs."

TZ> I'll keep maintaining gnus-sync.el separately from you, so I can keep it
TZ> open to other MUAs.

Due to external commitments I'm not working on this for a while, but
wanted to write a followup so people can comment.  I think general
"cloudiness" is a good thing nowadays, but sacrificing Lisp's power to
be compatible with less powerful MUAs is not.  So a Lispy cloudiness is
a better goal, perhaps implemented with an IMAP backend.  All of the
Emacs ecosystem could benefit from such functionality: BBDB, general
data storage, auth-sources, .emacs and installed ELPA packages, and so
on.  But even just for Gnus it's a good thing.

I've been using CouchDB a lot with gnus-sync.el, and think you were
right about relying on CouchDB itself.  It's a pain in the long term,
like any database.  It's not Lispy at all, either.

I'd like to work with you set up an IMAP sync backend, perhaps driven by
gnus-sync.el, and rework gnus-sync.el as needed.  The file backend
should stay if possible; the LeSync protocol should simply become a Lisp
data merge (we'll give the users the tools to upgrade).

The main reason I agree is that I think the Lisp to JSON and back
conversion is not good.  Lisp data is so much more capable than JSON,
that flattening it into JSON makes sync very complicated.  Also I think
there's no benefit to sharing that data with other MUAs, because the
standalone MUA is a dying breed.  Web-based MUAs are the norm now, and
they don't share well... unless you use IMAP, in which case Gnus works
fine already.  For NNTP marks, the need for sharing is just not there.

Ted

Re: I can haz cloud idea

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Due to external commitments I'm not working on this for a while, but
> wanted to write a followup so people can comment.  I think general
> "cloudiness" is a good thing nowadays, but sacrificing Lisp's power to
> be compatible with less powerful MUAs is not.

I agree.

> So a Lispy cloudiness is a better goal, perhaps implemented with an
> IMAP backend.

I agree.

> All of the Emacs ecosystem could benefit from such functionality:
> BBDB, general data storage, auth-sources, .emacs and installed ELPA
> packages, and so on.  But even just for Gnus it's a good thing.

I agree (again...).

> I've been using CouchDB a lot with gnus-sync.el, and think you were
> right about relying on CouchDB itself.  It's a pain in the long term,
> like any database.

Yep.  The current problem I have (when traveling) is getting auth to
work, so I don't have to do "Allow from" subnets.

> It's not Lispy at all, either.

Nope.

> I'd like to work with you set up an IMAP sync backend, perhaps driven by
> gnus-sync.el, and rework gnus-sync.el as needed.  The file backend
> should stay if possible; the LeSync protocol should simply become a Lisp
> data merge (we'll give the users the tools to upgrade).

I like this. :-)
I'm volunteering to try this out as an early adopter...:-)

Re: I can haz cloud idea

[David Engster]

Lars Ingebrigtsen writes:
> But how about if we just cheat and put all the information on the IMAP
> server?

I think it's a good idea, but I'd like it even more if this could be
somehow hooked into some existing framework for storing data remotely,
and that framework in Emacs is Tramp.

Ted coded IMAP support for Tramp a while ago, but it was removed again
for reasons I don't know.

-David

Re: I can haz cloud idea

[Lars Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> I think it's a good idea, but I'd like it even more if this could be
> somehow hooked into some existing framework for storing data remotely,
> and that framework in Emacs is Tramp.

I don't really see the point of using Tramp.  nnimap is perfectly
capable of up/downloading stuff on its own.  And this has to be done
with a high degree of asynchronocity and waiting on IDLE, etc, so Tramp
would just get in the way.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Rome

Re: I can haz cloud idea

[David Engster]

Lars Ingebrigtsen writes:
> David Engster <deng@randomsample.de> writes:
>
>> I think it's a good idea, but I'd like it even more if this could be
>> somehow hooked into some existing framework for storing data remotely,
>> and that framework in Emacs is Tramp.
>
> I don't really see the point of using Tramp.

It would make things configurable. People could easily switch between
imap, ssh, rsync, etc. for syncing their Gnus stuff.

>  nnimap is perfectly capable of up/downloading stuff on its own.  And
> this has to be done with a high degree of asynchronocity and waiting
> on IDLE, etc, so Tramp would just get in the way.

Yes. In any case, I think IMAP is the way to go; configurability is
nice'n'all, but if it is too much pain, just use plain nnimap.

-David

Re: I can haz cloud idea

[Steinar Bang]

>>>>> David Engster <deng@randomsample.de>:

> It would make things configurable. People could easily switch between
> imap, ssh, rsync, etc. for syncing their Gnus stuff.

Speaking as the current gnus-sync user base: this isn't configurability
that the user base wants, or needs.

Re: I can haz cloud idea

[Steinar Bang]

>>>>> David Engster <deng@randomsample.de>:

> I think it's a good idea, but I'd like it even more if this could be
> somehow hooked into some existing framework for storing data remotely,
> and that framework in Emacs is Tramp.

The first version of gnus-sync used tramp, and I used an SSH URL with
tramp to do the syncing.  Tramp was slooooow.

The current gnus-sync, which uses an HTTP RESTful protocol (the CouchDB
protocol), is lightning fast in comparison.

> Ted coded IMAP support for Tramp a while ago, but it was removed again
> for reasons I don't know.

I think it was because it had many issues and few users...?

Re: I can haz cloud idea

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]


David Engster <deng@randomsample.de> writes:

> Lars Ingebrigtsen writes:
>> But how about if we just cheat and put all the information on the IMAP
>> server?
>
> I think it's a good idea, but I'd like it even more if this could be
> somehow hooked into some existing framework for storing data remotely,
> and that framework in Emacs is Tramp.
>
> Ted coded IMAP support for Tramp a while ago, but it was removed again
> for reasons I don't know.

I thin the key question here is

  Is what gnus-sync needs just a network filesystem, or something
  special?

If it works with a filesystem, then one can use all sorts of things,
depending on what makes sense.  Some people might want sync but not have
an IMAP account (but then they probably have a news-only use case).  But
IMAP seems like a good place, as many people will already have it set
up.

I think the hard part is the conflict/merge model, for use with clients
in different places.


[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: I can haz cloud idea

[David Engster]

Greg Troxel writes:
> David Engster <deng@randomsample.de> writes:
>
>> Lars Ingebrigtsen writes:
>>> But how about if we just cheat and put all the information on the IMAP
>>> server?
>>
>> I think it's a good idea, but I'd like it even more if this could be
>> somehow hooked into some existing framework for storing data remotely,
>> and that framework in Emacs is Tramp.
>>
>> Ted coded IMAP support for Tramp a while ago, but it was removed again
>> for reasons I don't know.
>
> I thin the key question here is
>
>   Is what gnus-sync needs just a network filesystem, or something
>   special?

Yes, it's probably leaning toward 'something special'. What I like about
the file-based approach is that you can easily do backups. Having
nightly versions of my .newsrc has saved me hours of work several times
already.

-David

Re: I can haz cloud idea

[Ted Zlatanov]

On Thu, 16 Feb 2012 06:55:53 -0500 Greg Troxel <gdt@work.lexort.com> wrote: 

GT> I thin the key question here is

GT>   Is what gnus-sync needs just a network filesystem, or something
GT>   special?

GT> If it works with a filesystem, then one can use all sorts of things,
GT> depending on what makes sense.  Some people might want sync but not have
GT> an IMAP account (but then they probably have a news-only use case).  But
GT> IMAP seems like a good place, as many people will already have it set
GT> up.

GT> I think the hard part is the conflict/merge model, for use with clients
GT> in different places.

gnus-sync.el works against backends.  The idea is to be flexible, so it
should work against any storage.  CouchDB happens to be the best
schemaless document storage available today for our purposes, especially
because of the built-in REST methods but also because JSON is a good
portable format.

Ted

Re: I can haz cloud idea

[Ted Zlatanov]

On Thu, 16 Feb 2012 09:25:45 +0100 David Engster <deng@randomsample.de> wrote: 

DE> Ted coded IMAP support for Tramp a while ago, but it was removed again
DE> for reasons I don't know.

There were no users of that code, so I didn't think it worth the effort.

Ted

Re: I can haz cloud idea

[Ted Zlatanov]

On Thu, 16 Feb 2012 06:03:47 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> The main problems with going all cloudy is that you have to have yet
LI> another server out there somewhere.  Whether you use Dropbox/sshfs/a yet
LI> to be determined Gnus-specific server, that's not very sexy.

LI> But how about if we just cheat and put all the information on the IMAP
LI> server?

LI> That is: Gnus would create and maintain a special group on one of your
LI> IMAP servers that would contain all the data Gnus needs.

LI> You'd set this up by setting, er, something, like:

LI> (setq gnus-cloud-server '(nnimap "imap.gmail.com"))

Why not add a gnus-sync backend to do this?

LI> This would make Gnus save all the data on that server whenever you exit
LI> Gnus.  (Or on certain intervals.)  It could gzip and encrypt the data
LI> before uploading, I guess.

LI> So when you go to your other machine, you'd say `M-x gnus-cloud', and it
LI> would prompt you for the server and password(s), download the data, and
LI> then start up more or less as normal.

LI> And it would upload the data on exit, as well, so you'd be all
LI> cloudy.

This is what `gnus-sync-read' and `gnus-sync-save' do already, for some of
the newsrc data.  There's no problem with storing more, my only issue
was to figure out how to synchronize it all cleanly.

I also want to use gnus-sync to store the gnus-registry and even BBDB
data.  So it makes sense to make this a general facility.

LI> What would be stored on the server?  Well, the .newsrc.eld file, for
LI> one.  And the SCORE files, I guess.  And...  well, whatever else you
LI> want.  It could be a simple archive "file" type, and Gnus could just
LI> decrypt, uncompress and unpack the files.  Some sanity check about
LI> overwriting newer files with older files, I guess.

The CouchDB gnus-sync backend has built-in document versioning, which I
think is really nice.  Also it's portable through JSON and HTTP (what I
call the LeSync protocol), so other newsreaders can support it too.  The
developer of NewsTap, a iOS app, is on board with this and told me it's
on top of his feature list.

LI> If this is to work smoothly (i.e., at all), then every time you hit `g'
LI> in a Gnus, it'll have to know whether something else updated the state
LI> group.  This also means that every time you exit a group or does
LI> something else that changes state, it'll have to be uploaded pretty much
LI> immediately.  This also means that there would be no explicit "upload"
LI> commands.

So basically a `gnus-sync-save' on group exit.  I would amortize it so
it's not done so frequently (currently, gnus-sync saves when the newsrc
is saved) because network transactions are expensive.

LI> Here's how I see this working:

LI> When you first start the thing, it does a total upload.  Then, every
LI> time you (say) exit a group, it only uploads the delta between the
LI> previous total upload and the current state.  So after you've read a
LI> couple of groups, the delta file will look like

LI> ("gwene.com.feedburner.cheezburger" (add (read (56 . 59))) (del (tick 58)))

LI> or something.  Gnus will continue to upload the entire delta file to the
LI> server on each group exit (and delete any previous deltas).  Until the
LI> delta gets so big that Gnus just does a total upload again, and removes
LI> all previous data.

LI> It sucks that `g' has to query the state group, though.  But it really
LI> doesn't.  If we maintain a separate cloud connection, then we can just
LI> IDLE on the state group.  Then every time a different Gnus somewhere
LI> does something, then the other Gnus instances can just download the
LI> state change, and update itself automatically.  Totally transparently
LI> and in the background.

LI> Unless the IMAP server has dropped the connection, of course.  Then `g'
LI> will have to reconnect and do the update while you wait.

With CouchDB, at least, you can do a differential poll: tell me what has
changed since change X (integer numeric sequence).  You can also open a
persistent connection and get changes live when they are made.  So this
could be, again, a gnus-sync feature per backend, and the gnus-sync
LeSync and IMAP backends would support live updates while the file
backend wouldn't.

CouchDB's document model is interesting: you have to read the current
revision to write a new one back.  So you can't accidentally overwrite a
document, you have to read it first (which gives you a chance to
merge).  So at least for CouchDB/LeSync you don't need to do delta
files.

Ted

Re: I can haz cloud idea

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Why not add a gnus-sync backend to do this?

Because I want it all to be asynchronous, opportunistic and streaming.

When exiting a group, we'll be squirting off an APPEND command to the
IMAP server (with the current delta), but we don't wait for it to
return finish.  We queue them up, and if we get three group exits before
the first APPEND has finished, we just skip the second one.

Conversely, using IDLE on the mail box will keep the other instances
updated asynchronously without anybody having to issue any commands.

It'll be like using the IMAP synchronisation methodology, only for all
backends and stuff.

> The CouchDB gnus-sync backend has built-in document versioning, which I
> think is really nice.

I don't think having CouchDB as a prerequisite is totally reasonable.
It should just work without having to install anything.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Rome

Re: I can haz cloud idea

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:

> I don't think having CouchDB as a prerequisite is totally reasonable.
> It should just work without having to install anything.

Even though I have set up CouchDB myself, I tend to agree: one extra
server, one separate type of connection that has to work, one extra set
of authentication info that has to be supplied

I think relying on having an IMAP server available, for this to work, is
reasonable.

I think writing something as close to the metal of nnimap will boost
performance.  I have tried running gnus-sync with tramp as the transport
mechanism, and that was slow (tramp over ssh.  I never got around to try
tramp over IMAP).

I also think that sending the full .newsrc.eld in gzipped form is
something that should be explored.

My .newsrc.eld on this netbook is 780kB, which isn't big by today's
network usage standards.  Using "gzip --best" brings it down to 140kB,
which is... nothing, really.

(of course sending the entire .newsrc.eld in gzipped form would break
the opportunistic streaming bit...)

Re: I can haz cloud idea

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> I also think that sending the full .newsrc.eld in gzipped form is
> something that should be explored.
>
> My .newsrc.eld on this netbook is 780kB, which isn't big by today's
> network usage standards.  Using "gzip --best" brings it down to 140kB,
> which is... nothing, really.

It's not much, but it would be nice to keep the network traffic down as
much as possible, in these 3G days...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: I can haz cloud idea

[TSUCHIYA Masatoshi]

>> On Thu, 16 Feb 2012 06:03:47 +0100
>> larsi@gnus.org (Lars Ingebrigtsen) said as follows:

>The main problems with going all cloudy is that you have to have yet
>another server out there somewhere.  Whether you use Dropbox/sshfs/a yet
>to be determined Gnus-specific server, that's not very sexy.

>But how about if we just cheat and put all the information on the IMAP
>server?

It sounds good.

First of all, I want putting my draft box and an addressbook on the IMAP
server.

I hope that your Gnus cloud implementation may provide generic API to
share files between machines.  If such API will be available, I will be
able to share the bookmark file and the list of visited URLs on
emacs-w3m between machines.

-- 
TSUCHIYA Masatoshi

Re: I can haz cloud idea

[Lars Magne Ingebrigtsen]

TSUCHIYA Masatoshi <tsuchiya@namazu.org> writes:

> First of all, I want putting my draft box and an addressbook on the IMAP
> server.

Yes, and Gnus should also start to defaulting to putting the archive
group on the IMAP server (if the user is using nnimap at all).

> I hope that your Gnus cloud implementation may provide generic API to
> share files between machines.  If such API will be available, I will be
> able to share the bookmark file and the list of visited URLs on
> emacs-w3m between machines.

That was more or less my original idea, but, er, implementing Dropbox
via IMAP had more conceptual sticky points that I liked.  So I'm not
currently thinking about doing a generic API.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: I can haz cloud idea

[TSUCHIYA Masatoshi]

>> On Thu, 16 Feb 2012 06:03:47 +0100
>> larsi@gnus.org (Lars Ingebrigtsen) said as follows:

>The main problems with going all cloudy is that you have to have yet
>another server out there somewhere.  Whether you use Dropbox/sshfs/a
>yet to be determined Gnus-specific server, that's not very sexy.

>But how about if we just cheat and put all the information on the IMAP
>server?

I use the following code to put my sent box on IMAP server.

(let ((imap-method
       (catch 'found-imap-method
	 (let ((methods (cons gnus-select-method
			      gnus-secondary-select-methods)))
	   (while methods
	     (when (eq (caar methods) 'nnimap)
	       (throw 'found-imap-method (car methods)))
	     (setq methods (cdr methods)))))))
  (when imap-method
    (setq gnus-message-archive-method imap-method
	  gnus-message-archive-group "Sent")))

I can share the same sent box between Gnus and other MUAs using this

I also tried a similar trick to put my draft box on IMAP server, but
failed.

-- 
TSUCHIYA Masatoshi

Re: I can haz cloud idea

[Steinar Bang]

>>>>> TSUCHIYA Masatoshi <tsuchiya@namazu.org>:

> I also tried a similar trick to put my draft box on IMAP server, but
> failed.

This could be because IMAP isn't rewritable...?

Ie. on IMAP you would need to delete the old version of the message and
write a new message with a new ID.

Re: I can haz cloud idea

[Ted Zlatanov]

On Thu, 16 Oct 2014 08:31:54 +0900 TSUCHIYA Masatoshi <tsuchiya@namazu.org> wrote: 

>>> On Thu, 16 Feb 2012 06:03:47 +0100
>>> larsi@gnus.org (Lars Ingebrigtsen) said as follows:

>> But how about if we just cheat and put all the information on the IMAP
>> server?

TM> I use the following code to put my sent box on IMAP server.

Lars was talking about more than mail, we need all the metadata.
Basically your newsrc.eld.

Ted