Gnus Cloud

Gnus Cloud

[Lars Ingebrigtsen]

So, Paso Robles (well, the wines as Justin) inspired me to do some more
thinking about Gnus syncing and stuff.

I'm juggling two laptops, as well as my home account when reading
news/mail now.  And I think these sort of syncing problems are going to
get to be a bigger deal this summer, when the new MeeGo phones are being
released.  They're real Linux machines, so you have X and Emacs running
on your phone.  So you (i.e., I) need to have a way to keep stuff in
sync.  And since the networks are mainly going to be 3G, we need to keep
the data being synced as small as possible.

So here's my current thought:  The Gnus Cloud thing needs to be as
non-general as possible.  We need to sync only the things that need
syncing, and nothing more.

This is how it would work.

You cloud-enable the servers/groups you're interested in having be
device-independent.  For me, this would be Gmane/Gwene groups, as well
as my home IMAP server.

Gnus will then ping the new RESTful cloud.gnus.org server with updates
on new groups and marks.  From the Gnus side, this will look pretty much
like what's Gnus is doing now with IMAP -- we tell the server what we've
read, and what groups we're (un)subscribing.  Signing up for the server
would be a zero-click process.  I.e., you just say what your cloud user
name and password is, and it'll be created if it doesn't exist, and if
it does, Gnus will keep the account updated.

Then if you say `M-x gnus-on-the-go' on your cell phone, and it's a new
installation, it'll just prompt you for user name/password, and download
pretty much the entire .newsrc.eld file (well, the cloud-enabled
portion, at least).  Then subsequent updates will be on the form "send
me all deltas since timestamp YYYYMMDDTHHMMS".  The server will figure
out the diff between that timestamp and the current state, create a nice
data package, gzip it up, and squirt it out.

So the idea is that the cloud server will be a sort of journaled file
system, where it knows the current state, and knows all the actions
taken to get back to an arbitrary point in time, so it can generate
group/marks deltas that Gnus can read to get everything updated.  I
think this can be implemented pretty efficiently, and Gnus can slurp in
the new data fast.

Keeping the server updated would be, as I said, mostly a matter of
hooking into the backend marks updating subsystem.  But a nice wrinkle
here is that these are basically ordered transactions, so you can queue
them up (in a file, for instance), and then squirt them over
asynchronously at Gnus's leisure.  For instance, you might be using Gnus
offline, and Gnus would squirt stuff over when you go online.  And if
cloud.gnus.org is down, Gnus can just queue stuff and send it when it's
back up.  This stuff can be implemented in a way that's fairly robust to
transient failures.

I think the things that should be kept updated are:

* server definitions
* groups
* marks
* topic topology
* score files
* some group parameters

We'd need to be somewhat careful about what we're putting in the cloud.
For instance, some group parameters are eval-ed, and it would not be
nice to provide a way to hack all Gnus users remotely by hacking
cloud.gnus.org.

One sticky issue is whether to sync passwords.  .authinfo stuff is
necessary to contact some servers, but passwords are way more secret
than the rest of the data.  Even if cloud.gnus.org is https only, it
would still be a juicy hacking target if we just stored passwords (in
clear text) there.  Put perhaps we could keep .authinfo.gpg stuff
synced?  I think that area needs more thought, but the rest seems fairly
straightforward to me.

And one nice thing about this setup is that there really would be no
administration or data loss importance (for me).  Since cloud.gnus.org
is basically just a cache of data you have elsewhere, if you lose your
cloud password, you can just create a new account.  If the service loses
your data -- the same thing.

Oops.  I think somebody wants to do something touristy...  gotta run...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gnus Cloud

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1170 bytes --]

On Wed, Feb 09 2011, Lars Ingebrigtsen wrote:

> So, Paso Robles (well, the wines as Justin) inspired me to do some more
> thinking about Gnus syncing and stuff.
>
> I'm juggling two laptops, as well as my home account when reading
> news/mail now.  And I think these sort of syncing problems are going to
> get to be a bigger deal this summer, when the new MeeGo phones are being
> released. 

You can stop here:

    http://lwn.net/Articles/427035/rss

> You cloud-enable the servers/groups you're interested in having be
> device-independent.  For me, this would be Gmane/Gwene groups, as well
> as my home IMAP server.

The cloud needs to know IMAP has already a subscription methods in its
protocol.

The rest does not interest me. I will never trust any external service I
do not run to store any personal information.
And I think there's much important thing to do/fix in Gnus than such a
thing. :)

And developing a cloud service to store a .el file is not something I
consider worth it; people using things like Org have already to find a
solution to store and share their data.

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Gnus Cloud

[Ted Zlatanov]

On Wed, 09 Feb 2011 11:24:37 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Then if you say `M-x gnus-on-the-go' on your cell phone, and it's a new
LI> installation, it'll just prompt you for user name/password, and download
LI> pretty much the entire .newsrc.eld file (well, the cloud-enabled
LI> portion, at least).  Then subsequent updates will be on the form "send
LI> me all deltas since timestamp YYYYMMDDTHHMMS".  The server will figure
LI> out the diff between that timestamp and the current state, create a nice
LI> data package, gzip it up, and squirt it out.

Just do a generic "send me all deltas since X" where X can be 0.  So
then the client state is just (current data, last check).

The time should be in microseconds.

LI> Keeping the server updated would be, as I said, mostly a matter of
LI> hooking into the backend marks updating subsystem.  But a nice wrinkle
LI> here is that these are basically ordered transactions, so you can queue
LI> them up (in a file, for instance), and then squirt them over
LI> asynchronously at Gnus's leisure.  For instance, you might be using Gnus
LI> offline, and Gnus would squirt stuff over when you go online.  And if
LI> cloud.gnus.org is down, Gnus can just queue stuff and send it when it's
LI> back up.  This stuff can be implemented in a way that's fairly robust to
LI> transient failures.

LI> I think the things that should be kept updated are:

LI> * server definitions
LI> * groups
LI> * marks
LI> * topic topology
LI> * score files
LI> * some group parameters

Yes to all of these.

LI> One sticky issue is whether to sync passwords.  .authinfo stuff is
LI> necessary to contact some servers, but passwords are way more secret
LI> than the rest of the data.  Even if cloud.gnus.org is https only, it
LI> would still be a juicy hacking target if we just stored passwords (in
LI> clear text) there.  Put perhaps we could keep .authinfo.gpg stuff
LI> synced?  I think that area needs more thought, but the rest seems fairly
LI> straightforward to me.

A .GPG file could be stored.  It wouldn't be synchronized, just set/get.

LI> And one nice thing about this setup is that there really would be no
LI> administration or data loss importance (for me).  Since cloud.gnus.org
LI> is basically just a cache of data you have elsewhere, if you lose your
LI> cloud password, you can just create a new account.  If the service loses
LI> your data -- the same thing.

As long as there's a way to "back up all my data to a file" and "write
all my data back from a file" I think that's OK.

On Wed, 09 Feb 2011 21:22:08 +0100 Julien Danjou <julien@danjou.info> wrote: 

JD> On Wed, Feb 09 2011, Lars Ingebrigtsen wrote:

JD> The rest does not interest me. I will never trust any external service I
JD> do not run to store any personal information.

I think it should be easy (trivial) to run this service yourself, like
you would an IMAP server.

JD> And I think there's much important thing to do/fix in Gnus than such a
JD> thing. :)

I think this is very important, personally.  I would put it on top of my
priority list, although I had a slightly different proposal earlier
(Lars' proposal is purely data-driven and confined to Gnus only, but I
can fix that :)

Ted

Re: Gnus Cloud

[Lars Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> You can stop here:
>
>     http://lwn.net/Articles/427035/rss

He didn't actually say that they're dropping MeeGo.  :-)

> The cloud needs to know IMAP has already a subscription methods in its
> protocol.

Not really.  IMAP syncing totally sucks bandwidth-wise, so the less you
can sync via IMAP, the better.

> The rest does not interest me. I will never trust any external service I
> do not run to store any personal information.

I don't really consider anything that would be stored in the Gnus Cloud
to be particularly personal.  (For me.)  It's just (basically) a list of
the groups I read, and not much more.

Other people might be ashamed of reading
gmane.comp.windows.windows95.fans, though.

> And I think there's much important thing to do/fix in Gnus than such a
> thing. :)

Like what?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gnus Cloud

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Just do a generic "send me all deltas since X" where X can be 0.  So
> then the client state is just (current data, last check).

Yes, that's what I meant.

> The time should be in microseconds.

Probably.  Although re-sending deltas wouldn't hurt Gnus.  If you send
the data "article 45 has been read" twice, Gnus doesn't really care.

> A .GPG file could be stored.  It wouldn't be synchronized, just set/get.

Well, it should be updated when it's updated on one of the Gnus
installations.

You might also envision totally storage-less Gnus installations, sort
of.  That is, `M-x gnus-ephemeral-on-the-go' that would just get the
data from the cloud, read a bit, and then exit without saving anything
locally.  If you're afraid storing your secret .newsrc.eld/.authinfo.gpg
files on your device.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gnus Cloud

[Ted Zlatanov]

On Wed, 09 Feb 2011 17:02:53 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:

>> A .GPG file could be stored.  It wouldn't be synchronized, just set/get.

LI> Well, it should be updated when it's updated on one of the Gnus
LI> installations.

Right.  So I'd allow for a generic "save a file to the cloud" facility
that doesn't have any of the deltas.

LI> You might also envision totally storage-less Gnus installations, sort
LI> of.  That is, `M-x gnus-ephemeral-on-the-go' that would just get the
LI> data from the cloud, read a bit, and then exit without saving anything
LI> locally.  If you're afraid storing your secret .newsrc.eld/.authinfo.gpg
LI> files on your device.

Basically Google Reader.  I don't like that as much as just
synchronizing the configuration, although I'm sure some would like it.
It seems like an extreme compared to just synchronizing the things you
listed.

I would make a final Gnus release of the current series and start a new
"Cloudy" or "Cumulonimbus Gnus" whichever way you go.  FWIW I'd make it
flexible enough that people can sync just their marks or go all the way
up to 100% synchronized configuration.

Ted

Re: Gnus Cloud

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

On Wed, Feb 09 2011, Ted Zlatanov wrote:

> JD> And I think there's much important thing to do/fix in Gnus than such a
> JD> thing. :)
>
> I think this is very important, personally.  I would put it on top of my
> priority list, although I had a slightly different proposal earlier
> (Lars' proposal is purely data-driven and confined to Gnus only, but I
> can fix that :)

Ahah, that'd be fun.

A NoSQL database like CouchDB, but replacing JSON with Lisp?

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Gnus Cloud

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 350 bytes --]

On Thu, Feb 10 2011, Lars Ingebrigtsen wrote:

> Like what?  :-)

Make IMAP partial article download and mime rendering works together, allow Gnus agent to sync both ways [1]

gnus/todo? :-)

I probably can find more. :-)

[1]  That just the opposite of making Gnus cloudy I think. :-)

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Gnus Cloud

[Philipp Haselwarter]

"TZ" == Ted Zlatanov <tzz@lifelogs.com> writes:

TZ> On Wed, 09 Feb 2011 17:02:53 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote:

LI> Ted Zlatanov <tzz@lifelogs.com> writes:

>>> A .GPG file could be stored.  It wouldn't be synchronized, just
>>> set/get.

LI> Well, it should be updated when it's updated on one of the Gnus
LI> installations.

TZ> Right.  So I'd allow for a generic "save a file to the cloud"
TZ> facility that doesn't have any of the deltas.

I'm not sure that's a very good idea; there's no guarantee a
.authinfo.gpg file is encrypted or anything.
But I'd want to run the cloud on my own server anyways. After all, the
emacs does let you shoot yourself in the foot if you insist, so, go for
it.

LI> You might also envision totally storage-less Gnus installations,
LI> sort of.  That is, `M-x gnus-ephemeral-on-the-go' that would just
LI> get the data from the cloud, read a bit, and then exit without
LI> saving anything locally.  If you're afraid storing your secret
LI> .newsrc.eld/.authinfo.gpg files on your device.

TZ> Basically Google Reader.  I don't like that as much as just
TZ> synchronizing the configuration, although I'm sure some would like
TZ> it.  It seems like an extreme compared to just synchronizing the
TZ> things you listed.

I'd love that, just caching the authinfo and state, that way you can get
a fully blown gnus experience anywhere you have a recent emacs. Sounds
like what I'd want at uni.

TZ> I would make a final Gnus release of the current series and start a
TZ> new "Cloudy" or "Cumulonimbus Gnus" whichever way you go.  FWIW I'd
TZ> make it flexible enough that people can sync just their marks or go
TZ> all the way up to 100% synchronized configuration.

TZ> Ted

Give it sane defaults, but keep configuration very fine grained, down to
group level (maybe I want to read everything but my football groups at
work).

-- 
Philipp Haselwarter

Re: Gnus Cloud

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:

> Julien Danjou <julien@danjou.info> writes:
>> You can stop here:
>> 
>> http://lwn.net/Articles/427035/rss

> He didn't actually say that they're dropping MeeGo.  :-)

FWIW you _can_ get GNU emacs up and running on Android.  Others have
done so.  But the way for installing it, is pretty drastic.

Re: Gnus Cloud

[Ted Zlatanov]

On Thu, 10 Feb 2011 08:33:11 +0100 Julien Danjou <julien@danjou.info> wrote: 

JD> A NoSQL database like CouchDB, but replacing JSON with Lisp?

Yes, that was my original idea but with custom merge logic.  Lars'
proposal to just stack changes is probably better for the server but
definitely more expensive for the user.  Membase seems like a better fit
to me as the backend, though.  CouchDB is miserably slow under heavy
load.

(Yes, I know Membase and Couch* merged, but any actual results from that
union will take a while.)

I would really like to serve and accept JSON as an alternate format.
Then others can share this facility so it's not just a Gnus thing.

On Thu, 10 Feb 2011 11:58:53 +0100 Philipp Haselwarter <philipp.haselwarter@gmx.de> wrote: 

PH> "TZ" == Ted Zlatanov <tzz@lifelogs.com> writes:
TZ> Right.  So I'd allow for a generic "save a file to the cloud"
TZ> facility that doesn't have any of the deltas.

PH> I'm not sure that's a very good idea; there's no guarantee a
PH> .authinfo.gpg file is encrypted or anything.
PH> But I'd want to run the cloud on my own server anyways. After all, the
PH> emacs does let you shoot yourself in the foot if you insist, so, go for
PH> it.

Let's put it this way: if you choose to call an unencrypted file x.gpg
and save it on a server, why should the server stop you?

I think running the sync server on your own is a basic requirement,
yes.  The target user base for Gnus tends to be a bit paranoid[1] :)

Ted

[1] Including myself, of course.

Re: Gnus Cloud

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Basically Google Reader.  I don't like that as much as just
> synchronizing the configuration, although I'm sure some would like it.
> It seems like an extreme compared to just synchronizing the things you
> listed.

Yes, but implementation-wise it would be virtually "free".  Both the
cloud server and Gnus would do the same in the "everything is ephemeral"
and "this is the first start" cases, so it might be nice to have.  

> I would make a final Gnus release of the current series and start a new
> "Cloudy" or "Cumulonimbus Gnus" whichever way you go.

Yes.  Well, I'm not really planning on implementing this right away.
I'm still keeping my fingers crossed that I'll find time to implement
Reticule, although Outlook Not Good until I get home in a couple of
weeks, I guess...

But, yes, wrap up No Gnus, and then proceed on to something starting
with M.  Preferably a Japanese word that has kick-ass kanji that'll look
good on a t-shirt.  (I've got a new design for No Gnus t-shirts in my
head now, so I'll get cracking when I get home.)

M-word and Japanese to celebrate that it's been 25 years since Masanobu
UMEDA released the first version of Gnus.  That was 1987, wasn't it?
(- 2012 1987) => 25 or is that an off-by-one-error?  Anyway.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gnus Cloud

[Lars Ingebrigtsen]

Philipp Haselwarter <philipp.haselwarter@gmx.de> writes:

> Give it sane defaults, but keep configuration very fine grained, down to
> group level (maybe I want to read everything but my football groups at
> work).

Yes, I was thinking you could say "this server is cloud-enabled" (from
the server buffer), but then you could also bar/enable some groups from
the group buffer, too.  I mean, syncing my IMAP spam groups to other
devices probably isn't very interesting, but I'd like to have most of
the other mail groups available.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gnus Cloud

[Lars Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> Make IMAP partial article download and mime rendering works together,
> allow Gnus agent to sync both ways [1]

Yeah...  well, I think the partial download thing needs a rethink more
than anything.

> gnus/todo? :-)

I did skim through gnus/todo once or twice, but I didn't really see
anything very compelling there.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gnus Cloud

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I think running the sync server on your own is a basic requirement,
> yes. 

I'm not gonna run the cloud server myself.  I'll trust the
cloud.gnus.org admin to run it.

Er.

Anyway, running your own server should be no prob.  The requirements for
running the server will probably be something like sbcl + some NoSQL
backend or other.  Plus possibly Apache, but hopefully not.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gnus Cloud

[Ted Zlatanov]

On Thu, 10 Feb 2011 10:27:16 -0800 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:

>> I would make a final Gnus release of the current series and start a new
>> "Cloudy" or "Cumulonimbus Gnus" whichever way you go.

LI> Yes.  Well, I'm not really planning on implementing this right away.
LI> I'm still keeping my fingers crossed that I'll find time to implement
LI> Reticule, although Outlook Not Good until I get home in a couple of
LI> weeks, I guess...

Maybe I'll beat you to it.  Nah, I'm probably stuck in GnuTLS land after
auth-source.el is done.

LI> But, yes, wrap up No Gnus, and then proceed on to something starting
LI> with M.  Preferably a Japanese word that has kick-ass kanji that'll look
LI> good on a t-shirt.  (I've got a new design for No Gnus t-shirts in my
LI> head now, so I'll get cracking when I get home.)

OK, so pushing the cloud idea...  I see two cloud types starting with
'M' on Wikipedia:

Mediocris – medium size cumulus with small bulges at the top – indicates moderate updrafts

Mammatus – breast cloud – round pouches on surface of cloud

Can you really resist the allure of either of those words?  Can you?

Sadly, 'cloud' in Japanese is 雲 (くも, kumo) so that's not a 'M'
candidate.

Ted

Re: Gnus Cloud

[Sivaram Neelakantan]

On Thu, Feb 10 2011,Lars Ingebrigtsen wrote:

> Julien Danjou <julien@danjou.info> writes:
>
>> You can stop here:
>>
>>     http://lwn.net/Articles/427035/rss
>
> He didn't actually say that they're dropping MeeGo.  :-)
>

http://www.theregister.co.uk/2011/02/11/nokia_microsoft_more_details/print.html

--8<---------------cut here---------------start------------->8---
Remarkably, there's no pretence at continuity: almost everything old
is now dead, or soon will be. Symbian and Meego are sidelined and Qt
becomes irrelevant. Ovi lives on in name only. If you're a developer,
you write for Windows Mobile, to the APIs Microsoft wants you to write
to.
--8<---------------cut here---------------end--------------->8---

[snipped 20 lines]

Make of that what you will....Just saying.

 sivaram
 -- 

Re: Gnus Cloud

[Richard Riley]

Sivaram Neelakantan <nsivaram.net@gmail.com> writes:

> On Thu, Feb 10 2011,Lars Ingebrigtsen wrote:
>
>> Julien Danjou <julien@danjou.info> writes:
>>
>>> You can stop here:
>>>
>>>     http://lwn.net/Articles/427035/rss
>>
>> He didn't actually say that they're dropping MeeGo.  :-)
>>
>
> http://www.theregister.co.uk/2011/02/11/nokia_microsoft_more_details/print.html
>
        >
>
> --8<---------------cut here---------------start------------->8---
> Remarkably, there's no pretence at continuity: almost everything old
> is now dead, or soon will be. Symbian and Meego are sidelined and Qt
> becomes irrelevant. Ovi lives on in name only. If you're a developer,
> you write for Windows Mobile, to the APIs Microsoft wants you to write
> to.
> --8<---------------cut here---------------end--------------->8---


I dont know how relevant it might be but org-mode developers have done a
lot of work to sync smartphones etc with emacs org-data.

Re: Gnus Cloud

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Mediocris – medium size cumulus with small bulges at the top –
> indicates moderate updrafts
>
> Mammatus – breast cloud – round pouches on surface of cloud
>
> Can you really resist the allure of either of those words?  Can you?

Hm.  I think I can.

:-)

> Sadly, 'cloud' in Japanese is 雲 (くも, kumo) so that's not a 'M'
> candidate.

Darn.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gnus Cloud

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]


Lars Ingebrigtsen <larsi@gnus.org> writes:

> You cloud-enable the servers/groups you're interested in having be
> device-independent.  For me, this would be Gmane/Gwene groups, as well
> as my home IMAP server.
>
> Gnus will then ping the new RESTful cloud.gnus.org server with updates
> on new groups and marks.  From the Gnus side, this will look pretty much
> like what's Gnus is doing now with IMAP -- we tell the server what we've
> read, and what groups we're (un)subscribing.  Signing up for the server
> would be a zero-click process.  I.e., you just say what your cloud user
> name and password is, and it'll be created if it doesn't exist, and if
> it does, Gnus will keep the account updated.

There was rightful outrage at putting private data in a cloud server,
but it's easy to deflect by separating the cloud concept and having to
use a particular server.  As long as the server code is also Free,
people can just point their gnus clients at their own server.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]