gnus sync

gnus sync

[Richard Riley]

I'm looking for a working sync solution for gnus amongst multple
machiens : specifically with the local nnml7archive backend. The problem
might be be that .newsrc.eld controls everything pretty much and yet I
would like "local" stuff to remain local. Stuff like imap and nntp are
not an issue - the .newsrc.eld marks are valid across machiens which I
currently sync with rsync. Any comments or pointers appreciated.

Re: gnus sync

[Ted Zlatanov]

On Thu, 30 Jun 2011 14:07:35 +0200 Richard Riley <rileyrg@googlemail.com> wrote: 

RR> I'm looking for a working sync solution for gnus amongst multple
RR> machiens : specifically with the local nnml7archive backend. The problem
RR> might be be that .newsrc.eld controls everything pretty much and yet I
RR> would like "local" stuff to remain local. Stuff like imap and nntp are
RR> not an issue - the .newsrc.eld marks are valid across machiens which I
RR> currently sync with rsync. Any comments or pointers appreciated.

It's on my TODO list for this release of Gnus.

Ted

Re: gnus sync

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> On Thu, 30 Jun 2011 14:07:35 +0200 Richard Riley <rileyrg@googlemail.com> wrote: 
RR> I'm looking for a working sync solution for gnus amongst multple
RR> machiens : specifically with the local nnml7archive backend. The problem
RR> might be be that .newsrc.eld controls everything pretty much and yet I
RR> would like "local" stuff to remain local. Stuff like imap and nntp are
RR> not an issue - the .newsrc.eld marks are valid across machiens which I
RR> currently sync with rsync. Any comments or pointers appreciated.

> It's on my TODO list for this release of Gnus.

In the meanwhile your first cut kinda works for me, as long as I accept
its limitations (ie. only read the file on gnus startup, stop each gnus
session when finished with it, and always being online when reading mail
and news (agent doesn't work anymore, so...)).

I've reenabled it and been running with it for some months now.

Re: gnus sync

[Ted Zlatanov]

On Fri, 01 Jul 2011 09:22:53 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> On Thu, 30 Jun 2011 14:07:35 +0200 Richard Riley <rileyrg@googlemail.com> wrote: 
RR> I'm looking for a working sync solution for gnus amongst multple
RR> machiens : specifically with the local nnml7archive backend. The problem
RR> might be be that .newsrc.eld controls everything pretty much and yet I
RR> would like "local" stuff to remain local. Stuff like imap and nntp are
RR> not an issue - the .newsrc.eld marks are valid across machiens which I
RR> currently sync with rsync. Any comments or pointers appreciated.

>> It's on my TODO list for this release of Gnus.

SB> In the meanwhile your first cut kinda works for me, as long as I accept
SB> its limitations (ie. only read the file on gnus startup, stop each gnus
SB> session when finished with it, and always being online when reading mail
SB> and news (agent doesn't work anymore, so...)).

SB> I've reenabled it and been running with it for some months now.

Yeah, same here.  It's definitely useful to have the functionality.

My plan is the same as before: server-centric, accepts a few formats,
does merge/intersection/reset, and outputs into JSON and Lisp
expressions.  It will be simple enough that other MUAs will be able to
use it, which is where the *real* fun will begin.  For example, NewsTap
(one of my favorite applications on the iPad) could synchronize with
this sync server.  It already lets you synchronize the newsrc file
directly.

I've cc-ed Alexander Clauss, the NewsTap author, to see if he's
interested in developing a standard for synchronization between NewsTap
and Gnus and other newsreaders.  I think the Gnus mailing list is the
right place to discuss this (and hope Alexander doesn't mind the
intrusion), but if there's a better forum I'll go there.  I'd also love
to get other newsreader authors involved.

Alexander, if you need some background on this thread, please let me
know and I'll fill you in.  I'd love to make NewsTap work with Gnus to
synchronize newsgroup subscriptions and article marks.  This is purely
selfish, since I use both, but I think it would benefit many users to
develop an open standard for this that's better than the newsrc format.

Also, if anyone knows of an existing standard for newsreader
synchronization, please let me know.  So far I've only seen solutions
that synchronize the newsrc or use a proprietary format.  My current
idea for the structure is:

subscription-set (just a name, only one per storage instance)
  params-list
  subscription-topic (e.g. "Programming" or "All")
    params-list
    group-name (e.g. "nntp servername.com groupname")
      params-list
      timestamp (floating-point unix epoch)
      "unread" unread-articles
      "read" read-articles
      "ticked" ticked-articles (favorites)
      mark1-name mark1-articles
      mark2-name mark2-articles
      ...

So in Lisp this might look like this:

(("Ted's sync" . 
   (() ;; no parameters
    ("Programming" . 
      (((created . 123.456)) ;; example parameter
       ("nntp gmane.org gmane.programming.articles" .
         (((level . 3)) ;; example parameter
          (timestamp . 123.456) ;; millisecond-precision
          (unread (10 200)) ;; seen but not read
          (read (1 9))
          (ticked (3 4) 5)
          (specialmark1 (8)))))))))

The equivalent JSON and XML representations would be trivial, though I
think just JSON is enough.

Let me know what you think.  I hope to get this done by August,
including the server side, and put up a free server that offers this
service (with the source code open, so others can do the same).

Thanks!
Ted

Re: gnus sync

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

[snip!]
> My plan is the same as before: server-centric, accepts a few formats,
> does merge/intersection/reset, and outputs into JSON and Lisp
> expressions.

I've had some thoughts about the implementation myself.  I've been
thinking of a RESTful approach.

You would have a URL space where you PUT and GET URLs, and where the
local path part changes, according to the resource you are PUTing and
GETing.

By GET, PUT, and DELETE, I mean the HTTP commands.

User authentication and user selection would be done, using HTTP
authentication, such as basic, or digest-md5 (not excluding unofficial
ones like NTLM or kerb, if emacs and the server both can get support for
it.

Format selection, and date information, will be provided using HTTP
headers.

At the top you would have something like
 https://myserver/gnussync/backends

Doing GET on this URL would list all synced backends as a lisp or JSON
list.  Which format you desire you can set with the Accept-Format
header.

You can create a new backend with a PUT to a child of "backends", eg.
 https://myserver/gnussync/backends/nnmybackend
Trying to re-PUT an existing backend should fail.  You would explicitly
have to DELETE it first.

If you GET a backend, eg.
 https://myserver/gnussync/backends/nntp
you would get the list of servers for that backend, as a JSON or lisp
list.

Like for backends, creating a new server will be done with PUT, and
trying to PUT an existing server should fail.  An existing server would
have to be DELETE'd before it could be re-PUT.

If you GET a server, eg. 
 https://myserver/gnussync/backends/nntp/news.gmane.org
you will get a list with two members "parameters" and "groups"

I haven't given much thought to the behaviour of
 https://myserver/gnussync/backends/nntp/news.gmane.org/parameters

If you GET the "groups" of a server, eg.
 https://myserver/gnussync/backends/nntp/news.gmane.org/groups
you will receive the groups of that server as a lisp or JSON list.

A new group can be created by doing a PUT of a sub-URL of the "groups",
eg. 
 https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.new.group

Trying to PUT an existing group should fail.  An existing group would
have to be DELETE's befor a new one could be PUT in its place.

If you GET a group, you will get a lisp, or JSON list containing two
members: "parameters" and "marks".

I haven't given much thought to the behaviours of group parameters, like
eg. 
 https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/parameters

If you GET a group's "marks", eg.
 https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/marks
you will receive a lisp or JSON list of all of the articles of that
group where tickmarks have been set.

Doing a "marks" GET with no qualifiers should return all of the marks
set.  I haven't figured out if the appropriate way of getting marks
newer than a cut-off, is by doing a conditional GET, or by adding an
argument, like eg.
 https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/marks?cutoff=2011-07-10-09:02:55

New marks are created, or old ones are updated, by doing a PUT of the
article number, eg.
 https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/marks/79313

Doing a PUT of an existing mark isn't an error.  The newest will win.

A tick mark be retreived by using a direct GET of a tickmark URL.

Even though it may be in breakage of REST rules (is it...?) it may be
convenient to get the tick marks together with the lisp or JSON list of
tick marks, eg. by using an argument:
 https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/marks?cutoff=2011-07-10-09:02:55&inlinemarks=true

That's how far my thoughts have gone.

Re: gnus sync

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

[snip!]
> If you GET a server, eg. 
>  https://myserver/gnussync/backends/nntp/news.gmane.org
> you will get a list with two members "parameters" and "groups"

> I haven't given much thought to the behaviour of
>  https://myserver/gnussync/backends/nntp/news.gmane.org/parameters
[snip!]

> If you GET a group, you will get a lisp, or JSON list containing two
> members: "parameters" and "marks".

> I haven't given much thought to the behaviours of group parameters, like
> eg. 
>  https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/parameters
[snip!]

When I think about it, and as was discussed in a recent thread, the
contents here, is something I really would prefer to see managed by a
VCS (like my .emacs and .gnus.el files are...).

On the other hand, one of the "cloud(y) gnus" requirements, was that
distributing everything gnus to a priestine gnus should "just work".

And that would definitely include the server and group parameters into
what should be synced...

hm...

Re: gnus sync

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:
> [snip!]
>> If you GET a server, eg. 
>> https://myserver/gnussync/backends/nntp/news.gmane.org
>> you will get a list with two members "parameters" and "groups"

>> I haven't given much thought to the behaviour of
>> https://myserver/gnussync/backends/nntp/news.gmane.org/parameters
> [snip!]

>> If you GET a group, you will get a lisp, or JSON list containing two
>> members: "parameters" and "marks".

>> I haven't given much thought to the behaviours of group parameters,
>> like eg.
>> https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/parameters
> [snip!]
> When I think about it, and as was discussed in a recent thread, the
> contents here, is something I really would prefer to see managed by a
> VCS (like my .emacs and .gnus.el files are...).

> On the other hand, one of the "cloud(y) gnus" requirements, was that
> distributing everything gnus to a priestine gnus should "just work".

> And that would definitely include the server and group parameters into
> what should be synced...

But how should .gnus.el-specified group parameters, like the one Tassilo
Horn showed in the "Where are group parameters stored?" thread, fit into
this picture...?  Perhaps the way they do today?  How are they combined
today, btw?

Reference to the article showing the .gnus.el example:
 http://thread.gmane.org/gmane.emacs.gnus.general/79418/focus=79447

Re: gnus sync

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> I've had some thoughts about the implementation myself.  I've been
> thinking of a RESTful approach.

Some thoughts on gnus behaviour:

I think that on startup it would be best to build up a complete
".newsrc.eld" in memory, that is the combination of the ~/.newsrc.eld on
disk, and what is received from the server.  If the server can't be
contacted (ie. if using the agent), then only the ~/.newsrc.eld comes
into play.

This means that when saving the .newsrc.eld, one is saving the
combination of what has been done locally and what has been received
from the server.

This means that merging of server and client information all the way
down to the tick marks, have been done, before starting to read news.

There should be a "last synchronized" time stamp in the .newsrc.eld, and
when polling the server it should relate everything against that (it
might be that there needs to be more fine-grained sync timestamping in
the .newsrc.eld, but I suggest just going with the single timestamp
initially).

When getting the backend list, use conditional GET against the time
stamp, and if you get a 304 not modified, then .newsrc.eld is the master
in conflicts, ie. differences in the number of backends.

Adding backends is easy and conflictless, but DELETE of a backend feels
pretty drastic... it would delete all of the backend's servers, and
their groups.

I should definitely be possible to DELETE a backend, but maybe this
should be excluded from the regular sync...?  There is a reason rsync has
--delete separate from --archive, it's so... destructive and final.

The Gnus client should PUT any new backends immediately.

When you get the server list for a backend, use conditional GET against
the .newsrc "last synchronized" timestamp, and if you get a 304 not
modified, then .newsrc.eld is the master in conflicts.

I see the same concerns about DELETE of servers, as for delete of
backends: ie. it should be possible, but maybe not part of the regular
sync.

The gnus client should PUT any new servers it has, as compared to the
gnus server, immediately.

I have no clear thoughts on GET of server parameters, and how they
should behave in case on conflicts.

When you get the group list for a server, use conditional GET against
the .newsrc "last synchronized" timestamp, and if you get a 304 not
modified, then .newsrc.eld is the master in conflicts.

Here I think it's OK to do a full sync, including DELETE of groups.
Worst case is that you lose the tick marks of a group, and that has
happened to me many times.

The gnus client should PUT and DELETE groups from detected differences
immediately. 

I have no clear thoughts on GET of group parameters, and how they should
be merged.

When doing a conditional GET of a group's marks, you will get a 304 not
modified if there are no marks newer than the timestamp.  If there are
marks newer than the timestamp, you will get a lisp or JSON list
containing their article numbers (possibly with the actual mark values
inlined). 

The gnus client should PUT all marks it has that has been set between
the "last synchronized" timestamp, and now, immediately.  This means
that the .newsrc.eld needs to hold some more timestamping.  Having a
timestamp on the individual tickmarks feels a bit heavy, but maybe it
should be tried...?  (I think the server needs to hold individual mark
timestamps, though)

When actually setting new tick marks, while online, the Gnus client
should just fire of PUTs for the tick marks as they are set.

When subscribing to new groups when online, the Gnus client should just
PUT them on the server.  Ditto for backends and servers (though they
would be rare).

That's all I have thought about, so far.

Re: gnus sync

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> Steinar Bang <sb@dod.no>:
>> I've had some thoughts about the implementation myself.  I've been
>> thinking of a RESTful approach.

> Some thoughts on gnus behaviour:

> I think that on startup it would be best to build up a complete
> ".newsrc.eld" in memory, that is the combination of the ~/.newsrc.eld on
> disk, and what is received from the server.

The RESTful approach I suggested earlier isn't very appropriate for
that, however.  It has nice URLs, but would be clumsy to use.

Eg. structuring servers under their backends looks nice, but has no
practical use for gnus.  And structuring groups under the servers also
looks nice, but has no practical use.

Also structuring different information under a server (properties and
newsgroups), and under newsgroups (properties and tickmarks), makes it
hard to request information across servers and groups (eg. what servers
or groups have properties more recent than a given timestamp).

Basically, what gnus wants from a server, would be:
 - The list of servers (with some way of identifying servers that have
   been added or deleted, compared to what the gnus instance has)
 - A simple way to get the server properties for the servers that has
   more recent server properties than the one the gnus instance has
 - The list of active newsgroups (with some way of identifying groups
   that have been added or removed compared to what the gnus instance
   has)
 - A simple way to get the group properties for groups that has more
   recent properties than what the gnus client has
 - A simple way to get the tick marks for a group, that is more recent
   than a given time stamp

So here's a new suggestion:

At the top there are three URLs:
 https://myserver/gnussync/servers
 https://myserver/gnussync/groups
 https://myserver/gnussync/groupprops

Doing a GET on
 https://myserver/gnussync/servers
will return at JSON or lisp list of all servers the sync server knows
about.

The list will contain server names on the form "backend+servername".
(Question: should the list also contain creation and last-modified time
stamps?)

Doing a PUT of a new server, eg
 https://myserver/gnussync/servers/backend+servername
then the body of the PUT request is expected to be server properties in
.newsrc.eld form (will be treated as a blob on the sync server).

Doing a PUT of an already existing server, should fail.  Do a DELETE of
the existing server first (should probably also delete all groups
associated with that server...?).

Doing a GET of the server, will result in the server properties in
.newsrc.eld form.

Doing a GET on either
 https://myserver/gnussync/groups
or
 https://myserver/gnussync/groupprops
will result in the same JSON or lisp list of group names, that follow
the gnus group name form of "backend+servername:group.name".

The "groupprops" results may also contain time stamps ("creation" and
"last modified"), if that's more convenient.

GET of a groupprops group URL, eg
 https://myserver/gnussync/groupprops/backend+servername:group.name
will return the group properties in .newsrc.eld form.

PUT on a grouppropes group URL will take the body of the PUT request and
set that as the new group properties for that group on the sync server.
PUT on an existing groupprops URL will overwrite the data.

GET on a groups group URL
 https://myserver/gnussync/groups/backend+servername:group.name
will return a JSON or lisp list containing all the article numbers of
the articles of the group, that have tick marks set.

To get the actual tick mark, GET the article, eg.
 https://myserver/gnussync/groups/backend+servername:group.name/articlenumber

Doing a GET with a cutoff argument, eg
 https://myserver/gnussync/groups/backend+servername:group.name?cutoff=2011-07-10-09:02:55
will return only the tick marks that have been set after the cutoff
date.

(It should probably be possible to get the tick marks inlined into the
group GET results...?)

Doing a PUT of an article number in a group, will take the body of the
PUT request to be the new tickmark.  PUTing over an existing tick mark
will overwrite that tick mark.

DELETE on an article number will remove the tick mark information for
that article.