* gnus sync @ 2011-06-30 12:07 Richard Riley 2011-06-30 14:38 ` Ted Zlatanov 0 siblings, 1 reply; 9+ messages in thread From: Richard Riley @ 2011-06-30 12:07 UTC (permalink / raw) To: ding I'm looking for a working sync solution for gnus amongst multple machiens : specifically with the local nnml7archive backend. The problem might be be that .newsrc.eld controls everything pretty much and yet I would like "local" stuff to remain local. Stuff like imap and nntp are not an issue - the .newsrc.eld marks are valid across machiens which I currently sync with rsync. Any comments or pointers appreciated. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: gnus sync 2011-06-30 12:07 gnus sync Richard Riley @ 2011-06-30 14:38 ` Ted Zlatanov 2011-07-01 7:22 ` Steinar Bang 0 siblings, 1 reply; 9+ messages in thread From: Ted Zlatanov @ 2011-06-30 14:38 UTC (permalink / raw) To: ding On Thu, 30 Jun 2011 14:07:35 +0200 Richard Riley <rileyrg@googlemail.com> wrote: RR> I'm looking for a working sync solution for gnus amongst multple RR> machiens : specifically with the local nnml7archive backend. The problem RR> might be be that .newsrc.eld controls everything pretty much and yet I RR> would like "local" stuff to remain local. Stuff like imap and nntp are RR> not an issue - the .newsrc.eld marks are valid across machiens which I RR> currently sync with rsync. Any comments or pointers appreciated. It's on my TODO list for this release of Gnus. Ted ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: gnus sync 2011-06-30 14:38 ` Ted Zlatanov @ 2011-07-01 7:22 ` Steinar Bang 2011-07-01 12:58 ` Ted Zlatanov 0 siblings, 1 reply; 9+ messages in thread From: Steinar Bang @ 2011-07-01 7:22 UTC (permalink / raw) To: ding >>>>> Ted Zlatanov <tzz@lifelogs.com>: > On Thu, 30 Jun 2011 14:07:35 +0200 Richard Riley <rileyrg@googlemail.com> wrote: RR> I'm looking for a working sync solution for gnus amongst multple RR> machiens : specifically with the local nnml7archive backend. The problem RR> might be be that .newsrc.eld controls everything pretty much and yet I RR> would like "local" stuff to remain local. Stuff like imap and nntp are RR> not an issue - the .newsrc.eld marks are valid across machiens which I RR> currently sync with rsync. Any comments or pointers appreciated. > It's on my TODO list for this release of Gnus. In the meanwhile your first cut kinda works for me, as long as I accept its limitations (ie. only read the file on gnus startup, stop each gnus session when finished with it, and always being online when reading mail and news (agent doesn't work anymore, so...)). I've reenabled it and been running with it for some months now. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: gnus sync 2011-07-01 7:22 ` Steinar Bang @ 2011-07-01 12:58 ` Ted Zlatanov 2011-07-10 19:31 ` Steinar Bang 0 siblings, 1 reply; 9+ messages in thread From: Ted Zlatanov @ 2011-07-01 12:58 UTC (permalink / raw) To: Ding Mailing List; +Cc: alexander On Fri, 01 Jul 2011 09:22:53 +0200 Steinar Bang <sb@dod.no> wrote: >>>>>> Ted Zlatanov <tzz@lifelogs.com>: >> On Thu, 30 Jun 2011 14:07:35 +0200 Richard Riley <rileyrg@googlemail.com> wrote: RR> I'm looking for a working sync solution for gnus amongst multple RR> machiens : specifically with the local nnml7archive backend. The problem RR> might be be that .newsrc.eld controls everything pretty much and yet I RR> would like "local" stuff to remain local. Stuff like imap and nntp are RR> not an issue - the .newsrc.eld marks are valid across machiens which I RR> currently sync with rsync. Any comments or pointers appreciated. >> It's on my TODO list for this release of Gnus. SB> In the meanwhile your first cut kinda works for me, as long as I accept SB> its limitations (ie. only read the file on gnus startup, stop each gnus SB> session when finished with it, and always being online when reading mail SB> and news (agent doesn't work anymore, so...)). SB> I've reenabled it and been running with it for some months now. Yeah, same here. It's definitely useful to have the functionality. My plan is the same as before: server-centric, accepts a few formats, does merge/intersection/reset, and outputs into JSON and Lisp expressions. It will be simple enough that other MUAs will be able to use it, which is where the *real* fun will begin. For example, NewsTap (one of my favorite applications on the iPad) could synchronize with this sync server. It already lets you synchronize the newsrc file directly. I've cc-ed Alexander Clauss, the NewsTap author, to see if he's interested in developing a standard for synchronization between NewsTap and Gnus and other newsreaders. I think the Gnus mailing list is the right place to discuss this (and hope Alexander doesn't mind the intrusion), but if there's a better forum I'll go there. I'd also love to get other newsreader authors involved. Alexander, if you need some background on this thread, please let me know and I'll fill you in. I'd love to make NewsTap work with Gnus to synchronize newsgroup subscriptions and article marks. This is purely selfish, since I use both, but I think it would benefit many users to develop an open standard for this that's better than the newsrc format. Also, if anyone knows of an existing standard for newsreader synchronization, please let me know. So far I've only seen solutions that synchronize the newsrc or use a proprietary format. My current idea for the structure is: subscription-set (just a name, only one per storage instance) params-list subscription-topic (e.g. "Programming" or "All") params-list group-name (e.g. "nntp servername.com groupname") params-list timestamp (floating-point unix epoch) "unread" unread-articles "read" read-articles "ticked" ticked-articles (favorites) mark1-name mark1-articles mark2-name mark2-articles ... So in Lisp this might look like this: (("Ted's sync" . (() ;; no parameters ("Programming" . (((created . 123.456)) ;; example parameter ("nntp gmane.org gmane.programming.articles" . (((level . 3)) ;; example parameter (timestamp . 123.456) ;; millisecond-precision (unread (10 200)) ;; seen but not read (read (1 9)) (ticked (3 4) 5) (specialmark1 (8))))))))) The equivalent JSON and XML representations would be trivial, though I think just JSON is enough. Let me know what you think. I hope to get this done by August, including the server side, and put up a free server that offers this service (with the source code open, so others can do the same). Thanks! Ted ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: gnus sync 2011-07-01 12:58 ` Ted Zlatanov @ 2011-07-10 19:31 ` Steinar Bang 2011-07-14 6:24 ` Steinar Bang 2011-07-14 7:39 ` Steinar Bang 0 siblings, 2 replies; 9+ messages in thread From: Steinar Bang @ 2011-07-10 19:31 UTC (permalink / raw) To: ding >>>>> Ted Zlatanov <tzz@lifelogs.com>: [snip!] > My plan is the same as before: server-centric, accepts a few formats, > does merge/intersection/reset, and outputs into JSON and Lisp > expressions. I've had some thoughts about the implementation myself. I've been thinking of a RESTful approach. You would have a URL space where you PUT and GET URLs, and where the local path part changes, according to the resource you are PUTing and GETing. By GET, PUT, and DELETE, I mean the HTTP commands. User authentication and user selection would be done, using HTTP authentication, such as basic, or digest-md5 (not excluding unofficial ones like NTLM or kerb, if emacs and the server both can get support for it. Format selection, and date information, will be provided using HTTP headers. At the top you would have something like https://myserver/gnussync/backends Doing GET on this URL would list all synced backends as a lisp or JSON list. Which format you desire you can set with the Accept-Format header. You can create a new backend with a PUT to a child of "backends", eg. https://myserver/gnussync/backends/nnmybackend Trying to re-PUT an existing backend should fail. You would explicitly have to DELETE it first. If you GET a backend, eg. https://myserver/gnussync/backends/nntp you would get the list of servers for that backend, as a JSON or lisp list. Like for backends, creating a new server will be done with PUT, and trying to PUT an existing server should fail. An existing server would have to be DELETE'd before it could be re-PUT. If you GET a server, eg. https://myserver/gnussync/backends/nntp/news.gmane.org you will get a list with two members "parameters" and "groups" I haven't given much thought to the behaviour of https://myserver/gnussync/backends/nntp/news.gmane.org/parameters If you GET the "groups" of a server, eg. https://myserver/gnussync/backends/nntp/news.gmane.org/groups you will receive the groups of that server as a lisp or JSON list. A new group can be created by doing a PUT of a sub-URL of the "groups", eg. https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.new.group Trying to PUT an existing group should fail. An existing group would have to be DELETE's befor a new one could be PUT in its place. If you GET a group, you will get a lisp, or JSON list containing two members: "parameters" and "marks". I haven't given much thought to the behaviours of group parameters, like eg. https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/parameters If you GET a group's "marks", eg. https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/marks you will receive a lisp or JSON list of all of the articles of that group where tickmarks have been set. Doing a "marks" GET with no qualifiers should return all of the marks set. I haven't figured out if the appropriate way of getting marks newer than a cut-off, is by doing a conditional GET, or by adding an argument, like eg. https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/marks?cutoff=2011-07-10-09:02:55 New marks are created, or old ones are updated, by doing a PUT of the article number, eg. https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/marks/79313 Doing a PUT of an existing mark isn't an error. The newest will win. A tick mark be retreived by using a direct GET of a tickmark URL. Even though it may be in breakage of REST rules (is it...?) it may be convenient to get the tick marks together with the lisp or JSON list of tick marks, eg. by using an argument: https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/marks?cutoff=2011-07-10-09:02:55&inlinemarks=true That's how far my thoughts have gone. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: gnus sync 2011-07-10 19:31 ` Steinar Bang @ 2011-07-14 6:24 ` Steinar Bang 2011-07-14 7:47 ` Steinar Bang 2011-07-14 7:39 ` Steinar Bang 1 sibling, 1 reply; 9+ messages in thread From: Steinar Bang @ 2011-07-14 6:24 UTC (permalink / raw) To: ding >>>>> Steinar Bang <sb@dod.no>: [snip!] > If you GET a server, eg. > https://myserver/gnussync/backends/nntp/news.gmane.org > you will get a list with two members "parameters" and "groups" > I haven't given much thought to the behaviour of > https://myserver/gnussync/backends/nntp/news.gmane.org/parameters [snip!] > If you GET a group, you will get a lisp, or JSON list containing two > members: "parameters" and "marks". > I haven't given much thought to the behaviours of group parameters, like > eg. > https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/parameters [snip!] When I think about it, and as was discussed in a recent thread, the contents here, is something I really would prefer to see managed by a VCS (like my .emacs and .gnus.el files are...). On the other hand, one of the "cloud(y) gnus" requirements, was that distributing everything gnus to a priestine gnus should "just work". And that would definitely include the server and group parameters into what should be synced... hm... ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: gnus sync 2011-07-14 6:24 ` Steinar Bang @ 2011-07-14 7:47 ` Steinar Bang 0 siblings, 0 replies; 9+ messages in thread From: Steinar Bang @ 2011-07-14 7:47 UTC (permalink / raw) To: ding >>>>> Steinar Bang <sb@dod.no>: > [snip!] >> If you GET a server, eg. >> https://myserver/gnussync/backends/nntp/news.gmane.org >> you will get a list with two members "parameters" and "groups" >> I haven't given much thought to the behaviour of >> https://myserver/gnussync/backends/nntp/news.gmane.org/parameters > [snip!] >> If you GET a group, you will get a lisp, or JSON list containing two >> members: "parameters" and "marks". >> I haven't given much thought to the behaviours of group parameters, >> like eg. >> https://myserver/gnussync/backends/nntp/news.gmane.org/groups/gmane.emacs.gnus.general/parameters > [snip!] > When I think about it, and as was discussed in a recent thread, the > contents here, is something I really would prefer to see managed by a > VCS (like my .emacs and .gnus.el files are...). > On the other hand, one of the "cloud(y) gnus" requirements, was that > distributing everything gnus to a priestine gnus should "just work". > And that would definitely include the server and group parameters into > what should be synced... But how should .gnus.el-specified group parameters, like the one Tassilo Horn showed in the "Where are group parameters stored?" thread, fit into this picture...? Perhaps the way they do today? How are they combined today, btw? Reference to the article showing the .gnus.el example: http://thread.gmane.org/gmane.emacs.gnus.general/79418/focus=79447 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: gnus sync 2011-07-10 19:31 ` Steinar Bang 2011-07-14 6:24 ` Steinar Bang @ 2011-07-14 7:39 ` Steinar Bang 2011-07-30 21:03 ` Steinar Bang 1 sibling, 1 reply; 9+ messages in thread From: Steinar Bang @ 2011-07-14 7:39 UTC (permalink / raw) To: ding >>>>> Steinar Bang <sb@dod.no>: > I've had some thoughts about the implementation myself. I've been > thinking of a RESTful approach. Some thoughts on gnus behaviour: I think that on startup it would be best to build up a complete ".newsrc.eld" in memory, that is the combination of the ~/.newsrc.eld on disk, and what is received from the server. If the server can't be contacted (ie. if using the agent), then only the ~/.newsrc.eld comes into play. This means that when saving the .newsrc.eld, one is saving the combination of what has been done locally and what has been received from the server. This means that merging of server and client information all the way down to the tick marks, have been done, before starting to read news. There should be a "last synchronized" time stamp in the .newsrc.eld, and when polling the server it should relate everything against that (it might be that there needs to be more fine-grained sync timestamping in the .newsrc.eld, but I suggest just going with the single timestamp initially). When getting the backend list, use conditional GET against the time stamp, and if you get a 304 not modified, then .newsrc.eld is the master in conflicts, ie. differences in the number of backends. Adding backends is easy and conflictless, but DELETE of a backend feels pretty drastic... it would delete all of the backend's servers, and their groups. I should definitely be possible to DELETE a backend, but maybe this should be excluded from the regular sync...? There is a reason rsync has --delete separate from --archive, it's so... destructive and final. The Gnus client should PUT any new backends immediately. When you get the server list for a backend, use conditional GET against the .newsrc "last synchronized" timestamp, and if you get a 304 not modified, then .newsrc.eld is the master in conflicts. I see the same concerns about DELETE of servers, as for delete of backends: ie. it should be possible, but maybe not part of the regular sync. The gnus client should PUT any new servers it has, as compared to the gnus server, immediately. I have no clear thoughts on GET of server parameters, and how they should behave in case on conflicts. When you get the group list for a server, use conditional GET against the .newsrc "last synchronized" timestamp, and if you get a 304 not modified, then .newsrc.eld is the master in conflicts. Here I think it's OK to do a full sync, including DELETE of groups. Worst case is that you lose the tick marks of a group, and that has happened to me many times. The gnus client should PUT and DELETE groups from detected differences immediately. I have no clear thoughts on GET of group parameters, and how they should be merged. When doing a conditional GET of a group's marks, you will get a 304 not modified if there are no marks newer than the timestamp. If there are marks newer than the timestamp, you will get a lisp or JSON list containing their article numbers (possibly with the actual mark values inlined). The gnus client should PUT all marks it has that has been set between the "last synchronized" timestamp, and now, immediately. This means that the .newsrc.eld needs to hold some more timestamping. Having a timestamp on the individual tickmarks feels a bit heavy, but maybe it should be tried...? (I think the server needs to hold individual mark timestamps, though) When actually setting new tick marks, while online, the Gnus client should just fire of PUTs for the tick marks as they are set. When subscribing to new groups when online, the Gnus client should just PUT them on the server. Ditto for backends and servers (though they would be rare). That's all I have thought about, so far. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: gnus sync 2011-07-14 7:39 ` Steinar Bang @ 2011-07-30 21:03 ` Steinar Bang 0 siblings, 0 replies; 9+ messages in thread From: Steinar Bang @ 2011-07-30 21:03 UTC (permalink / raw) To: ding >>>>> Steinar Bang <sb@dod.no>: >>>>> Steinar Bang <sb@dod.no>: >> I've had some thoughts about the implementation myself. I've been >> thinking of a RESTful approach. > Some thoughts on gnus behaviour: > I think that on startup it would be best to build up a complete > ".newsrc.eld" in memory, that is the combination of the ~/.newsrc.eld on > disk, and what is received from the server. The RESTful approach I suggested earlier isn't very appropriate for that, however. It has nice URLs, but would be clumsy to use. Eg. structuring servers under their backends looks nice, but has no practical use for gnus. And structuring groups under the servers also looks nice, but has no practical use. Also structuring different information under a server (properties and newsgroups), and under newsgroups (properties and tickmarks), makes it hard to request information across servers and groups (eg. what servers or groups have properties more recent than a given timestamp). Basically, what gnus wants from a server, would be: - The list of servers (with some way of identifying servers that have been added or deleted, compared to what the gnus instance has) - A simple way to get the server properties for the servers that has more recent server properties than the one the gnus instance has - The list of active newsgroups (with some way of identifying groups that have been added or removed compared to what the gnus instance has) - A simple way to get the group properties for groups that has more recent properties than what the gnus client has - A simple way to get the tick marks for a group, that is more recent than a given time stamp So here's a new suggestion: At the top there are three URLs: https://myserver/gnussync/servers https://myserver/gnussync/groups https://myserver/gnussync/groupprops Doing a GET on https://myserver/gnussync/servers will return at JSON or lisp list of all servers the sync server knows about. The list will contain server names on the form "backend+servername". (Question: should the list also contain creation and last-modified time stamps?) Doing a PUT of a new server, eg https://myserver/gnussync/servers/backend+servername then the body of the PUT request is expected to be server properties in .newsrc.eld form (will be treated as a blob on the sync server). Doing a PUT of an already existing server, should fail. Do a DELETE of the existing server first (should probably also delete all groups associated with that server...?). Doing a GET of the server, will result in the server properties in .newsrc.eld form. Doing a GET on either https://myserver/gnussync/groups or https://myserver/gnussync/groupprops will result in the same JSON or lisp list of group names, that follow the gnus group name form of "backend+servername:group.name". The "groupprops" results may also contain time stamps ("creation" and "last modified"), if that's more convenient. GET of a groupprops group URL, eg https://myserver/gnussync/groupprops/backend+servername:group.name will return the group properties in .newsrc.eld form. PUT on a grouppropes group URL will take the body of the PUT request and set that as the new group properties for that group on the sync server. PUT on an existing groupprops URL will overwrite the data. GET on a groups group URL https://myserver/gnussync/groups/backend+servername:group.name will return a JSON or lisp list containing all the article numbers of the articles of the group, that have tick marks set. To get the actual tick mark, GET the article, eg. https://myserver/gnussync/groups/backend+servername:group.name/articlenumber Doing a GET with a cutoff argument, eg https://myserver/gnussync/groups/backend+servername:group.name?cutoff=2011-07-10-09:02:55 will return only the tick marks that have been set after the cutoff date. (It should probably be possible to get the tick marks inlined into the group GET results...?) Doing a PUT of an article number in a group, will take the body of the PUT request to be the new tickmark. PUTing over an existing tick mark will overwrite that tick mark. DELETE on an article number will remove the tick mark information for that article. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-07-30 21:03 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-06-30 12:07 gnus sync Richard Riley 2011-06-30 14:38 ` Ted Zlatanov 2011-07-01 7:22 ` Steinar Bang 2011-07-01 12:58 ` Ted Zlatanov 2011-07-10 19:31 ` Steinar Bang 2011-07-14 6:24 ` Steinar Bang 2011-07-14 7:47 ` Steinar Bang 2011-07-14 7:39 ` Steinar Bang 2011-07-30 21:03 ` Steinar Bang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).