overriding marks for synchronization

overriding marks for synchronization

[Ted Zlatanov]

To override the Gnus marks, is it sufficient to change
gnus-read-init-file and modify the gnus-newsrc-alist?  I'm wondering
about the Gnus Agent, for instance.

I'm asking specifically because I want to synchronize marks
automatically across several Gnus installs.  This is Gnus-only.  My plan
is to provide a few sync backends, including plain files, imap-hash, and
others.

Thanks
Ted

Re: overriding marks for synchronization

[Ted Zlatanov]

On Tue, 10 Aug 2010 06:55:42 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> To override the Gnus marks, is it sufficient to change
TZ> gnus-read-init-file and modify the gnus-newsrc-alist?  I'm wondering
TZ> about the Gnus Agent, for instance.

TZ> I'm asking specifically because I want to synchronize marks
TZ> automatically across several Gnus installs.  This is Gnus-only.  My plan
TZ> is to provide a few sync backends, including plain files, imap-hash, and
TZ> others.

It looks like this is the way to override read ranges and marks:

(defun gnus-synchronization-do-backends ()

;;; example: set marks
 (gnus-info-set-marks (assoc "groupname here" gnus-newsrc-alist)
  '((tick 202) (seen (197 . 201))))

;;; example: set read ranges
 (gnus-info-set-read (assoc "groupname here" gnus-newsrc-alist)
  '((1 . 203)))

;;; necessary to sync the gnus-newsrc-hashtb IIUC
 (gnus-make-hashtable-from-newsrc-alist))

I will add that function on `gnus-read-newsrc-el-hook'.

I looked at the gnus-agent but it does so much more with headers and
active file and offline caching that synchronizing just the marks would
really stretch it.  Better to create a new gnus-synchronization.el
library IMO.  Reiner or anyone else, let me know if you have an opinion.

Ted

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> I'm asking specifically because I want to synchronize marks
> automatically across several Gnus installs.  This is Gnus-only.  My
> plan is to provide a few sync backends, including plain files,
> imap-hash, and others.

That is a feature I want! :-)

Primarily for the nntp backend.  The imap backend synchronizes itself,
once I've visited  the groups.

Re: overriding marks for synchronization

[Ted Zlatanov]

On Wed, 11 Aug 2010 10:51:33 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> I'm asking specifically because I want to synchronize marks
>> automatically across several Gnus installs.  This is Gnus-only.  My
>> plan is to provide a few sync backends, including plain files,
>> imap-hash, and others.

SB> That is a feature I want! :-)

SB> Primarily for the nntp backend.  The imap backend synchronizes itself,
SB> once I've visited  the groups.

Yeah, any backend that doesn't keep marks on a server.  So nntp and
nnrss would be the main candidates, though I dearly hope nnrss fades
into oblivion now that gwene is available.

Ted

Re: overriding marks for synchronization

[Ted Zlatanov]

On Wed, 11 Aug 2010 08:36:52 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Wed, 11 Aug 2010 10:51:33 +0200 Steinar Bang <sb@dod.no> wrote: 
>>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>>> I'm asking specifically because I want to synchronize marks
>>> automatically across several Gnus installs.  This is Gnus-only.  My
>>> plan is to provide a few sync backends, including plain files,
>>> imap-hash, and others.

SB> That is a feature I want! :-)

SB> Primarily for the nntp backend.  The imap backend synchronizes itself,
SB> once I've visited  the groups.

TZ> Yeah, any backend that doesn't keep marks on a server.  So nntp and
TZ> nnrss would be the main candidates, though I dearly hope nnrss fades
TZ> into oblivion now that gwene is available.

See gnus-sync.el, just comitted (I worked on it last night and tested it
today).  It's harmless unless you set the gnus-sync-backend to a
filename, in which case it will start synchronizing to that file.  It's
most useful, of course, when the file is remote over FTP or SSH or IMAP
(did you know Tramp supports reading and writing files as messages in an
IMAP mailbox?)

It uses cl.el liberally.  I like the loop syntax, dolist, and setf.

Ted

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> See gnus-sync.el, just comitted (I worked on it last night and tested
> it today). 

Cool!

> It's harmless unless you set the gnus-sync-backend to a filename, in
> which case it will start synchronizing to that file.  It's most
> useful, of course, when the file is remote over FTP or SSH or IMAP
> (did you know Tramp supports reading and writing files as messages in
> an IMAP mailbox?)

Yes, I know.  But of the two alternatives ssh and imap, ssh seemed most
obvious to use.

> It uses cl.el liberally.  I like the loop syntax, dolist, and setf.

I'm not predjudiced. :-)

What I've done is:
 - Used "git pull" in my local gnus repo clone
 - Done "make" in the same
 - Added the following to my ~/.gnus.el (domain name changed to protect
   the innocent):
	;; Synching reads in nntp groups across machines
	(setq gnus-sync-backend `("/ssh:sb@stoner.mydomain.no:News/gnussync") ; will use Tramp+EPA if loaded
	      gnus-sync-global-vars `(gnus-newsrc-last-checked-date)
	      gnus-sync-newsrc-groups `("nntp" "nnrss")
	      gnus-sync-newsrc-vars `(read marks))

And then I've restarted gnus and have tried visiting some nntp groups on
the server "news.gmane.org" and saved the .newsrc* files with `s'.

But no News/gnussync seems to be created on the server...?

Is there an explicit sync command, perhaps...?

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> What I've done is:
>  - Used "git pull" in my local gnus repo clone
>  - Done "make" in the same
>  - Added the following to my ~/.gnus.el (domain name changed to protect
>    the innocent):
> 	;; Synching reads in nntp groups across machines
> 	(setq gnus-sync-backend `("/ssh:sb@stoner.mydomain.no:News/gnussync") ; will use Tramp+EPA if loaded
> 	      gnus-sync-global-vars `(gnus-newsrc-last-checked-date)
> 	      gnus-sync-newsrc-groups `("nntp" "nnrss")
> 	      gnus-sync-newsrc-vars `(read marks))

> And then I've restarted gnus and have tried visiting some nntp groups on
> the server "news.gmane.org" and saved the .newsrc* files with `s'.

> But no News/gnussync seems to be created on the server...?

I've tested that tramp seems to be working by doing
 `C-x f ssh:sb@stoner.mydomain.no:News RET'
and that came up with a dired on the News directory on the remote
server. 

> Is there an explicit sync command, perhaps...?

Nah... there's the function gnus-sync-save that's added to
gnus-save-newsrc-hook and the function gnus-sync-read that's added to
gnus-read-newsrc-el-hook.

I've tried running `M-x gnus-sync-save RET' and that doesn't create a
gnussync file either.

Re: overriding marks for synchronization

[Ted Zlatanov]

On Fri, 13 Aug 2010 14:26:45 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> It's harmless unless you set the gnus-sync-backend to a filename, in
>> which case it will start synchronizing to that file.  It's most
>> useful, of course, when the file is remote over FTP or SSH or IMAP
>> (did you know Tramp supports reading and writing files as messages in
>> an IMAP mailbox?)

SB> Yes, I know.  But of the two alternatives ssh and imap, ssh seemed most
SB> obvious to use.

It's better unless your workplace blocks SSH ;)

I screwed up the docs.  Here's the updated version:

(setq gnus-sync-backend "/remote:/path.gpg" ; will use Tramp+EPA if loaded
      gnus-sync-global-vars `(gnus-newsrc-last-checked-date)
      gnus-sync-newsrc-groups `("nntp" "nnrss")
      gnus-sync-newsrc-offsets `(2 3))

SB> Nah... there's the function gnus-sync-save that's added to
SB> gnus-save-newsrc-hook and the function gnus-sync-read that's added to
SB> gnus-read-newsrc-el-hook.

SB> I've tried running `M-x gnus-sync-save RET' and that doesn't create a
SB> gnussync file either.

That's the right command.  But the docs told you to make
`gnus-sync-backend' a list, which does nothing.  It should be a string.
Sorry about that.

I also want to make a change today so missing groups (meaning you're not
subscribed to them on the current machine) don't get dropped from the
loader.  When that's done I'll push out another commit with the updated
docs.

Thanks
Ted

Re: overriding marks for synchronization

[Ted Zlatanov]

On Fri, 13 Aug 2010 07:52:04 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> That's the right command.  But the docs told you to make
TZ> `gnus-sync-backend' a list, which does nothing.  It should be a string.
TZ> Sorry about that.

TZ> I also want to make a change today so missing groups (meaning you're not
TZ> subscribed to them on the current machine) don't get dropped from the
TZ> loader.  When that's done I'll push out another commit with the updated
TZ> docs.

Both changes were just comitted.  Test and let me know.

Thanks
Ted

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> Both changes were just comitted.  Test and let me know.

Closer.  Firstly I seem to need to do a 
(require 'gnus-sync)
to actually load gnus-sync.  Is that expected? (ie. did I miss it from
the documentatio?)  Is it forgotten from the documentation?  Or
shouldn't it be neccessary?

When I do that it saves the file.  But the file doesn't have any read
marks.

Here are the contents of the remote file saved over ssh:

;; -*- mode:emacs-lisp; coding: emacs-mule; -*-
;; Gnus sync data v. 0.0.1


(setq gnus-newsrc-last-checked-date '"Fri, 13 Aug 2010 16:07:55 +0200")

Re: overriding marks for synchronization

[Ted Zlatanov]

On Fri, 13 Aug 2010 16:12:41 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> Both changes were just comitted.  Test and let me know.

SB> Closer.  Firstly I seem to need to do a 
SB> (require 'gnus-sync)
SB> to actually load gnus-sync.  Is that expected? (ie. did I miss it from
SB> the documentatio?)  Is it forgotten from the documentation?  Or
SB> shouldn't it be neccessary?

It's necessary currently.  I'm not sure if it should be always on (since
it does nothing by default) or on request, that's why I didn't put it
in the docs.  What do you thikn?

SB> When I do that it saves the file.  But the file doesn't have any read
SB> marks.

SB> Here are the contents of the remote file saved over ssh:

SB> ;; -*- mode:emacs-lisp; coding: emacs-mule; -*-
SB> ;; Gnus sync data v. 0.0.1


SB> (setq gnus-newsrc-last-checked-date '"Fri, 13 Aug 2010 16:07:55 +0200")

For me, there's quite a bit in between:

(setq gnus-sync-newsrc-loader '(
... many lines omitted ...)

That's where the marks are stored.  You should see a message like
"gnus-sync: stored variables %s and %d groups in %s" in your messages.
It seems like gnus-sync-newsrc-loader is empty so that group count is
probably 0.  I think I see the problem; it didn't happen for me because
gnus-sync-newsrc-loader was already populated.  Try the following
fixed function (nunion should be on loader; also I added a "skipping
empty variable" comment):

(defun gnus-sync-save ()
"Save the Gnus sync data to the backend."
  (interactive)
  (cond
   ((stringp gnus-sync-backend)
    (gnus-message 7 "gnus-sync: saving to backend %s" gnus-sync-backend)
    ;; populate gnus-sync-newsrc-loader from all but the first dummy
    ;; entry in gnus-newsrc-alist whose group matches any of the
    ;; gnus-sync-newsrc-groups
    (let* ((loader
            (loop for entry in (cdr gnus-newsrc-alist)
                  when (gnus-grep-in-list
                        (car entry)     ;the group name
                        gnus-sync-newsrc-groups)
                  collect (cons (car entry)
                                (mapcar (lambda (offset)
                                          (cons offset (nth offset entry)))
                                        gnus-sync-newsrc-offsets))))
           (gnus-sync-newsrc-loader
            (nunion loader
                    (set-difference gnus-sync-newsrc-loader loader :key 'car)
                    :key 'car)))

      (with-temp-file gnus-sync-backend
        (progn
          (let ((coding-system-for-write gnus-ding-file-coding-system)
                (standard-output (current-buffer)))
            (princ (format ";; -*- mode:emacs-lisp; coding: %s; -*-\n"
                           gnus-ding-file-coding-system))
            (princ ";; Gnus sync data v. 0.0.1\n")
            (let* ((print-quoted t)
                   (print-readably t)
                   (print-escape-multibyte nil)
                   (print-escape-nonascii t)
                   (print-length nil)
                   (print-level nil)
                   (print-circle nil)
                   (print-escape-newlines t)
                   (variables (cons 'gnus-sync-newsrc-loader
                                    gnus-sync-global-vars))
                   variable)
              (while variables
                (if (and (boundp (setq variable (pop variables)))
                           (symbol-value variable))
                    (progn
                      (princ "\n(setq ")
                      (princ (symbol-name variable))
                      (princ " '")
                      (prin1 (symbol-value variable))
                      (princ ")\n"))
                  (princ "\n;;; skipping empty variable ")
                  (princ (symbol-name variable)))))
            (gnus-message
             7
             "gnus-sync: stored variables %s and %d groups in %s"
             gnus-sync-global-vars
             (length gnus-sync-newsrc-loader)
             gnus-sync-backend)

            ;; Idea from Dan Christensen <jdc@chow.mat.jhu.edu>
            ;; Save the .eld file with extra line breaks.
            (gnus-message 8 "gnus-sync: adding whitespace to %s"
                          gnus-sync-backend)
            (save-excursion
              (goto-char (point-min))
              (while (re-search-forward "^(\\|(\\\"" nil t)
                (replace-match "\n\\&" t))
              (goto-char (point-min))
              (while (re-search-forward " $" nil t)
                (replace-match "" t t))))))))
    ;; the pass-through case: gnus-sync-backend is not a known choice
    (nil)))

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:
SB> (setq gnus-newsrc-last-checked-date '"Fri, 13 Aug 2010 16:07:55 +0200")

> For me, there's quite a bit in between:

> (setq gnus-sync-newsrc-loader '(
> ... many lines omitted ...)

> That's where the marks are stored.  You should see a message like
> "gnus-sync: stored variables %s and %d groups in %s" in your messages.
> It seems like gnus-sync-newsrc-loader is empty so that group count is
> probably 0.  I think I see the problem; it didn't happen for me because
> gnus-sync-newsrc-loader was already populated.  Try the following
> fixed function (nunion should be on loader; also I added a "skipping
> empty variable" comment):

> (defun gnus-sync-save ()
[snip!]

That's more like it! :-)

That is... I haven't verified that multiple Gnusen will synchronize
against this file yet, but at least it's now filled with what looks like
meaningful content...;-)

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> It's necessary currently.  I'm not sure if it should be always on
> (since it does nothing by default) or on request, that's why I didn't
> put it in the docs.  What do you thikn?

I think that if it's currently a neccessary step to get things working,
it should be in the doc, so that people can just copy-paste a chunk from
the example into their ~/.gnus.el and edit the URL for the sync file.

Re: overriding marks for synchronization

[Ted Zlatanov]

On Fri, 13 Aug 2010 21:39:55 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> It's necessary currently.  I'm not sure if it should be always on
>> (since it does nothing by default) or on request, that's why I didn't
>> put it in the docs.  What do you thikn?

SB> I think that if it's currently a neccessary step to get things working,
SB> it should be in the doc, so that people can just copy-paste a chunk from
SB> the example into their ~/.gnus.el and edit the URL for the sync file.

OK, I'll put that in the next update.

Ted

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> OK, I'll put that in the next update.

Here's another small issue to fix:

diff --git a/lisp/gnus-sync.el b/lisp/gnus-sync.el
index b6131ce..98d17a3 100644
--- a/lisp/gnus-sync.el
+++ b/lisp/gnus-sync.el
@@ -217,7 +217,7 @@ synchronized, I believe).  Also see `gnus-variable-list'."
   (interactive)
   ;; (add-hook 'gnus-get-new-news-hook 'gnus-sync-read)
   (add-hook 'gnus-save-newsrc-hook 'gnus-sync-save)
-  (add-hook 'gnus-read-newsrc-el-hoo4a 'gnus-sync-read))
+  (add-hook 'gnus-read-newsrc-el-hook 'gnus-sync-read))
 
 (defun gnus-sync-unload-hook ()
   "Uninstall the sync hooks."

Without this fix there isn't much loading of sync files going on...:-)

Re: overriding marks for synchronization

[Ted Zlatanov]

On Sat, 14 Aug 2010 15:38:16 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> OK, I'll put that in the next update.

SB> Here's another small issue to fix:
SB> -  (add-hook 'gnus-read-newsrc-el-hoo4a 'gnus-sync-read))
SB> +  (add-hook 'gnus-read-newsrc-el-hook 'gnus-sync-read))

SB> Without this fix there isn't much loading of sync files going on...:-)

Fixed, thanks.

Ted

Re: overriding marks for synchronization

[Steinar Bang]

Ok, some user experiences, once I've been able to save to the sync file:
 - Most of the time synchronization from other machines doesn't seem to
   take place
 - In one case yesterday, synchronization from my netbook to my desktop
   machine actually happened on the desktop machine, but only on the
   third consecutive start of M-x gnus on that machine
 - On my netbook tramp complains on Gnus startup that the process buffer
   has died, killing gnus in the startup, and the half-ready *Group*
   buffer, and then starting gnus again, seems to fix the issue
 - In two cases now, starting gnus on my netbook (which was the last one
   to save to the gnussync file), some already read articles on this
   group and on gmane.discuss were shown as unread

Do you need to see some of my gnus config (of which there is quite a
bit) to see what might be different from yours.

The netbook has Lucid Lynx Ubuntu Netbook Remix.  The desktop is running
debian "squeeze" (testing).  The netbook has emacs 22 (because 23 has
sizing problems on UNR) and the desktop runs emacs 23.

Re: overriding marks for synchronization

[Ted Zlatanov]

On Sun, 15 Aug 2010 13:18:57 +0200 Steinar Bang <sb@dod.no> wrote: 

SB> Ok, some user experiences, once I've been able to save to the sync file:
SB>  - Most of the time synchronization from other machines doesn't seem to
SB>    take place
SB>  - In one case yesterday, synchronization from my netbook to my desktop
SB>    machine actually happened on the desktop machine, but only on the
SB>    third consecutive start of M-x gnus on that machine
SB>  - On my netbook tramp complains on Gnus startup that the process buffer
SB>    has died, killing gnus in the startup, and the half-ready *Group*
SB>    buffer, and then starting gnus again, seems to fix the issue
SB>  - In two cases now, starting gnus on my netbook (which was the last one
SB>    to save to the gnussync file), some already read articles on this
SB>    group and on gmane.discuss were shown as unread

SB> Do you need to see some of my gnus config (of which there is quite a
SB> bit) to see what might be different from yours.

No, it's OK.  I have to work out a nicer merge algorithm instead of
"just overwrite."  In the process I'll put safeguards around the
existing code to catch the cases you describe.

Ted

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> No, it's OK.  I have to work out a nicer merge algorithm instead of
> "just overwrite."  In the process I'll put safeguards around the
> existing code to catch the cases you describe.

When merging: does it make more sense to merge on the nntp-address
rather than the server name?

I have one gnus that has a different "news" server than the rest of
them.

Hm... I see one problem with this: the nntp-address equivalent is
different, depending on the backend.  The server name is consistent
across backends.

Perhaps I should just rename the server name for the default server on
this particular Gnus, from "news" to something different?

I think I'll do that.

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> When merging: does it make more sense to merge on the nntp-address
> rather than the server name?

> I have one gnus that has a different "news" server than the rest of
> them.

> Hm... I see one problem with this: the nntp-address equivalent is
> different, depending on the backend.  The server name is consistent
> across backends.

> Perhaps I should just rename the server name for the default server on
> this particular Gnus, from "news" to something different?

Neither approach will work for the default server, because .newsrc.eld
doesn't store any info about the server identity for the groups on that
server. 

On the other hand: the gnus-sync could store the server name for groups
on the default server, and consider that when syncing.  Then, having a
distinct server name for the default server would allow one to have a
different default server on different gnusen.

Re: overriding marks for synchronization

[Ted Zlatanov]

On Mon, 30 Aug 2010 18:06:02 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Steinar Bang <sb@dod.no>:
>> When merging: does it make more sense to merge on the nntp-address
>> rather than the server name?

>> I have one gnus that has a different "news" server than the rest of
>> them.

>> Hm... I see one problem with this: the nntp-address equivalent is
>> different, depending on the backend.  The server name is consistent
>> across backends.

>> Perhaps I should just rename the server name for the default server on
>> this particular Gnus, from "news" to something different?

SB> Neither approach will work for the default server, because .newsrc.eld
SB> doesn't store any info about the server identity for the groups on that
SB> server. 

SB> On the other hand: the gnus-sync could store the server name for groups
SB> on the default server, and consider that when syncing.  Then, having a
SB> distinct server name for the default server would allow one to have a
SB> different default server on different gnusen.

I didn't notice that problem (my primary is nil).  I think I should
simply always expand the server name on store and DTRT on load.

Ted

Re: overriding marks for synchronization

[Ted Zlatanov]

On Mon, 30 Aug 2010 17:59:10 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>> No, it's OK.  I have to work out a nicer merge algorithm instead of
>> "just overwrite."  In the process I'll put safeguards around the
>> existing code to catch the cases you describe.

SB> When merging: does it make more sense to merge on the nntp-address
SB> rather than the server name?

SB> I have one gnus that has a different "news" server than the rest of
SB> them.

SB> Hm... I see one problem with this: the nntp-address equivalent is
SB> different, depending on the backend.  The server name is consistent
SB> across backends.

SB> Perhaps I should just rename the server name for the default server on
SB> this particular Gnus, from "news" to something different?

SB> I think I'll do that.

Yeah.  The idea is to do a logical merge, on the nn* level, not on the
lower protocol level (NNTP or whatever).  So the logical server name is
the right key.

Ted

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> No, it's OK.  I have to work out a nicer merge algorithm instead of
> "just overwrite."  In the process I'll put safeguards around the
> existing code to catch the cases you describe.

Hm... another observation: I just killed the emacs process on my
netbook, and started a new emacs, did (require 'gnus-sync) in the
*scratch* buffer (I haven't added it to my setup yet).'

And a lot of articles that had been ticked in the previous emacs process
on the netbook showed up as unread in the new emacs.

My pattern on all of the gnusen now, is to quit gnus when I leave the
computer (as well as saving frequently).  So gnus have been stopped and
started many times in the old emacs process, and the read behaviour have
seemed sane.

The only explanation for this, is that there are gnus datastructures
that keep state, that hasn't been serialized to neither .newsrc.eld nor
the sync file...?  Could that be the case?

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> The only explanation for this, is that there are gnus datastructures
> that keep state, that hasn't been serialized to neither .newsrc.eld
> nor the sync file...?  Could that be the case?

Another observation: when I do `M-x gnus RET' to start up gnus at work,
it shows the article counts more or less as expected after sync'ing in
the reads done at home.

But the .newsrc.eld isn't made dirty.  Perhaps it should be?  So that a
save, would mean a local store of the merged read information?

Of course, a save would also mean that the sync file is rewritten...

Hm... confusing.

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> On Fri, 13 Aug 2010 14:26:45 +0200 Steinar Bang <sb@dod.no> wrote: 

SB> Yes, I know.  But of the two alternatives ssh and imap, ssh seemed most
SB> obvious to use.

> It's better unless your workplace blocks SSH ;)

Good point!  Also it would provide _some_ kind of transaction control.
Two independent ssh processes accessing the same file won't.
(not that it is likely to happen since I can't operate more than a
single Gnus at any point in time, but anyway...:-) )

The reasons I didn't go for imaps was:
 - I didn't know the steps neccessary to make tramp imap use the same
   auth as nnimap
 - I didn't know what the relationship towards folder/message/file is?
   I.e. 
    - do I need to create a particular folder for the use of tramp?
    - will it create one message or many?
    - Can more than one file be stored in any particular folder?
   (I'm guessing that the answer is "yes", "one message for each file,
   deleting the existing message and creating a new message when saving
   a new version", and "yes".  But I didn't know for sure and it seemed
   like work to find out...:-) )

Re: overriding marks for synchronization

[Ted Zlatanov]

On Fri, 13 Aug 2010 20:48:07 +0200 Steinar Bang <sb@dod.no> wrote: 

SB> Good point!  Also it would provide _some_ kind of transaction control.
SB> Two independent ssh processes accessing the same file won't.
SB> (not that it is likely to happen since I can't operate more than a
SB> single Gnus at any point in time, but anyway...:-) )

SB> The reasons I didn't go for imaps was:
SB>  - I didn't know the steps neccessary to make tramp imap use the same
SB>    auth as nnimap

The filename in my case (Courier IMAP server) is
/imaps:tzz@mail.server.com:/INBOX.test/sync.eld

which results in messages with subject
"tramp-imap-subject-markersync.eld" (you can change the variable
tramp-imap-subject-marker to something else if you like, by default it
is its own symbol-name).

*Messages* will tell you:

auth-source-user-or-password: get password for mail.server.com (tramp-imap) + user=nil

Which means in the authinfo you need ("port" can be omitted):

machine mail.server.com login tzz password MYPASSWORD port tramp-imap

SB>     - do I need to create a particular folder for the use of tramp?

No, but it's tidier that way.  tramp-imap.el filters out all messages
whose subject doesn't start with the tramp-imap-subject-marker.

SB>     - will it create one message or many?

IMAP doesn't allow "write in place" so instead we write a new message
and mark the old one expired.  It's not ideal but neither is IMAP.

SB>     - Can more than one file be stored in any particular folder?

Sure.  In fact any key-value data (if you use the underlying
imap-hash.el library) can be stored in an IMAP mailbox.  That's how
tramp-imap.el does it.

Ted

Re: overriding marks for synchronization

[Didier Verna]

Ted Zlatanov <tzz@lifelogs.com> wrote:

> I'm asking specifically because I want to synchronize marks
> automatically across several Gnus installs. This is Gnus-only. My plan
> is to provide a few sync backends, including plain files, imap-hash,
> and others.

  I'm using Gnus on different machines and everytime I switch from one
to another, a simple rsync script is enough to synchronize everything
Gnus-related (including BBDB etc). So I'm curious, what exactly do you
want to do ?

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Didier Verna <didier@xemacs.org>:

>   I'm using Gnus on different machines and everytime I switch from one
> to another, a simple rsync script is enough to synchronize everything
> Gnus-related (including BBDB etc). So I'm curious, what exactly do you
> want to do ?

I suspect he wants to do the same things I do: share the read and tick
marks of different newsgroups shared across multiple gnusen that may not
have the exact same set of groups on different news servers.

Re: overriding marks for synchronization

[Didier Verna]

Steinar Bang <sb@dod.no> wrote:

> I suspect he wants to do the same things I do: share the read and tick
> marks of different newsgroups shared across multiple gnusen that may
> not have the exact same set of groups on different news servers.

  Ah, yeah that makes sense. In fact I avoid the problem by always
connecting to the same nntp servers through (different) tunnels. But one
might not be able to do that all the time.

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Didier Verna <didier@xemacs.org>:

> Ah, yeah that makes sense. In fact I avoid the problem by always
> connecting to the same nntp servers through (different) tunnels. But
> one might not be able to do that all the time.

Actually the servers have to be the same (the same server names matching
the same underlying NNTP servers).  If not, the article numbers won't
match.

But what I have, and I'm guessing Ted has, is the same servers (in my
case news.gmane.org, news.eclipse.org and news.opera.com) with a
different subset of groups on different Gnusen.

In particular, a different subset of gmane groups.

And for those gmane groups that overlap, I prefer to only read the new
stuff once.

Re: overriding marks for synchronization

[Ted Zlatanov]

On Tue, 17 Aug 2010 20:55:23 +0200 Steinar Bang <sb@dod.no> wrote: 

>>>>>> Didier Verna <didier@xemacs.org>:
>> Ah, yeah that makes sense. In fact I avoid the problem by always
>> connecting to the same nntp servers through (different) tunnels. But
>> one might not be able to do that all the time.

SB> Actually the servers have to be the same (the same server names matching
SB> the same underlying NNTP servers).  If not, the article numbers won't
SB> match.

SB> But what I have, and I'm guessing Ted has, is the same servers (in my
SB> case news.gmane.org, news.eclipse.org and news.opera.com) with a
SB> different subset of groups on different Gnusen.

SB> In particular, a different subset of gmane groups.

SB> And for those gmane groups that overlap, I prefer to only read the new
SB> stuff once.

Right.  My setup is:

work: server A has groups P, Q, R; server B has groups X, Y, Z
home: server A has groups P, Q; server B is not available

I considered storing the marks on a per-server basis (providing storage
for marks like IMAP does internally, for example).  But I think
gnus-sync.el can be used to synchronize more: the topic hierarchy and
parameters, for instance.  In this it also will be more powerful than
gnus-agent.el and that's why I didn't just use that code.

Ted

Re: overriding marks for synchronization

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> ...  But I think gnus-sync.el can be used to synchronize more: the
> topic hierarchy and parameters, for instance. 

Look forward to both types of functionality.  What groups that overlap,
I would like to have as similar behaviour as possible.

Another thing that would have had nice to have synchronized, are those
score rules that I've added manually, typically by doing `I' or `L' in
the summary buffer of a group.

I don't know how distinct this is from other stuff found in score files,
though. 

> In this it also will be more powerful than gnus-agent.el and that's
> why I didn't just use that code.

Speaking of the agent... my netbook spends parts of its time in offline
mode.  Is it possible to handle the offline case?  Would a particular
tramp backend be better than others?  E.g. would the imap backend be
better than the ssh backend?