Gnus Own-Cloudy?

Gnus Own-Cloudy?

[Eric Abrahamsen]

I never paid any attention to progress of Cloudy Gnus, or whatever we're
calling it, but then I got a desktop along with my laptop, and suddenly
I'm VERY INTERESTED.

Before Cloudy is a reality, have any of you had success simply using
rsync between two machines? I have a few nnimap servers, and one nntp,
and so far I haven't been able to get the unread counts properly
replicated between the two computers. I've copied/synced:

~/.newsrc.eld
~/.gnus.registry.eieio
~/News
~/.netrc
~/.authinfo.gpg

The mail itself is of course synced through imap.

Doing it this way causes huge unread counts on whichever machine I'm
trying to catch up to the other.

I was sure that Gnus kept more "active" files somewhere else, listing
message counts and all that -- I can't find anything beyond what's in
~/News/, though. Maybe I need to set `gnus-read-active-file' to t?

Where else should I be looking?

Thanks!
Eric

Re: Gnus Own-Cloudy?

[Eric S Fraga]

On Tuesday, 10 Feb 2015 at 14:05, Eric Abrahamsen wrote:
> I never paid any attention to progress of Cloudy Gnus, or whatever we're
> calling it, but then I got a desktop along with my laptop, and suddenly
> I'm VERY INTERESTED.
>
> Before Cloudy is a reality, have any of you had success simply using
> rsync between two machines?

I use unison to keep 4 systems (two desktops, a laptop and a handheld
computer, all running some version of Linux) in sync.  Unison is based
on rsync but is a symmetric synchronisation instead of
one-directional.  I hope that makes sense...

I am not sure why you see large unread counts.  I do not.  You do have
to be quite rigorous in the synchronisation and avoid using gnus on two
systems simultaneously, however.

-- 
: Eric S Fraga, GnuPG: 0xFFFCF67D
: in Emacs 25.0.50.1 + Ma Gnus v0.12 + evil-git-bdeb602
: BBDB version 3.1.2 (2014-05-06 11:45:08 -0500)

Re: Gnus Own-Cloudy?

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Tuesday, 10 Feb 2015 at 14:05, Eric Abrahamsen wrote:
>> I never paid any attention to progress of Cloudy Gnus, or whatever we're
>> calling it, but then I got a desktop along with my laptop, and suddenly
>> I'm VERY INTERESTED.
>>
>> Before Cloudy is a reality, have any of you had success simply using
>> rsync between two machines?
>
> I use unison to keep 4 systems (two desktops, a laptop and a handheld
> computer, all running some version of Linux) in sync.  Unison is based
> on rsync but is a symmetric synchronisation instead of
> one-directional.  I hope that makes sense...
>
> I am not sure why you see large unread counts.  I do not.  You do have
> to be quite rigorous in the synchronisation and avoid using gnus on two
> systems simultaneously, however.

Thanks for the note! I'm quite sure I'm not using both systems at
once -- every few days I spend the day out with the laptop, and it's a
little rigmarole of updating my ~/org git repository, rsync'ing a few
other directories, and then doing my best with Gnus.

The first thing that happens is that when Gnus starts up all my nnimap
servers appear offline (or at least whatever it means when I get an
asterisk instead of unread counts next to all the group names).
Refreshing Gnus after that gives me unread counts that might be
reasonable if I *hadn't* sync'ed over the most recent marks, but for
some groups it's in the thousands, which clearly isn't right.

Is there anything you're sync'ing beyond the files/directories that I
mentioned in my first post?

Thanks!
Eric

Re: Gnus Own-Cloudy?

[Eric S Fraga]

On Tuesday, 10 Feb 2015 at 18:20, Eric Abrahamsen wrote:

[...]

> Is there anything you're sync'ing beyond the files/directories that I
> mentioned in my first post?

Hi Eric,

So, my approach for synchronising is to have a single directory that is
synchronised, called very non-imaginatively "~/synced".  In there, I have
(for gnus) three directories: News, Mail and dotfiles.

In my home directory, I have News symbolically linked to
~/synced/News.  The same for Mail.  I also have .newsrc.eld linked to
~/synced/dotfiles/newsrd.eld.  Likewise for ~/.gnus.el.

Because I use the registery, I have ~/.gnus.registry.eioio linked to
~/synced/dotfiles/gnus.registry.eioio.

For completeness, I also have the following gnus variable defined:

(setq gnus-startup-file "~/synced/newsrc")

although I am not sure what effect this has but it's been in my .gnus.el
file for yonks...

I cannot see anything else that I synchronise.

However, my main motivation for synchronising gnus was/is to be able to
work offline so I use the gnus agent.  Maybe this makes a difference?

For completeness, my sync routine is:

1. boot up and log in.
2. run unison to synchronise
3. work work work play work
   a) if using gnus, every so often, execute gnus-agent-fetch-session
4. run unison
5. turn system off

and I do this on every system except for work desktop that is up 24/7
and is essentially the "server" for all my other systems.  All
synchronisations of the home system, the laptop and the handheld are
with this office desktop system.  Sort of a star configuration.

The gnus cloud initiative won't make much difference for me as gnus is
but one element that I synchronise.  I have all my working files in
~/synced (around 14 GB) including research papers, email, org, talks,
code, blah blah blah.  I use my various systems as a form of redundant
backup... but more importantly every system I have looks the same so I
can pick up any of them and get to work right away!

HTH,
eric

-- 
: Eric S Fraga, GnuPG: 0xFFFCF67D
: in Emacs 25.0.50.1 + Ma Gnus v0.12 + evil-git-bdeb602
: BBDB version 3.1.2 (2014-05-06 11:45:08 -0500)

Re: Gnus Own-Cloudy?

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Tuesday, 10 Feb 2015 at 18:20, Eric Abrahamsen wrote:
>
> [...]
>
>> Is there anything you're sync'ing beyond the files/directories that I
>> mentioned in my first post?
>
> Hi Eric,
>
> So, my approach for synchronising is to have a single directory that is
> synchronised, called very non-imaginatively "~/synced".  In there, I have
> (for gnus) three directories: News, Mail and dotfiles.

This is really helpful, thank you. Probably what I should do is simplify
my system a little bit, and see if I can get to the root of the problem.
I also like to work offline, but I use isync/dovecot for that, instead
of the agent. Though I also use the agent for outgoing stuff... Maybe I
should shut that off and see if I can clean things up a bit.

Thanks again,
Eric

> In my home directory, I have News symbolically linked to
> ~/synced/News.  The same for Mail.  I also have .newsrc.eld linked to
> ~/synced/dotfiles/newsrd.eld.  Likewise for ~/.gnus.el.
>
> Because I use the registery, I have ~/.gnus.registry.eioio linked to
> ~/synced/dotfiles/gnus.registry.eioio.
>
> For completeness, I also have the following gnus variable defined:
>
> (setq gnus-startup-file "~/synced/newsrc")
>
> although I am not sure what effect this has but it's been in my .gnus.el
> file for yonks...
>
> I cannot see anything else that I synchronise.
>
> However, my main motivation for synchronising gnus was/is to be able to
> work offline so I use the gnus agent.  Maybe this makes a difference?
>
> For completeness, my sync routine is:
>
> 1. boot up and log in.
> 2. run unison to synchronise
> 3. work work work play work
>    a) if using gnus, every so often, execute gnus-agent-fetch-session
> 4. run unison
> 5. turn system off
>
> and I do this on every system except for work desktop that is up 24/7
> and is essentially the "server" for all my other systems.  All
> synchronisations of the home system, the laptop and the handheld are
> with this office desktop system.  Sort of a star configuration.
>
> The gnus cloud initiative won't make much difference for me as gnus is
> but one element that I synchronise.  I have all my working files in
> ~/synced (around 14 GB) including research papers, email, org, talks,
> code, blah blah blah.  I use my various systems as a form of redundant
> backup... but more importantly every system I have looks the same so I
> can pick up any of them and get to work right away!
>
> HTH,
> eric

Re: Gnus Own-Cloudy?

[Eric S Fraga]

On Tuesday, 10 Feb 2015 at 18:53, Eric Abrahamsen wrote:

[...]

> This is really helpful, thank you. Probably what I should do is simplify
> my system a little bit, and see if I can get to the root of the problem.
> I also like to work offline, but I use isync/dovecot for that, instead

Ah, an extra variable!  I don't think it makes sense to synchronise your
.newsrc.eld if it refers to different IMAP servers?  that is, if you
have a local copy via isync/dovecot on each system, these are different
servers (even if synchronised with the real IMAP server)?

And, in fact, I don't see why you would want to synchronise gnus files
if you have a local IMAP server which is already synchronising?

The problem with gnus, in this context, is that presentation (e.g. order
of groups, topics etc) is intermixed with the status of individual
groups so even though you might want to synchronise the presentation
aspects, you cannot do this without synchronising the interaction with
the server...  The newsrc.eld file has several functionalities combined
unfortunately.


-- 
: Eric S Fraga, GnuPG: 0xFFFCF67D
: in Emacs 25.0.50.1 + Ma Gnus v0.12 + evil-git-bdeb602
: BBDB version 3.1.2 (2014-05-06 11:45:08 -0500)

Re: Gnus Own-Cloudy?

[Jorge A. Alfaro-Murillo]

Eric S Fraga writes:

> And, in fact, I don't see why you would want to synchronise gnus 
> files if you have a local IMAP server which is already 
> synchronising?

In my case, because besides imap I also use nntp for 
news.gwene.org to read RSS and I want to keep track of which ones 
I already read.

Best,

-- 
Jorge.

Re: Gnus Own-Cloudy?

[Jorge A. Alfaro-Murillo]

Eric Abrahamsen writes:
 
> Where else should I be looking?

I keep my gnus synchronized by moving the whole gnus directory to 
another place, and then synchronizing with git-annex

#+BEGIN_SRC emacs-lisp
  (setq gnus-home-directory "~/sync/gnus/") (setq 
  message-directory "~/sync/gnus/Mail/")
#+END_SRC

I do not know if when you synchronize, your files get "touched" 
but another issue to consider is that your local .newsrc might be 
newer than your "just-synchronized" .newsrc.eld, if so, gnus just 
reads the .newsrc. If you do not use another news client, you 
might prevent this by setting:

#+BEGIN_SRC emacs-lisp
  (setq gnus-save-newsrc-file nil)
  (setq gnus-read-newsrc-file nil)
#+END_SRC

Also, if you do not save gnus before closing emacs, maybe you also 
want to keep .newsrc-dribble synchronized. 

Best,
-- 
Jorge.

Re: Gnus Own-Cloudy?

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Tuesday, 10 Feb 2015 at 18:53, Eric Abrahamsen wrote:
>
> [...]
>
>> This is really helpful, thank you. Probably what I should do is simplify
>> my system a little bit, and see if I can get to the root of the problem.
>> I also like to work offline, but I use isync/dovecot for that, instead
>
> Ah, an extra variable!  I don't think it makes sense to synchronise your
> .newsrc.eld if it refers to different IMAP servers?  that is, if you
> have a local copy via isync/dovecot on each system, these are different
> servers (even if synchronised with the real IMAP server)?
>
> And, in fact, I don't see why you would want to synchronise gnus files
> if you have a local IMAP server which is already synchronising?

But I would have thought that the active values for each group are
stored on disk, so if you don't copy those over as well, Gnus will get
confused about how many messages are in each group. My understanding is
that those values are saved, not re-created, isn't that right?

> The problem with gnus, in this context, is that presentation (e.g. order
> of groups, topics etc) is intermixed with the status of individual
> groups so even though you might want to synchronise the presentation
> aspects, you cannot do this without synchronising the interaction with
> the server...  The newsrc.eld file has several functionalities combined
> unfortunately.

Re: Gnus Own-Cloudy?

[Eric S Fraga]

On Wednesday, 11 Feb 2015 at 10:12, Eric Abrahamsen wrote:

[...]

> But I would have thought that the active values for each group are
> stored on disk, so if you don't copy those over as well, Gnus will get
> confused about how many messages are in each group. My understanding is
> that those values are saved, not re-created, isn't that right?

but as you have a local server on each system, they may be numbering
messages differently depending on how they synchronise with the real
IMAP server.  As a result, the counts and various data structures stored
within the newsrc.eld file may not be consistent between machines?

Just a theory.  I have no idea how isync etc. actually synchronise.

-- 
: Eric S Fraga, GnuPG: 0xFFFCF67D
: in Emacs 24.4.1 + Ma Gnus v0.12 + evil-git-bdeb602
: BBDB version 3.1.2 (2014-04-27 15:05:20 -0500)

Re: Gnus Own-Cloudy?

[Dan Christensen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> But I would have thought that the active values for each group are
> stored on disk, so if you don't copy those over as well, Gnus will get
> confused about how many messages are in each group.

For accessing an IMAP server, there's no need to sync Gnus data.  That's
sort of the point of using IMAP for your mail:  multiple clients can
independently access it and stay in sync with each other via the IMAP
server.

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> but as you have a local server on each system, they may be numbering
> messages differently depending on how they synchronise with the real
> IMAP server.  As a result, the counts and various data structures stored
> within the newsrc.eld file may not be consistent between machines?

Right.  When you are accessing two different IMAP servers (even if
they are synced behind the scenes), you definitely don't want to
sync the Gnus data, since article numbers won't match.  For example, if
you tick an article in once Gnus instance, a different article may show
up as ticked in the other.

Dan

Re: Gnus Own-Cloudy?

[Eric Abrahamsen]

Dan Christensen <jdc@uwo.ca> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> But I would have thought that the active values for each group are
>> stored on disk, so if you don't copy those over as well, Gnus will get
>> confused about how many messages are in each group.
>
> For accessing an IMAP server, there's no need to sync Gnus data.  That's
> sort of the point of using IMAP for your mail:  multiple clients can
> independently access it and stay in sync with each other via the IMAP
> server.
>
> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> but as you have a local server on each system, they may be numbering
>> messages differently depending on how they synchronise with the real
>> IMAP server.  As a result, the counts and various data structures stored
>> within the newsrc.eld file may not be consistent between machines?
>
> Right.  When you are accessing two different IMAP servers (even if
> they are synced behind the scenes), you definitely don't want to
> sync the Gnus data, since article numbers won't match.  For example, if
> you tick an article in once Gnus instance, a different article may show
> up as ticked in the other.

Okay, I've received (and marked) a bunch of mail on the desktop (now my
primary machine), after not having touched the laptop for a day. Right
now the desktop's Gnus installation is showing a single unread message
in one account's INBOX. I haven't copied any Gnus-related files from
desktop to laptop.

Now I go to my laptop, boot, run isync, then start Gnus. I get the
unread counts that I would expect if Gnus was keeping track of which
messages I'd read _on this machine_. Except for the sent mail folder of
one of my accounts, which shows 3000+ unread messages, a number that
doesn't seem to correspond to anything.

The UIDs seem to remain the same -- at least my earlier ticked messages
are the same on both machines. But I'm having another issue with
something in my mail chain (gmail <-> isync <-> dovecot <-> gnus)
re-setting UIDs for previously received messages, something that I just
reported yesterday on the isync mailing list[1]. I can see now that, after
I did a couple of syncs concurrently on both machines, mail I received
on the laptop is not appearing on the desktop -- presumably because the
UID of the message was changed, then incremented, but the UIDVALIDITY of
the folder was untouched. This is a pretty big bug, but in which I'm not
sure.

Lastly, and also weird: I signed up for an outlook.com account so I
could test the other issue, about searching for non-ascii characters via
IMAP. I connected Gnus to that account directly, not via isync/dovecot.
When I first restarted Gnus, it showed the unread count for that
account's INBOX as 10004, which was the most recent UID, not the unread
count (why outlook starts its UIDs at 10000 I don't know). After
clearing that, it's behaving normally.

So something weird is happening, or many weird things. Gnus has
obviously got some remaining strangeness about group unread counts, but
I'm inclined to blame the shifting UIDs on the interaction between
dovecot, isync, and gmail. The whole thing is a mystery to me. I hope
Oswald will respond on the isync list, and I'll look into Gnus' unread
counts this weekend. Otherwise, it's back to work!

Thanks,
Eric


[1]: http://sourceforge.net/p/isync/mailman/message/33402006/

Re: Gnus Own-Cloudy?

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> But I would have thought that the active values for each group are
>>> stored on disk, so if you don't copy those over as well, Gnus will get
>>> confused about how many messages are in each group.
>>
>> For accessing an IMAP server, there's no need to sync Gnus data.  That's
>> sort of the point of using IMAP for your mail:  multiple clients can
>> independently access it and stay in sync with each other via the IMAP
>> server.
>>
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>
>>> but as you have a local server on each system, they may be numbering
>>> messages differently depending on how they synchronise with the real
>>> IMAP server.  As a result, the counts and various data structures stored
>>> within the newsrc.eld file may not be consistent between machines?
>>
>> Right.  When you are accessing two different IMAP servers (even if
>> they are synced behind the scenes), you definitely don't want to
>> sync the Gnus data, since article numbers won't match.  For example, if
>> you tick an article in once Gnus instance, a different article may show
>> up as ticked in the other.
>
> Okay, I've received (and marked) a bunch of mail on the desktop (now my
> primary machine), after not having touched the laptop for a day. Right
> now the desktop's Gnus installation is showing a single unread message
> in one account's INBOX. I haven't copied any Gnus-related files from
> desktop to laptop.
>
> Now I go to my laptop, boot, run isync, then start Gnus. I get the
> unread counts that I would expect if Gnus was keeping track of which
> messages I'd read _on this machine_. Except for the sent mail folder of
> one of my accounts, which shows 3000+ unread messages, a number that
> doesn't seem to correspond to anything.

After thinking about this for a bit, I think I have a partial
explanation, and it's related to Gmail's emulation of IMAP folders.

I do client-side splitting. So I sync gmail with dovecot, then check
mail with Gnus. Gnus splits the mail, and tells dovecot to *remove* the
split mail from INBOX, and put it into another folder.

Next time I sync, dovecot and isync tell gmail to do the same thing. (I
don't know how IMAP works -- is it a "move" operation, or a "remove from
one group and add to another" operation? And when does the \Seen flag
get added?)

Gmail only does half of what it's told: it "adds" the new folder to the
message (as a label), but doesn't remove it from INBOX. This explains
the slowly-diverging folder message counts between my dovecot and Gmail.
It also helps explain the high unread-message count in my Gmail inbox
(via webmail): the messages must be first getting moved, then flagged as
\Seen, so the message that stays behind in Gmail's INBOX never gets
flagged.

When I sync on another computer, Gmail now has two copies of the
previously-split messages: one still in the INBOX, and one in the
destination folder, and both of these messages come down to dovecot,
then to Gnus. From what I can tell, neither has been marked \Seen, as
they show up as unread in both places in Gnus.

This still doesn't explain the shifting UID numbers, though.

Perhaps this is something I can fix by changing how isync talks to Gmail
(ie, sync directionality and all that). I don't know.

E

> The UIDs seem to remain the same -- at least my earlier ticked messages
> are the same on both machines. But I'm having another issue with
> something in my mail chain (gmail <-> isync <-> dovecot <-> gnus)
> re-setting UIDs for previously received messages, something that I just
> reported yesterday on the isync mailing list[1]. I can see now that, after
> I did a couple of syncs concurrently on both machines, mail I received
> on the laptop is not appearing on the desktop -- presumably because the
> UID of the message was changed, then incremented, but the UIDVALIDITY of
> the folder was untouched. This is a pretty big bug, but in which I'm not
> sure.
>
> Lastly, and also weird: I signed up for an outlook.com account so I
> could test the other issue, about searching for non-ascii characters via
> IMAP. I connected Gnus to that account directly, not via isync/dovecot.
> When I first restarted Gnus, it showed the unread count for that
> account's INBOX as 10004, which was the most recent UID, not the unread
> count (why outlook starts its UIDs at 10000 I don't know). After
> clearing that, it's behaving normally.
>
> So something weird is happening, or many weird things. Gnus has
> obviously got some remaining strangeness about group unread counts, but
> I'm inclined to blame the shifting UIDs on the interaction between
> dovecot, isync, and gmail. The whole thing is a mystery to me. I hope
> Oswald will respond on the isync list, and I'll look into Gnus' unread
> counts this weekend. Otherwise, it's back to work!
>
> Thanks,
> Eric
>
>
> [1]: http://sourceforge.net/p/isync/mailman/message/33402006/

Re: Gnus Own-Cloudy?

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> But I would have thought that the active values for each group are
>>> stored on disk, so if you don't copy those over as well, Gnus will get
>>> confused about how many messages are in each group.
>>
>> For accessing an IMAP server, there's no need to sync Gnus data.  That's
>> sort of the point of using IMAP for your mail:  multiple clients can
>> independently access it and stay in sync with each other via the IMAP
>> server.
>>
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>
>>> but as you have a local server on each system, they may be numbering
>>> messages differently depending on how they synchronise with the real
>>> IMAP server.  As a result, the counts and various data structures stored
>>> within the newsrc.eld file may not be consistent between machines?
>>
>> Right.  When you are accessing two different IMAP servers (even if
>> they are synced behind the scenes), you definitely don't want to
>> sync the Gnus data, since article numbers won't match.  For example, if
>> you tick an article in once Gnus instance, a different article may show
>> up as ticked in the other.
>
> Okay, I've received (and marked) a bunch of mail on the desktop (now my
> primary machine), after not having touched the laptop for a day. Right
> now the desktop's Gnus installation is showing a single unread message
> in one account's INBOX. I haven't copied any Gnus-related files from
> desktop to laptop.
>
> Now I go to my laptop, boot, run isync, then start Gnus. I get the
> unread counts that I would expect if Gnus was keeping track of which
> messages I'd read _on this machine_. Except for the sent mail folder of
> one of my accounts, which shows 3000+ unread messages, a number that
> doesn't seem to correspond to anything.
>
> The UIDs seem to remain the same -- at least my earlier ticked messages
> are the same on both machines. But I'm having another issue with
> something in my mail chain (gmail <-> isync <-> dovecot <-> gnus)
> re-setting UIDs for previously received messages, something that I just
> reported yesterday on the isync mailing list[1]. I can see now that, after
> I did a couple of syncs concurrently on both machines, mail I received
> on the laptop is not appearing on the desktop -- presumably because the
> UID of the message was changed, then incremented, but the UIDVALIDITY of
> the folder was untouched. This is a pretty big bug, but in which I'm not
> sure.

(This was a false alarm, the mail was just getting split to a group I
wasn't subscribed to on the other machine. The UIDs are still getting
changed, though.)

Re: Gnus Own-Cloudy?

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> But I would have thought that the active values for each group are
>>> stored on disk, so if you don't copy those over as well, Gnus will get
>>> confused about how many messages are in each group.
>>
>> For accessing an IMAP server, there's no need to sync Gnus data.  That's
>> sort of the point of using IMAP for your mail:  multiple clients can
>> independently access it and stay in sync with each other via the IMAP
>> server.
>>
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>
>>> but as you have a local server on each system, they may be numbering
>>> messages differently depending on how they synchronise with the real
>>> IMAP server.  As a result, the counts and various data structures stored
>>> within the newsrc.eld file may not be consistent between machines?
>>
>> Right.  When you are accessing two different IMAP servers (even if
>> they are synced behind the scenes), you definitely don't want to
>> sync the Gnus data, since article numbers won't match.  For example, if
>> you tick an article in once Gnus instance, a different article may show
>> up as ticked in the other.

[...]

> The UIDs seem to remain the same -- at least my earlier ticked messages
> are the same on both machines. But I'm having another issue with
> something in my mail chain (gmail <-> isync <-> dovecot <-> gnus)
> re-setting UIDs for previously received messages, something that I just
> reported yesterday on the isync mailing list[1].

I've been thinking about this more, and looking at recorded imap
commands. Here's the records for a message that's getting a new UID:

14:00:22 [localhost] 1910 SELECT "INBOX"
14:00:22 [localhost] 1911 UID FETCH 1:* FLAGS
14:00:22 [localhost] 1912 UID FETCH 9975 (UID BODY.PEEK[HEADER])
14:00:22 [localhost] 1913 UID COPY 9975 "INBOX"
14:00:22 [localhost] 1914 UID STORE 9975 +FLAGS.SILENT (\Deleted)
14:00:22 [localhost] 1915 UID EXPUNGE 9975

I have a hunch that's what happening is this:

1. Gnus gets the new message, tries to split it.
2. It doesn't get split, and so falls through to the final default split
   clause, which is "INBOX"
3. Gnus moves the message from INBOX to INBOX, then deletes it (see
   above)
4. The UID EXPUNGE command interacts badly with something else in my
   mail sync chain, and the message is re-created with a new UID.

That's what I'm guessing, anyway! Hopefully I'll get a little closer
tomorrow.

> [1]: http://sourceforge.net/p/isync/mailman/message/33402006/

[PATCH] Re: Gnus Own-Cloudy?

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Dan Christensen <jdc@uwo.ca> writes:
>>
>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>>> But I would have thought that the active values for each group are
>>>> stored on disk, so if you don't copy those over as well, Gnus will get
>>>> confused about how many messages are in each group.
>>>
>>> For accessing an IMAP server, there's no need to sync Gnus data.  That's
>>> sort of the point of using IMAP for your mail:  multiple clients can
>>> independently access it and stay in sync with each other via the IMAP
>>> server.
>>>
>>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>>
>>>> but as you have a local server on each system, they may be numbering
>>>> messages differently depending on how they synchronise with the real
>>>> IMAP server.  As a result, the counts and various data structures stored
>>>> within the newsrc.eld file may not be consistent between machines?
>>>
>>> Right.  When you are accessing two different IMAP servers (even if
>>> they are synced behind the scenes), you definitely don't want to
>>> sync the Gnus data, since article numbers won't match.  For example, if
>>> you tick an article in once Gnus instance, a different article may show
>>> up as ticked in the other.
>
> [...]
>
>> The UIDs seem to remain the same -- at least my earlier ticked messages
>> are the same on both machines. But I'm having another issue with
>> something in my mail chain (gmail <-> isync <-> dovecot <-> gnus)
>> re-setting UIDs for previously received messages, something that I just
>> reported yesterday on the isync mailing list[1].
>
> I've been thinking about this more, and looking at recorded imap
> commands. Here's the records for a message that's getting a new UID:
>
> 14:00:22 [localhost] 1910 SELECT "INBOX"
> 14:00:22 [localhost] 1911 UID FETCH 1:* FLAGS
> 14:00:22 [localhost] 1912 UID FETCH 9975 (UID BODY.PEEK[HEADER])
> 14:00:22 [localhost] 1913 UID COPY 9975 "INBOX"
> 14:00:22 [localhost] 1914 UID STORE 9975 +FLAGS.SILENT (\Deleted)
> 14:00:22 [localhost] 1915 UID EXPUNGE 9975
>
> I have a hunch that's what happening is this:
>
> 1. Gnus gets the new message, tries to split it.
> 2. It doesn't get split, and so falls through to the final default split
>    clause, which is "INBOX"
> 3. Gnus moves the message from INBOX to INBOX, then deletes it (see
>    above)
> 4. The UID EXPUNGE command interacts badly with something else in my
>    mail sync chain, and the message is re-created with a new UID.

This does appear to be what's happening, though I can't claim to be
completely following the full chain of cause and effect. If I refresh
Gnus while I still have unread messages in my INBOX, all those unread
messages get resplit back into the INBOX, with new UIDs.

Somehow (this is the bit I'm unclear about), the \Seen/read state then
gets confused between Gnus and my Dovecot server. Something like: when I
finally mark those messages as read in Gnus, Gnus stores the read status
correctly, but maybe applies the \Seen flag to the previous UID (before
the last round of splitting COPY'ed the message to a new UID), and the
next time I sync, the "real" message is marked as read in Gnus, but not
as \Seen in Dovecot. I've been getting messages like that ever since I
started using IMAP with Gnus.

So there may be a fundamental problem in the way Gnus applies its own
read status, versus \Seen marks on the server. On the other hand, the
simplest solution is probably just to not split messages at all, if they
would be split into the group they're already in. I can think of no
reason to do that. Conserve precious UIDs!

I'm attaching a patch that fixes that. Even if there is a more
fundamental problem, this patch is probably still worth putting through.
I'll run it for a week or so, and unless there's any objection (or
better ideas), push it to master.

Hopefully the end to a long, one-man thread...

Eric