initial sync, over and over

initial sync, over and over

[Michael Welsh Duggan]

I only just made the jump to Ma Gnus from No Gnus, and haven't really
been keeping track of recent development.  

On one particular IMAP server (Exchange), I am always getting an initial
sync of 7 groups.  Which seems rather non-initial, at least after the
first time.  Is this expected?

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: initial sync, over and over

[Tassilo Horn]

Michael Welsh Duggan <md5i@md5i.com> writes:

Hi Michael,

> I only just made the jump to Ma Gnus from No Gnus, and haven't really
> been keeping track of recent development.
>
> On one particular IMAP server (Exchange), I am always getting an
> initial sync of 7 groups.  Which seems rather non-initial, at least
> after the first time.

;-)

> Is this expected?

No, but it occurs to me here and there with my university's IMAP
(Courier), too.  And when it happens, I end up with all the groups that
were killed before now being subscribed.

Bye,
Tassilo

Re: initial sync, over and over

[Peder O. Klingenberg]

Michael Welsh Duggan <md5i@md5i.com> writes:

> On one particular IMAP server (Exchange), I am always getting an initial
> sync of 7 groups.  Which seems rather non-initial, at least after the
> first time.  Is this expected?

I saw something similar a while back.  I analyzed the problem a bit, but
never got to the point of writing code to fix it, because I found a
workaround that seemed to solve it.  Recently, I've seen an increase of
groups that seem to need an initial sync again, so it might be worth
looking into a proper fix.  I haven't looked into why the problem
reappears.

I'll share what analysis I did, including the workaround, in case
someone feels the urge to delve into it.

I saw that the unexist mark didn't exist in the info objects for the
groups that are repeatedly initial-synced.

And the reason unexist isn't set is because permanent-flags isn't set.
And permanent-flags doesn't get set because the nnimap-update-info
function receives 'not-scanned in its marks-parameter.

The reason for _that_ again is that EXAMINE is used instead of SELECT
to, well, examine the group.  Using EXAMINE to look at a group in IMAP
apparently causes the server to claim that every group is read-only
(observed).  nnimap-parse-flags ignores the PERMANENTFLAGS sent by the
server and sets permanent-flags to 'not-scanned.  This causes
nnimap-parse-flags to not store permanent-flags in the group parameters.

Gnus decides to use EXAMINE instead of SELECT if we've seen the group
before, determined by the presence of 'uidvalidity in gnus-info-params.
So basically, if we've seen the group before, we've not seen the group
before and need to do an initial sync.  This seems a bit contradictory.

To sum up:
 
* nnimap-retrieve-group-data-early finds uidvalidity in the group
  parameters => 
* EXAMINE is used instead of SELECT =>
* nnimap-update-info does not set permanent-flags in the group
  parameters =>
* nnimap-update-info believes the group is read-only and doesn't update
  unexists =>
* nnimap-retrieve-group-data-early believes it needs to do an initial
  sync.

I found two workarounds.  One was to manually move a message into one of
the affected group (B m).  This was too slow for me (I had around 50 of
the afflicted groups), so I forced nnimap-retrieve-group-data-early to
issue SELECT commands in all cases, and pressed 'g' in the group
buffer.  The first 'g' did initial sync again, but this time, unexist
was updated, so further 'g's didn't.  I then changed
nnimap-retrieve-group-data-early back the way it was, on the assumption
that the speedup from EXAMINE was worth keeping.

What I don't understand is how/why the permanent-flags group parameter
got lost.  It should have been set the first time (the _real_ first
time) the group was seen.  This may be the real root cause.

Anyway, that was as far as I got.  I'm not 100% sure of my analysis, I
haven't really looked at the gnus code before this, so it needs to be
checked.  Still, I hope it's a useful starting point.

...Peder...
-- 
This must be Thursday.  I never could get the hang of Thursdays.