cyrus imapd upgrade: articles appear out of order/don't appear

cyrus imapd upgrade: articles appear out of order/don't appear

[Steinar Bang]

Platform: Intel PII, debian testing/unstable (sarge),
	  Oort Gnus v0.08 (CVS update from October 12),
	  XEmacs 21.4 (patch 6), cyrus imapd 2.1.11-4

After upgrading my cyrus imapd installation from cyrus 1.5.9 to
2.1.11, I see the following strange effects:

 1. in some nnimap groups new articles don't appear at the end when
    entering the folder, as they used to do in 1.5.9

 2. in one nnimap group the count for new articles display correctly,
    however when I enter the group there appears to be no articles there.
    However, when I do
	C-u number-of-new-articles RET
    on top of the group in the *Group* buffer, I can see the new
    articles 

The nnimap server is not agentized (it's on the same machine as the
Gnus client)

What I'm wondering now is if these effects are caused by:
 1. my manual upgrade of the cyrus database from 1.5.9 to 2.1.11?
 2. stuff left in .newsrc.eld, ~/News/ or ~/Mail/ from the 1.5.9
    server?
 3. problems between my Gnus version and cyrus 2.1.11?
 4. something else entirely?

Thanx!


- Steinar

Re: cyrus imapd upgrade: articles appear out of order/don't appear

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

> Platform: Intel PII, debian testing/unstable (sarge),
> 	  Oort Gnus v0.08 (CVS update from October 12),

Could you try upgrading?  I don't remember when the new
agent-by-default was added, but I remember fixing at least one problem
that appearead in nnimap groups after that, which may be what you are
seeing.

>  1. in some nnimap groups new articles don't appear at the end when
>     entering the folder, as they used to do in 1.5.9

They are shown earlier in the summary buffer, even when threading is
disabled?  That's very weird, I think it means that the server
assigned an unused UID to a message, which should never happen.  Is
this reproduable?  Could you look at the article number of the new
articles?

>  2. in one nnimap group the count for new articles display correctly,
>     however when I enter the group there appears to be no articles there.

This might be normal, and means that Gnus hadn't updated its marks for
that group yet.  The marks are updated when a group is entered, and if
it turns out no new articles arrived you see no new articles.

One caveat: since the default-agentification was enabled, one article
that existsed on the IMAP server never showed up in a summary buffer.
The reason was that the agent NOV file didn't include it.  I think I
moved that article into the group, and perhaps moving it failed, or
the NOV file generation somehow missed to generate a NOV entry for
articles added that way.  I haven't tried to reproduce it, but if you
think it sounds like your problem, maybe it is.

>     However, when I do
> 	C-u number-of-new-articles RET
>     on top of the group in the *Group* buffer, I can see the new
>     articles 

Then the articles was marked as read immediately.  Maybe you read the
articles in another client?

> What I'm wondering now is if these effects are caused by:
>  1. my manual upgrade of the cyrus database from 1.5.9 to 2.1.11?

Not impossible, maybe the mark databases got corrupted somehow.

>  2. stuff left in .newsrc.eld, ~/News/ or ~/Mail/ from the 1.5.9
>     server?

It =should= not have an effect if the server is behaving correctly
(i.e., not assigned new articles old UIDs).

>  3. problems between my Gnus version and cyrus 2.1.11?

I'm using Cyrus 2.1.11 and hasn't seen anything weird.

>  4. something else entirely?

Do you still see the problem?

Re: cyrus imapd upgrade: articles appear out of order/don't appear

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

> Steinar Bang <sb@dod.no> writes:
>> Platform: Intel PII, debian testing/unstable (sarge),
>> Oort Gnus v0.08 (CVS update from October 12),

> Could you try upgrading? 

I'd rather not, just right now.  I saw a thread on some XEmacs
problems in the current CVS.

[snip!]
>> 1. in some nnimap groups new articles don't appear at the end when
>> entering the folder, as they used to do in 1.5.9

> They are shown earlier in the summary buffer, even when threading is
> disabled?

No.  When I toggle off threading, the newest articles can be found at
the end of the buffer.

But I still think it's strange that new articles are shown before
articles I received several months ago, when threading is on.

>> 2. in one nnimap group the count for new articles display correctly,
>> however when I enter the group there appears to be no articles there.

> This might be normal, and means that Gnus hadn't updated its marks for
> that group yet.  The marks are updated when a group is entered, and if
> it turns out no new articles arrived you see no new articles.

This behaviour has happened repeatedly in my SPAM folder (ie. the
folder that gets everything not addressed directly to me, or to a
mailing list I subscribe to).

[snip!]
>> However, when I do
>> C-u number-of-new-articles RET
>> on top of the group in the *Group* buffer, I can see the new
>> articles 

> Then the articles was marked as read immediately.  Maybe you read
> the articles in another client?

Nope.  Only Gnus, and only the same Gnus on a single machine.

>> What I'm wondering now is if these effects are caused by:
>> 1. my manual upgrade of the cyrus database from 1.5.9 to 2.1.11?

> Not impossible, maybe the mark databases got corrupted somehow.

All marks were lost in the upgrade.  I had to walk through the nnimap
groups manually and tick everything I wanted ticked.

The strangeness I mentioned earlier happened once new mail started
arriving into the IMAP groups.

>> 2. stuff left in .newsrc.eld, ~/News/ or ~/Mail/ from the 1.5.9
>> server?

> It =should= not have an effect if the server is behaving correctly
> (i.e., not assigned new articles old UIDs).

OK.  Not sure how much about the structure of the old 1.5.9 database
that survived.

[snip!]
>> 4. something else entirely?

> Do you still see the problem?

I still see the out-of-order articles.  I haven't had any spam
arriving recently so I can't tell how that group is behaving right
now. 

Re: cyrus imapd upgrade: articles appear out of order/don't appear

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> Simon Josefsson <jas@extundo.com>:
>> Steinar Bang <sb@dod.no> writes:

> [snip!]

>>> 1. in some nnimap groups new articles don't appear at the end when
>>> entering the folder, as they used to do in 1.5.9

>> They are shown earlier in the summary buffer, even when threading
>> is disabled?

> No.  When I toggle off threading, the newest articles can be found
> at the end of the buffer.

> But I still think it's strange that new articles are shown before
> articles I received several months ago, when threading is on.

FYI I found the reason for this: the two nnimap groups in question had
lost their group parameters, and by that, this setting:

 (gnus-thread-sort-functions
  '(gnus-thread-sort-by-number)))

When I put it back in, the expected behaviour was back.

Why two groups lost their parameters and the other nnimap groups kept
them, during the server upgrade, I have no idea.