Gnus is losing IMAP articles

Gnus is losing IMAP articles

[Edi Weitz]

Hi!

I'm connecting to an IMAP server (Courier) on a Linux machine which
has a couple of folders on it.  For several weeks now Gnus hasn't been
able to show all messages that are in these folders.  One of them,
e.g., has about 3,800 articles in it but Gnus shows less than 1,400.
This doesn't happen with other clients like, cough, Outlook and it
also doesn't happen with Emacs/Gnus on the server machine.

The setup for the two Emacs/Gnus combos are quite different, though,
and I don't really know how to debug this problem.  Any hints (where
to look at, what to try) would be greatly appreciated.

[FSF Emacs 22 (CVS from May), Gnus 5.10.6, Windows XP pro]

Thanks,
Edi.

-- 

Lisp is not dead, it just smells funny.

Real email: (replace (subseq "spamtrap@agharta.de" 5) "edi")

Re: Gnus is losing IMAP articles

[Ted Zlatanov]

On Tue, 05 Jul 2005, spamtrap@agharta.de wrote:

> I'm connecting to an IMAP server (Courier) on a Linux machine which
> has a couple of folders on it.  For several weeks now Gnus hasn't been
> able to show all messages that are in these folders.  One of them,
> e.g., has about 3,800 articles in it but Gnus shows less than 1,400.
> This doesn't happen with other clients like, cough, Outlook and it
> also doesn't happen with Emacs/Gnus on the server machine.
> 
> The setup for the two Emacs/Gnus combos are quite different, though,
> and I don't really know how to debug this problem.  Any hints (where
> to look at, what to try) would be greatly appreciated.
> 
> [FSF Emacs 22 (CVS from May), Gnus 5.10.6, Windows XP pro]

Do you mean that not all the articles you expect to see show up when
you just enter the group?  You may have caught up with those articles;
Gnus does not show "read" articles by default ("read" is an adjective
here, the language can be confusing).  Hit C-u Enter on the group
instead of just Enter and you should see all the articles in that
group.

Also the articles may have been deleted off the IMAP server for some
reason, but I would assume you are just not seeing read articles.

Ted

----== Posted via Newsfeeds.Com - Unlimited-Uncensored-Secure Usenet News==----
http://www.newsfeeds.com The #1 Newsgroup Service in the World! 120,000+ Newsgroups
----= East and West-Coast Server Farms - Total Privacy via Encryption =----

Re: Gnus is losing IMAP articles

[Edi Weitz]

On Tue, 05 Jul 2005 16:09:14 -0400, Ted Zlatanov <tzz@lifelogs.com> wrote:

> Do you mean that not all the articles you expect to see show up when
> you just enter the group?  You may have caught up with those
> articles; Gnus does not show "read" articles by default ("read" is
> an adjective here, the language can be confusing).  Hit C-u Enter on
> the group instead of just Enter and you should see all the articles
> in that group.

No.  Maybe I wasn't clear enough: If I press C-u Enter I see only
about 1,400 of roughly 3,800 articles in this IMAP folder.  Similar
effects in other folders.

> Also the articles may have been deleted off the IMAP server for some
> reason, but I would assume you are just not seeing read articles.

No, they aren't deleted.  I can see them from Outlook or from another
Gnus running on another machine.

Cheers,
Edi.

-- 

Lisp is not dead, it just smells funny.

Real email: (replace (subseq "spamtrap@agharta.de" 5) "edi")

Re: Gnus is losing IMAP articles

[Christian Lynbech]

>>>>> "Edi" == Edi Weitz <spamtrap@agharta.de> writes:

Edi> No, they aren't deleted.  I can see them from Outlook or from another
Edi> Gnus running on another machine.

I read mail via IMAP towards an Windows Exchange server here at
work. Mail arrive in "INBOX" and I have gnus retrieve mail from
"INBOX" and split them out into a bunch of other folders where I do
the reading.

When I occasionally fire up outlook, it will go to my "INBOX" and
display its contents. It furthermore reads the first message.

I have noticed that if outlook has marked a message as read, gnus will
not handle it anymore, at least in the sense that further splitting of
the "INBOX" will leave behind those messages that Outlook has touched.

The idea here is that it could be that "foreign" mailreaders does
stuff to the messages that makes them disappear to gnus.

I am unsure whether reverting (through Outlook) the read state of such
mails will make it reappear in front of gnus. I also think (but can't
really remember) that if I ask gnus to go to the "INBOX" folder all
such messages do in fact appear to gnus, indicating that the behaviour
described is not really what you ae experiencing.

One approach would to identify some of the missing messages and bash
them around a bit (copy to a new folder, change state etc) with some
of the other readers to see if that will make any difference to gnus.


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: Gnus is losing IMAP articles

[Edi Weitz]

On Wed, 06 Jul 2005 13:39:52 +0200, Christian Lynbech <christian.lynbech@ericsson.com> wrote:

> I read mail via IMAP towards an Windows Exchange server here at
> work. Mail arrive in "INBOX" and I have gnus retrieve mail from
> "INBOX" and split them out into a bunch of other folders where I do
> the reading.
>
> When I occasionally fire up outlook, it will go to my "INBOX" and
> display its contents. It furthermore reads the first message.
>
> I have noticed that if outlook has marked a message as read, gnus
> will not handle it anymore, at least in the sense that further
> splitting of the "INBOX" will leave behind those messages that
> Outlook has touched.
>
> The idea here is that it could be that "foreign" mailreaders does
> stuff to the messages that makes them disappear to gnus.
>
> I am unsure whether reverting (through Outlook) the read state of
> such mails will make it reappear in front of gnus. I also think (but
> can't really remember) that if I ask gnus to go to the "INBOX"
> folder all such messages do in fact appear to gnus, indicating that
> the behaviour described is not really what you ae experiencing.
>
> One approach would to identify some of the missing messages and bash
> them around a bit (copy to a new folder, change state etc) with some
> of the other readers to see if that will make any difference to
> gnus.

I've now done the following:

1. Mark all (3,800) messages as unread in Outlook, then quit Outlook.

2. Start Gnus and go to that IMAP folder.  Gnus still reports only
   1,400 (now unread) articles.

3. Catch up with this folder from Gnus.

4. Quit Gnus, start Outlook again.  Gnus has marked all 3,800 articles
   as read although it can only see 1,400 of them.

5. Via Outlook move one of the missing articles into another IMAP
   folder.

6. Gnus can now see it.  I can also move this article back into the
   offending folder from within Gnus and now also see it there.  But
   only this one - the other 2,400 are still missing.

Sigh - I still have no idea what's happening... :(

Thanks for your help,
Edi.

-- 

Lisp is not dead, it just smells funny.

Real email: (replace (subseq "spamtrap@agharta.de" 5) "edi")

Re: Gnus is losing IMAP articles

[Lars Tobias Børsting]

Edi Weitz <spamtrap@agharta.de> writes:

> 4. Quit Gnus, start Outlook again.  Gnus has marked all 3,800 articles
>    as read although it can only see 1,400 of them.
>
> 5. Via Outlook move one of the missing articles into another IMAP
>    folder.
>
> 6. Gnus can now see it.  I can also move this article back into the
>    offending folder from within Gnus and now also see it there.  But
>    only this one - the other 2,400 are still missing.
>
> Sigh - I still have no idea what's happening... :(

Could it be that Exchange does something wrong with the UIDVAILIDITY
value?

-- 
Regards,

Lars Tobias

Re: Gnus is losing IMAP articles

[Edi Weitz]

On Wed, 06 Jul 2005 14:46:55 +0200, Lars Tobias Børsting <larstobi@itea.ntnu.no> wrote:

> Could it be that Exchange does something wrong with the UIDVAILIDITYw
> value?

Well, certainly not Exchange because I use Courier on a Linux server
as my IMAP server.  How could I find out what the UIDVAILIDITY value
is and if it is wrong?

Thanks,
Edi.

-- 

Lisp is not dead, it just smells funny.

Real email: (replace (subseq "spamtrap@agharta.de" 5) "edi")

Re: Gnus is losing IMAP articles

[Ted Zlatanov]

On Wed, 06 Jul 2005, spamtrap@agharta.de wrote:

On Wed, 06 Jul 2005 14:46:55 +0200, Lars Tobias Børsting <larstobi@itea.ntnu.no> wrote:
> 
>> Could it be that Exchange does something wrong with the UIDVAILIDITYw
>> value?
> 
> Well, certainly not Exchange because I use Courier on a Linux server
> as my IMAP server.  How could I find out what the UIDVAILIDITY value
> is and if it is wrong?

The best thing to do at this point is to do an IMAP debug session (set
imap-debug to t and reproduce the problem on the Gnus side as best you
can).

If you can also log the Outlook interactions with the IMAP server
(perhaps snoop on the network traffic on the IMAP port) that would
probably also be helpful.

Remember to blank out passwords if they occur in the output.

Ted

----== Posted via Newsfeeds.Com - Unlimited-Uncensored-Secure Usenet News==----
http://www.newsfeeds.com The #1 Newsgroup Service in the World! 120,000+ Newsgroups
----= East and West-Coast Server Farms - Total Privacy via Encryption =----

Re: Gnus is losing IMAP articles

[Edi Weitz]

On Wed, 06 Jul 2005 10:50:52 -0400, Ted Zlatanov <tzz@lifelogs.com> wrote:

> The best thing to do at this point is to do an IMAP debug session
> (set imap-debug to t and reproduce the problem on the Gnus side as
> best you can).

I've now investigated this a little bit more and finally came to the
conclusion that Gnus agent was the problem.  One of the offending IMAP
folders was called INBOX.lists.lispworks and after deleting all files
in the local

  ~/News/agent/nnimap/mail-imap/INBOX/lists/lispworks

folder and restarting Emacs/Gnus I could finally see the whole IMAP
folder again.

Is there a better way to `re-sync' Gnus with the IMAP server without
getting rid of all the articles cached?  I though that
`gnus-agent-regenerate-group' would do it but obviously it doesn't.

Any why did this happen at all?  (It's not the only IMAP folder that
had problems.)

Now, this reminds me of other problems I had with Gnus agent and IMAP
but they were the other way around: After deleting IMAP articles with
another client (like SquirrelMail or Outlook) they were still shown by
Gnus and I had to manually delete the corresponding lines from the
`.overview' file.

FWIW, here are the settings related to Gnus agent from my .gnus file:

  (setq gnus-agent-cache t
        gnus-agent-consider-all-articles t
        gnus-agent-enable-expiration 'DISABLE
        gnus-select-article-hook 'gnus-agent-fetch-selected-article)

I freely admit that for some for them I don't know exactly what
they're for - I've probaly copied them monkey-see-monkey-do from other
Usenet articles.

Thanks,
Edi.

-- 

Lisp is not dead, it just smells funny.

Real email: (replace (subseq "spamtrap@agharta.de" 5) "edi")

Re: Gnus is losing IMAP articles

[Christian Lynbech]

>>>>> "Edi" == Edi Weitz <spamtrap@agharta.de> writes:

Edi> On Wed, 06 Jul 2005 10:50:52 -0400, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> The best thing to do at this point is to do an IMAP debug session
>> (set imap-debug to t and reproduce the problem on the Gnus side as
>> best you can).

Edi> I've now investigated this a little bit more and finally came to the
Edi> conclusion that Gnus agent was the problem.

I have seen funny stuff with the agent as well. 

I started up with `gnus-plugged' as I usually do and subscribed to an
RSS feed; it worked fine until I restarted my emacs. Next time, the
RSS group was still there but I had to type in the URL again; for some
reason it was not saved correctly. I then restarted emacs and called
`gnus' and resubscribed to the RSS feed and now emacs is able to
remember the URL (even when using the agent).

                             -- Christian

Re: Gnus is losing IMAP articles

[Lars Tobias Børsting]

Edi Weitz <spamtrap@agharta.de> writes:

> Well, certainly not Exchange because I use Courier on a Linux server
> as my IMAP server.  How could I find out what the UIDVAILIDITY value
> is and if it is wrong?

Oh, right, I confused you with Christian Lynbech which uses
Exchange. Sorry'bout that.

In Courier you can find the UIDVALIDITY value in every maildir folder in
the uiddb file called "courierimapuiddb". It is the second field in the
first line of the file. But Courier handles this correctly, so the
problem lies not with Courier.

--
Regards,

Lars Tobias

Re: Gnus is losing IMAP articles

[Christian Lynbech]

>>>>> "Lars" == Lars Tobias Børsting <larstobi@itea.ntnu.no> writes:

Lars> Oh, right, I confused you with Christian Lynbech which uses
Lars> Exchange. Sorry'bout that.

What an insult :-)

I do not use Exchange other than occassionally to respond to other
peoples meeting calls.

   

                             -- Christian

Re: Gnus is losing IMAP articles

[Lars Tobias Børsting]

Christian Lynbech <christian.lynbech@ericsson.com> writes:

> I read mail via IMAP towards an Windows Exchange server here at
> work. Mail arrive in "INBOX" and I have gnus retrieve mail from
> "INBOX" and split them out into a bunch of other folders where I do
> the reading.

Christian Lynbech <christian.lynbech@ericsson.com> writes:

> Lars Tobias Børsting <larstobi@itea.ntnu.no> writes:
>
>> Oh, right, I confused you with Christian Lynbech which uses
>> Exchange. Sorry'bout that.
>
> What an insult :-)
>
> I do not use Exchange other than occassionally to respond to other
> peoples meeting calls.

I'm terribly sorry about the insult! It wasn't really meant as one, but
I can see now how it totally is. :-)

Of course, «uses Exchange» should have been «has experimented with
Exchange». Better now?

We are just implementing Oracle Calendar at my uni, to replace
Exchange. I can't say I'm very sorry about that, anything should be
better than Exchange - even Oracle Calendar! ;-)

--
Regards,

Lars Tobias Børsting

Re: Gnus is losing IMAP articles

[David Z Maze]

Christian Lynbech <christian.lynbech@ericsson.com> writes:

> I read mail via IMAP towards an Windows Exchange server here at
> work. Mail arrive in "INBOX" and I have gnus retrieve mail from
> "INBOX" and split them out into a bunch of other folders where I do
> the reading.

Using nnimap, or an imap mail-source?

> When I occasionally fire up outlook, it will go to my "INBOX" and
> display its contents. It furthermore reads the first message.
>
> I have noticed that if outlook has marked a message as read, gnus will
> not handle it anymore, at least in the sense that further splitting of
> the "INBOX" will leave behind those messages that Outlook has
> touched.

The specific thing Outlook does is probably to mark the message as
read.  On my nnimap setup, I set nnimap-split-predicate to
"UNDELETED"; its default value is "UNSEEN UNDELETED", so if Outlook
reads the message then Gnus' IMAP mail splitting won't touch it.
Similarly, with an IMAP mail-source, you can set :predicate
"UNDELETED".  See e.g. info://gnus/Mail+Source+Specifiers under
`imap'.

> I am unsure whether reverting (through Outlook) the read state of such
> mails will make it reappear in front of gnus. I also think (but can't
> really remember) that if I ask gnus to go to the "INBOX" folder all
> such messages do in fact appear to gnus, indicating that the behaviour
> described is not really what you ae experiencing.

In that case you should be able to mark the articles as unread (in
Gnus, 'M SPC' from the summary buffer, perhaps 'M P b M-& M SPC' to
unread everything) and Gnus will happily split them for you.

  --dzm