Gnus sometimes reports new messages but not showing them on IMAP server

Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Hi,

I'm having some issues that Gnus sometimes reports that there are new
messages on an IMAP server but it doesn't show them.  In my case it's
using Gmail IMAP, and if I check using other mail clients
(e.g. Thunderbird) or directly check on Gmail website I can seem them.
I'm not sure whether this issue is specific to IMAP either.  As I'm not
sure how to get more info, I didn't use report bug yet, but would like
to ask around and see whether this is a known issue, or otherwise I can
get some suggestions to get more debug info before I formally file a
bug.

Any suggestions are welcome!  TIA!

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Greg Troxel]

Xiyue Deng <manphiz@gmail.com> writes:

> I'm having some issues that Gnus sometimes reports that there are new
> messages on an IMAP server but it doesn't show them.  In my case it's
> using Gmail IMAP, and if I check using other mail clients
> (e.g. Thunderbird) or directly check on Gmail website I can seem them.
> I'm not sure whether this issue is specific to IMAP either.  As I'm not
> sure how to get more info, I didn't use report bug yet, but would like
> to ask around and see whether this is a known issue, or otherwise I can
> get some suggestions to get more debug info before I formally file a
> bug.

I have been having a problem for several years that may be the same
thing.

My mail is in IMAP via dovecot.  Other than the following I have zero
problems with it.

I read mail with gnus (emacs built w/o x11 running in tmux in an xterm,
on NetBSD, currently 28.2 and thus the included gnus), with about 300
folders, about half of which have server-side filing of list mail.  I
also use K-9 Mail on Android (INBOX and maybe 3 more mailboxes, more or
less, ignoring the folders for lists and storage).  Occasionally I use
thunderbird for INBOX, to read something that should have been plain
text and isn't :-) but this issue has happened with no tbird running at
all.

Often, I get a message on my phone that I just leave (marked read) to
deal with later.  Sometimes, maybe 1 in 50, I can't find it in gnus, and
on looking harder find that it is just not showing up there.  Yet K-9
sees it as present and looking in ~/IMAP/cur shows it was well.
Whenever this has happened and I look with Thunderbird, it sees it fine
also.

I have found that if I move the message to another folder (I have TMP
for this very purpose) and then back to INBOX, it can then be found in
gnus.  So something about the message as it sits in the folder was
triggering gnus to miss it, while the very same message refiled does not
trigger this.

I have not been able to determine any pattern of which messages get this
disappearing treatment.  It's not really the message, since moving and
back fixes it.

Early on, I sort of suspected dovecot, but other people don't have
trouble with dovecot, and I have had zero flakiness other than this
specific issue.

I have not perceived "gnus says there are new messages but there
aren't".  I am not at all sure I would have noticed, especially if gnus
said 6 and there were 5.

In your case, if you do "100<spc>" in group, do you see the missing
message?  If not, you should be able to turn on some kind of debugging
to figure out what's going on, but I am unclear on the details.

Greg

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Greg Troxel <gdt@lexort.com> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> I'm having some issues that Gnus sometimes reports that there are new
>> messages on an IMAP server but it doesn't show them.  In my case it's
>> using Gmail IMAP, and if I check using other mail clients
>> (e.g. Thunderbird) or directly check on Gmail website I can seem them.
>> I'm not sure whether this issue is specific to IMAP either.  As I'm not
>> sure how to get more info, I didn't use report bug yet, but would like
>> to ask around and see whether this is a known issue, or otherwise I can
>> get some suggestions to get more debug info before I formally file a
>> bug.
>
> I have been having a problem for several years that may be the same
> thing.
>
> My mail is in IMAP via dovecot.  Other than the following I have zero
> problems with it.
>
> I read mail with gnus (emacs built w/o x11 running in tmux in an xterm,
> on NetBSD, currently 28.2 and thus the included gnus), with about 300
> folders, about half of which have server-side filing of list mail.  I
> also use K-9 Mail on Android (INBOX and maybe 3 more mailboxes, more or
> less, ignoring the folders for lists and storage).  Occasionally I use
> thunderbird for INBOX, to read something that should have been plain
> text and isn't :-) but this issue has happened with no tbird running at
> all.
>
> Often, I get a message on my phone that I just leave (marked read) to
> deal with later.  Sometimes, maybe 1 in 50, I can't find it in gnus, and
> on looking harder find that it is just not showing up there.  Yet K-9
> sees it as present and looking in ~/IMAP/cur shows it was well.
> Whenever this has happened and I look with Thunderbird, it sees it fine
> also.
>

Exactly this symptom.

> I have found that if I move the message to another folder (I have TMP
> for this very purpose) and then back to INBOX, it can then be found in
> gnus.  So something about the message as it sits in the folder was
> triggering gnus to miss it, while the very same message refiled does not
> trigger this.
>

I haven't tried this in this case, but there is actually another issue
that I haven't talked about: sometimes one of the empty folders may
report a number of new messages in it!  In one of the occurrences it
reports 52 new message on an empty folder.  The solution, however, is
similar to what you did but reverse: you move a number of messages from
another folder to it, and the new message count will reduce by the
number of messages you moved!  So in the previous case if you move 10
messages to the folder, the count becomes 42.  You move them out, and
move them back again, 32, etc.

Let me also try your solution in future.

> I have not been able to determine any pattern of which messages get this
> disappearing treatment.  It's not really the message, since moving and
> back fixes it.
>
> Early on, I sort of suspected dovecot, but other people don't have
> trouble with dovecot, and I have had zero flakiness other than this
> specific issue.
>
> I have not perceived "gnus says there are new messages but there
> aren't".  I am not at all sure I would have noticed, especially if gnus
> said 6 and there were 5.
>

I have only encountered such cases when the folder is empty.

> In your case, if you do "100<spc>" in group, do you see the missing
> message?  If not, you should be able to turn on some kind of debugging
> to figure out what's going on, but I am unclear on the details.
>

No, specifying a number didn't work for me, unfortunately.

And I guess to really understand this issue one may have to read the
code of Gnus or IMAP backend or hopefully someone that does may be able
to help.

> Greg
>
>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Apr 11, 2024, Xiyue Deng <manphiz@gmail.com> wrote:

> I'm having some issues that Gnus sometimes reports that there are new
> messages on an IMAP server but it doesn't show them.

On Apr 16, 2024, Greg Troxel <gdt@lexort.com> wrote:

> Often, I get a message on my phone that I just leave (marked read) to
> deal with later.  Sometimes, maybe 1 in 50, I can't find it in gnus, and
> on looking harder find that it is just not showing up there.

For both of you, does hitting `M-g' with point on the group in the
*Group* buffer fix things?

I regularly have similar problems, and I've resorted to binding the
following function to a key:

(defun jdc-gnus-update-IMAP ()
  (interactive)
  ; Only run this in the group buffer or it corrupts other buffers:
  (switch-to-buffer gnus-group-buffer)
  (gnus-topic-jump-to-topic "IMAP groups")
  (gnus-topic-get-new-news-this-topic)
  (gnus-group-list-groups) ; "l", as previous command shows all groups in topic
  )

I think it's doing the equivalent of `M-g' on all groups in the topic
I have that contains my IMAP groups.  For the record, I'm also using
dovecot and k9-mail.

Dan

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Greg Troxel]

Dan Christensen <jdc@uwo.ca> writes:

> On Apr 11, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>
>> I'm having some issues that Gnus sometimes reports that there are new
>> messages on an IMAP server but it doesn't show them.
>
> On Apr 16, 2024, Greg Troxel <gdt@lexort.com> wrote:
>
>> Often, I get a message on my phone that I just leave (marked read) to
>> deal with later.  Sometimes, maybe 1 in 50, I can't find it in gnus, and
>> on looking harder find that it is just not showing up there.
>
> For both of you, does hitting `M-g' with point on the group in the
> *Group* buffer fix things?

I will try that, but I have been exiting the group and ^U SPC to
re-enter and display all, and when a message has gone missing, I have
not even once been able to get gnus to show it.  I have exited and
restarted gnus, and done "g".

> I regularly have similar problems, and I've resorted to binding the
> following function to a key:
>
> (defun jdc-gnus-update-IMAP ()
>   (interactive)
>   ; Only run this in the group buffer or it corrupts other buffers:
>   (switch-to-buffer gnus-group-buffer)
>   (gnus-topic-jump-to-topic "IMAP groups")
>   (gnus-topic-get-new-news-this-topic)
>   (gnus-group-list-groups) ; "l", as previous command shows all groups in topic
>   )
>
> I think it's doing the equivalent of `M-g' on all groups in the topic
> I have that contains my IMAP groups.  For the record, I'm also using
> dovecot and k9-mail.

Are you saying that "g" in the summary does not allow viewing a missing
article, but M-g will?

To me, the problem is that these message are somehow invisible, even
after multiple attempts to fetch/display them.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Apr 18, 2024, Greg Troxel <gdt@lexort.com> wrote:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> For both of you, does hitting `M-g' with point on the group in the
>> *Group* buffer fix things?
>
> I will try that, but I have been exiting the group and ^U SPC to
> re-enter and display all, and when a message has gone missing, I have
> not even once been able to get gnus to show it.  I have exited and
> restarted gnus, and done "g".

Neither of those does the full resync that `M-g' does for IMAP groups.
I don't know what is going on behind the scenes, but `M-g' is necessary
for me to see up-to-date information sometimes.

> Are you saying that "g" in the summary does not allow viewing a missing
> article, but M-g will?

Yes, in many cases.

Dan

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Arash Esbati]

Dan Christensen <jdc@uwo.ca> writes:

> Neither of those does the full resync that `M-g' does for IMAP groups.

I can confirm this.

> I don't know what is going on behind the scenes, but `M-g' is necessary
> for me to see up-to-date information sometimes.

I haven't tried to read/understand the code, but in the Group buffer,
neither 'g' nor 'C-u g' does what 'M-g' does inside the Summary buffer
of an IMAP group.  And I would have expected that 'C-u g' in Group
buffer fits the bill:

,----[ C-h f gnus-group-get-new-news RET ]
| gnus-group-get-new-news is an autoloaded interactive subr-native-elisp
| in ‘gnus-group.el’.
| 
| (gnus-group-get-new-news &optional ARG ONE-LEVEL)
| 
| Get newly arrived articles.
| If ARG is a number, it specifies which levels you are interested in
| re-scanning.  If ARG is non-nil and not a number, this will force
| "hard" re-reading of the active files from all servers.
| If ONE-LEVEL is not nil, then re-scan only the specified level,
| otherwise all levels below ARG will be scanned too.
| 
`----

Hopefully Eric can shed some light on this.

Best, Arash

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Apr 21, 2024, Arash Esbati <arash@gnu.org> wrote:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> I don't know what is going on behind the scenes, but `M-g' is necessary
>> for me to see up-to-date information sometimes.
>
> I haven't tried to read/understand the code, but in the Group buffer,
> neither 'g' nor 'C-u g' does what 'M-g' does inside the Summary buffer
> of an IMAP group.

Just to clarify, I use `M-g' in the Group buffer, but looking at the
code I think using it in the Summary buffer does the same thing.

Aside:  While looking at the code, I think I noticed an error in the doc
string of the function that ends up getting called:

  (defun gnus-group-get-new-news-this-group (&optional n dont-scan)
    "Check for newly arrived news in the current group (and the N-1 next groups).
  The difference between N and the number of newsgroup checked is returned.
  If N is negative, this group and the N-1 previous groups will be checked.
  If DONT-SCAN is non-nil, scan non-activated groups as well."

I think the description of DONT-SCAN is inverted (as its name suggests).
It should say:

  If DONT-SCAN is nil, scan non-activated groups as well.

Or

  If DONT-SCAN is non-nil, then non-activated groups are not scanned.

Dan

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Andreas Schwab]

On Apr 21 2024, Arash Esbati wrote:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Neither of those does the full resync that `M-g' does for IMAP groups.
>
> I can confirm this.
>
>> I don't know what is going on behind the scenes, but `M-g' is necessary
>> for me to see up-to-date information sometimes.
>
> I haven't tried to read/understand the code, but in the Group buffer,
> neither 'g' nor 'C-u g' does what 'M-g' does inside the Summary buffer
> of an IMAP group.

There is also M-g (gnus-group-get-new-news-this-group) in the Group
buffer.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Eric Abrahamsen]

Arash Esbati <arash@gnu.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Neither of those does the full resync that `M-g' does for IMAP groups.
>
> I can confirm this.
>
>> I don't know what is going on behind the scenes, but `M-g' is necessary
>> for me to see up-to-date information sometimes.
>
> I haven't tried to read/understand the code, but in the Group buffer,
> neither 'g' nor 'C-u g' does what 'M-g' does inside the Summary buffer
> of an IMAP group.  And I would have expected that 'C-u g' in Group
> buffer fits the bill:
>
> ,----[ C-h f gnus-group-get-new-news RET ]
> | gnus-group-get-new-news is an autoloaded interactive subr-native-elisp
> | in ‘gnus-group.el’.
> | 
> | (gnus-group-get-new-news &optional ARG ONE-LEVEL)
> | 
> | Get newly arrived articles.
> | If ARG is a number, it specifies which levels you are interested in
> | re-scanning.  If ARG is non-nil and not a number, this will force
> | "hard" re-reading of the active files from all servers.
> | If ONE-LEVEL is not nil, then re-scan only the specified level,
> | otherwise all levels below ARG will be scanned too.
> | 
> `----
>
> Hopefully Eric can shed some light on this.

This is a long-standing bug, probably the most annoying thing about
nnimap. I've made a few attempts at it over the years, but so far
haven't been successful. I believe what it comes down to is that "M-g"
dumps Gnus' "active" number and re-reads it from the IMAP server, while
regular "g" incrementally modifies the active number based on newest
changes. So if anything goes wrong with Gnus' active number, errors will
accumulate from that point until the user does "M-g".

I've focused on figuring out why the active number goes wrong, but in
the interest of pragmatism it might be better just to make sure that
nnimap always does the equivalent of "M-g" at Gnus startup.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Arash Esbati]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> This is a long-standing bug, probably the most annoying thing about
> nnimap. I've made a few attempts at it over the years, but so far
> haven't been successful. I believe what it comes down to is that "M-g"
> dumps Gnus' "active" number and re-reads it from the IMAP server, while
> regular "g" incrementally modifies the active number based on newest
> changes. So if anything goes wrong with Gnus' active number, errors will
> accumulate from that point until the user does "M-g".

Thanks for your response.  I agree, this is indeed annoying -- hitting
on 'M-g' on every IMAP-Group or inside a Group isn't much fun.

> I've focused on figuring out why the active number goes wrong, but in
> the interest of pragmatism it might be better just to make sure that
> nnimap always does the equivalent of "M-g" at Gnus startup.

That might be a good compromise.

Best, Arash

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Greg Troxel]

Arash Esbati <arash@gnu.org> writes:

>> I've focused on figuring out why the active number goes wrong, but in
>> the interest of pragmatism it might be better just to make sure that
>> nnimap always does the equivalent of "M-g" at Gnus startup.
>
> That might be a good compromise.

Agreed.  It seems obvious (said knowing that's dangerous) that on
starting up the program, it's best not to rely on state that could be
wrong.   That's tricky, because mail programs keep messages and don't
refetch them, but relying on counts of unread seems very trouble prone,
vs "I have this message body and the timestamp matches so I won't fetch
it."

I think it would also be good to separate

   check for new

   get all the messages that are there

   really rescan and don't believe state

in terms of being much louder about the differences.  I have the
impression that the assembled gnususers here are at least somewhat hazy
on those boundaries.  At least I am.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Eric Abrahamsen]

Greg Troxel <gdt@lexort.com> writes:

> Arash Esbati <arash@gnu.org> writes:
>
>>> I've focused on figuring out why the active number goes wrong, but in
>>> the interest of pragmatism it might be better just to make sure that
>>> nnimap always does the equivalent of "M-g" at Gnus startup.
>>
>> That might be a good compromise.
>
> Agreed.  It seems obvious (said knowing that's dangerous) that on
> starting up the program, it's best not to rely on state that could be
> wrong.   That's tricky, because mail programs keep messages and don't
> refetch them, but relying on counts of unread seems very trouble prone,
> vs "I have this message body and the timestamp matches so I won't fetch
> it."
>
> I think it would also be good to separate
>
>    check for new
>
>    get all the messages that are there
>
>    really rescan and don't believe state
>
> in terms of being much louder about the differences.  I have the
> impression that the assembled gnususers here are at least somewhat hazy
> on those boundaries.  At least I am.

I think this is an artifact of Gnus starting out as a newsreader, and
adding email capability later. In a NNTP context, the read/unread
message count is entirely private to your local machine, and something
you can manipulate at will. In an IMAP context, it's very much a part of
multi-machine state. Gnus should always defer to the remote server,
particularly at start up. I do wish I knew where the bug was coming
from, though.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Greg Troxel <gdt@lexort.com> writes:
>
>> Arash Esbati <arash@gnu.org> writes:
>>
>>>> I've focused on figuring out why the active number goes wrong, but in
>>>> the interest of pragmatism it might be better just to make sure that
>>>> nnimap always does the equivalent of "M-g" at Gnus startup.
>>>
>>> That might be a good compromise.
>>
>> Agreed.  It seems obvious (said knowing that's dangerous) that on
>> starting up the program, it's best not to rely on state that could be
>> wrong.   That's tricky, because mail programs keep messages and don't
>> refetch them, but relying on counts of unread seems very trouble prone,
>> vs "I have this message body and the timestamp matches so I won't fetch
>> it."
>>
>> I think it would also be good to separate
>>
>>    check for new
>>
>>    get all the messages that are there
>>
>>    really rescan and don't believe state
>>
>> in terms of being much louder about the differences.  I have the
>> impression that the assembled gnususers here are at least somewhat hazy
>> on those boundaries.  At least I am.
>
> I think this is an artifact of Gnus starting out as a newsreader, and
> adding email capability later. In a NNTP context, the read/unread
> message count is entirely private to your local machine, and something
> you can manipulate at will. In an IMAP context, it's very much a part of
> multi-machine state. Gnus should always defer to the remote server,
> particularly at start up. I do wish I knew where the bug was coming
> from, though.
>
>
>

This makes much sense.  I'll try M-g the next time this happens.  Thanks
everyone!

Also I would expect the equivalent of M-g to happen on startup so that
there are less surprises.

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Xiyue Deng <manphiz@gmail.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Greg Troxel <gdt@lexort.com> writes:
>>
>>> Arash Esbati <arash@gnu.org> writes:
>>>
>>>>> I've focused on figuring out why the active number goes wrong, but in
>>>>> the interest of pragmatism it might be better just to make sure that
>>>>> nnimap always does the equivalent of "M-g" at Gnus startup.
>>>>
>>>> That might be a good compromise.
>>>
>>> Agreed.  It seems obvious (said knowing that's dangerous) that on
>>> starting up the program, it's best not to rely on state that could be
>>> wrong.   That's tricky, because mail programs keep messages and don't
>>> refetch them, but relying on counts of unread seems very trouble prone,
>>> vs "I have this message body and the timestamp matches so I won't fetch
>>> it."
>>>
>>> I think it would also be good to separate
>>>
>>>    check for new
>>>
>>>    get all the messages that are there
>>>
>>>    really rescan and don't believe state
>>>
>>> in terms of being much louder about the differences.  I have the
>>> impression that the assembled gnususers here are at least somewhat hazy
>>> on those boundaries.  At least I am.
>>
>> I think this is an artifact of Gnus starting out as a newsreader, and
>> adding email capability later. In a NNTP context, the read/unread
>> message count is entirely private to your local machine, and something
>> you can manipulate at will. In an IMAP context, it's very much a part of
>> multi-machine state. Gnus should always defer to the remote server,
>> particularly at start up. I do wish I knew where the bug was coming
>> from, though.
>>
>>
>>
>
> This makes much sense.  I'll try M-g the next time this happens.  Thanks
> everyone!
>
> Also I would expect the equivalent of M-g to happen on startup so that
> there are less surprises.

So the issue Greg described just happened again that an empty
folder/topic "Trash" from one of my outlook accounts just shown 38
unread messages and selecting it fails with "Can't select group", while
other clients like Thunderbird shows 0 in it.  I have tried "M-g" and it
doesn't help unfortunately :(

Any more ideas for debugging?

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Eric Abrahamsen]

Xiyue Deng <manphiz@gmail.com> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Greg Troxel <gdt@lexort.com> writes:
>>>
>>>> Arash Esbati <arash@gnu.org> writes:
>>>>
>>>>>> I've focused on figuring out why the active number goes wrong, but in
>>>>>> the interest of pragmatism it might be better just to make sure that
>>>>>> nnimap always does the equivalent of "M-g" at Gnus startup.
>>>>>
>>>>> That might be a good compromise.
>>>>
>>>> Agreed.  It seems obvious (said knowing that's dangerous) that on
>>>> starting up the program, it's best not to rely on state that could be
>>>> wrong.   That's tricky, because mail programs keep messages and don't
>>>> refetch them, but relying on counts of unread seems very trouble prone,
>>>> vs "I have this message body and the timestamp matches so I won't fetch
>>>> it."
>>>>
>>>> I think it would also be good to separate
>>>>
>>>>    check for new
>>>>
>>>>    get all the messages that are there
>>>>
>>>>    really rescan and don't believe state
>>>>
>>>> in terms of being much louder about the differences.  I have the
>>>> impression that the assembled gnususers here are at least somewhat hazy
>>>> on those boundaries.  At least I am.
>>>
>>> I think this is an artifact of Gnus starting out as a newsreader, and
>>> adding email capability later. In a NNTP context, the read/unread
>>> message count is entirely private to your local machine, and something
>>> you can manipulate at will. In an IMAP context, it's very much a part of
>>> multi-machine state. Gnus should always defer to the remote server,
>>> particularly at start up. I do wish I knew where the bug was coming
>>> from, though.
>>>
>>>
>>>
>>
>> This makes much sense.  I'll try M-g the next time this happens.  Thanks
>> everyone!
>>
>> Also I would expect the equivalent of M-g to happen on startup so that
>> there are less surprises.
>
> So the issue Greg described just happened again that an empty
> folder/topic "Trash" from one of my outlook accounts just shown 38
> unread messages and selecting it fails with "Can't select group", while
> other clients like Thunderbird shows 0 in it.  I have tried "M-g" and it
> doesn't help unfortunately :(
>
> Any more ideas for debugging?

Maybe move a message into the folder, then move it back out? I'll try to
get to debugging this area of Gnus in the next few weeks...

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>>> Greg Troxel <gdt@lexort.com> writes:
>>>>
>>>>> Arash Esbati <arash@gnu.org> writes:
>>>>>
>>>>>>> I've focused on figuring out why the active number goes wrong, but in
>>>>>>> the interest of pragmatism it might be better just to make sure that
>>>>>>> nnimap always does the equivalent of "M-g" at Gnus startup.
>>>>>>
>>>>>> That might be a good compromise.
>>>>>
>>>>> Agreed.  It seems obvious (said knowing that's dangerous) that on
>>>>> starting up the program, it's best not to rely on state that could be
>>>>> wrong.   That's tricky, because mail programs keep messages and don't
>>>>> refetch them, but relying on counts of unread seems very trouble prone,
>>>>> vs "I have this message body and the timestamp matches so I won't fetch
>>>>> it."
>>>>>
>>>>> I think it would also be good to separate
>>>>>
>>>>>    check for new
>>>>>
>>>>>    get all the messages that are there
>>>>>
>>>>>    really rescan and don't believe state
>>>>>
>>>>> in terms of being much louder about the differences.  I have the
>>>>> impression that the assembled gnususers here are at least somewhat hazy
>>>>> on those boundaries.  At least I am.
>>>>
>>>> I think this is an artifact of Gnus starting out as a newsreader, and
>>>> adding email capability later. In a NNTP context, the read/unread
>>>> message count is entirely private to your local machine, and something
>>>> you can manipulate at will. In an IMAP context, it's very much a part of
>>>> multi-machine state. Gnus should always defer to the remote server,
>>>> particularly at start up. I do wish I knew where the bug was coming
>>>> from, though.
>>>>
>>>>
>>>>
>>>
>>> This makes much sense.  I'll try M-g the next time this happens.  Thanks
>>> everyone!
>>>
>>> Also I would expect the equivalent of M-g to happen on startup so that
>>> there are less surprises.
>>
>> So the issue Greg described just happened again that an empty
>> folder/topic "Trash" from one of my outlook accounts just shown 38
>> unread messages and selecting it fails with "Can't select group", while
>> other clients like Thunderbird shows 0 in it.  I have tried "M-g" and it
>> doesn't help unfortunately :(
>>
>> Any more ideas for debugging?
>
> Maybe move a message into the folder, then move it back out?

Ah I was aware of this workaround.  Just wonder whether there is
metadata/debug info that I can try to get to help the developer such as
you to debug this?

> I'll try to get to debugging this area of Gnus in the next few
> weeks...

Really appreciate this!  Meanwhile probably I should try to submit a bug
for tracking.  Is there a Gnus specific bug tracker or I should just use
report-emacs-bug?

>
>
>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On May  2, 2024, Xiyue Deng <manphiz@gmail.com> wrote:

> Ah I was aware of this workaround.  Just wonder whether there is
> metadata/debug info that I can try to get to help the developer such as
> you to debug this?

You could hit `G p' on the group in the *Group* buffer and show us
what is displayed.  (You can hit `C-c C-k' in that buffer to kill it.)

I sometimes get a situation where the active range starts at 0 instead
of 1, and it's very hard to fix.  I fiddle around with changing things
in that buffer, closing emacs, etc, and can fix it, but I'm not sure
exactly what the recipe is.

Dan

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Hi Dan,

Dan Christensen <jdc@uwo.ca> writes:

> On May  2, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>
>> Ah I was aware of this workaround.  Just wonder whether there is
>> metadata/debug info that I can try to get to help the developer such as
>> you to debug this?
>
> You could hit `G p' on the group in the *Group* buffer and show us
> what is displayed.  (You can hit `C-c C-k' in that buffer to kill it.)
>

Thanks for the tip!  However I don't know what happened the count of
problematic topic was back to 0 since this morning, so I'll probably
have to wait for the next occurrence to try this.

> I sometimes get a situation where the active range starts at 0 instead
> of 1, and it's very hard to fix.  I fiddle around with changing things
> in that buffer, closing emacs, etc, and can fix it, but I'm not sure
> exactly what the recipe is.
>

Noted.  Will keep an eye on this next time.

> Dan
>
>
>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Xiyue Deng <manphiz@gmail.com> writes:

> Hi Dan,
>
> Dan Christensen <jdc@uwo.ca> writes:
>
>> On May  2, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>>
>>> Ah I was aware of this workaround.  Just wonder whether there is
>>> metadata/debug info that I can try to get to help the developer such as
>>> you to debug this?
>>
>> You could hit `G p' on the group in the *Group* buffer and show us
>> what is displayed.  (You can hit `C-c C-k' in that buffer to kill it.)
>>
>
> Thanks for the tip!  However I don't know what happened the count of
> problematic topic was back to 0 since this morning, so I'll probably
> have to wait for the next occurrence to try this.
>

So this happened again in one of my outlook accounts' Sent folder, and
the "G p" information is as below:

,----
| ((modseq)
|  (uidvalidity . "11")
|  (active 1 . 7)
|  (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent))
`----

>> I sometimes get a situation where the active range starts at 0 instead
>> of 1, and it's very hard to fix.  I fiddle around with changing things
>> in that buffer, closing emacs, etc, and can fix it, but I'm not sure
>> exactly what the recipe is.
>>
>
> Noted.  Will keep an eye on this next time.
>

This time the active range does start from 1.  So it seems that it does
think there are 7 active items?

>> Dan
>>
>>
>>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On May  5, 2024, Xiyue Deng <manphiz@gmail.com> wrote:

> So this happened again in one of my outlook accounts' Sent folder, and
> the "G p" information is as below:
>
> ,----
> | ((modseq)
> |  (uidvalidity . "11")
> |  (active 1 . 7)
> |  (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent))
> `----

I guess you are having a different problem than I am.  I assume that
you tried `M-g', and the above doesn't change?  What if you go into
the server buffer with `^' and then hit `C' on the server to close it
and `O' on the server to open it?

Dan

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Dan Christensen <jdc@uwo.ca> writes:

> On May  5, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>
>> So this happened again in one of my outlook accounts' Sent folder, and
>> the "G p" information is as below:
>>
>> ,----
>> | ((modseq)
>> |  (uidvalidity . "11")
>> |  (active 1 . 7)
>> |  (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent))
>> `----
>
> I guess you are having a different problem than I am.  I assume that
> you tried `M-g', and the above doesn't change?

Yes, and it didn't fix it.

> What if you go into the server buffer with `^' and then hit `C' on the
> server to close it and `O' on the server to open it?
>

Tried this as well, and unfortunately it didn't fix it either.

> Dan
>
>
>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Xiyue Deng <manphiz@gmail.com> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> On May  5, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>>
>>> So this happened again in one of my outlook accounts' Sent folder, and
>>> the "G p" information is as below:
>>>
>>> ,----
>>> | ((modseq)
>>> |  (uidvalidity . "11")
>>> |  (active 1 . 7)
>>> |  (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent))
>>> `----
>>
>> I guess you are having a different problem than I am.  I assume that
>> you tried `M-g', and the above doesn't change?
>
> Yes, and it didn't fix it.
>
>> What if you go into the server buffer with `^' and then hit `C' on the
>> server to close it and `O' on the server to open it?
>>
>
> Tried this as well, and unfortunately it didn't fix it either.
>

Just wonder is there anything else that I can try to debug this?  In
worst case I can just use the moving-messages-in-and-out workaround to
reset the count, but wondering whether there is any other things to try.

>> Dan
>>
>>
>>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Xiyue Deng <manphiz@gmail.com> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> On May  5, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>>
>>> So this happened again in one of my outlook accounts' Sent folder, and
>>> the "G p" information is as below:
>>>
>>> ,----
>>> | ((modseq)
>>> |  (uidvalidity . "11")
>>> |  (active 1 . 7)
>>> |  (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent))
>>> `----
>>
>> I guess you are having a different problem than I am.  I assume that
>> you tried `M-g', and the above doesn't change?
>
> Yes, and it didn't fix it.
>
>> What if you go into the server buffer with `^' and then hit `C' on the
>> server to close it and `O' on the server to open it?
>>
>
> Tried this as well, and unfortunately it didn't fix it either.
>

Just wonder is there anything else that I can try to debug this?  In
worst case I can just use the moving-messages-in-and-out workaround to
reset the count, but wondering whether there is any other things to try
and hopefully help identifying the issue from the Gnus side.

>> Dan
>>
>>
>>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Andrew Cohen]

>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes:

    XD> Xiyue Deng <manphiz@gmail.com> writes:
    >> Dan Christensen <jdc@uwo.ca> writes:
    >> 
    >>> On May  5, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
    >>> 
    >>>> So this happened again in one of my outlook accounts' Sent
    >>>> folder, and the "G p" information is as below:
    >>>> 
    >>>> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 7) |
    >>>> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft
    >>>> $MDNSent)) `----
    >>> 

When the count is restored to normal, does "G p" show modseq still
without a value?

-- 
Andrew Cohen

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Andrew Cohen <acohen@ust.hk> writes:

>>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes:
>
>     XD> Xiyue Deng <manphiz@gmail.com> writes:
>     >> Dan Christensen <jdc@uwo.ca> writes:
>     >> 
>     >>> On May  5, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>     >>> 
>     >>>> So this happened again in one of my outlook accounts' Sent
>     >>>> folder, and the "G p" information is as below:
>     >>>> 
>     >>>> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 7) |
>     >>>> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft
>     >>>> $MDNSent)) `----
>     >>> 
>
> When the count is restored to normal, does "G p" show modseq still
> without a value?

Yes.  This is "G p" after I used the move-in-move-out trick to reset the
count:

,----
| ((modseq)
|  (uidvalidity . "11")
|  (active 1 . 12)
|  (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent))
`----


-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Andrew Cohen]

>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes:

    >> When the count is restored to normal, does "G p" show modseq
    >> still without a value?

    XD> Yes.  This is "G p" after I used the move-in-move-out trick to
    XD> reset the count:

    XD> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 12) |
    XD> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft
    XD> $MDNSent)) `----

Thanks. It seems that two different cases have been reported: those
which are fixed with "M-g" and those (like this case of yours) that are
not.  I was hoping that the absence of a value for modseq might be a
clue as to the difference, but apparently not :(

The absence of this value suggests that the server doesn't support
QRESYNC. Out of curiosity, what server is this? I think most of the
common ones support it.

-- 
Andrew Cohen

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Andrew Cohen <acohen@ust.hk> writes:

>>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes:
>
>     >> When the count is restored to normal, does "G p" show modseq
>     >> still without a value?
>
>     XD> Yes.  This is "G p" after I used the move-in-move-out trick to
>     XD> reset the count:
>
>     XD> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 12) |
>     XD> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft
>     XD> $MDNSent)) `----
>
> Thanks. It seems that two different cases have been reported: those
> which are fixed with "M-g" and those (like this case of yours) that are
> not.  I was hoping that the absence of a value for modseq might be a
> clue as to the difference, but apparently not :(
>
> The absence of this value suggests that the server doesn't support
> QRESYNC. Out of curiosity, what server is this? I think most of the
> common ones support it.

This is an outlook account.  AAMOF this happened to both of my gmail and
outlook accounts, where empty topic showing unread count like this case
happens in outlook accounts more often.

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Andreas Schwab]

On Mai 10 2024, Andrew Cohen wrote:

>>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes:
>
>     >> When the count is restored to normal, does "G p" show modseq
>     >> still without a value?
>
>     XD> Yes.  This is "G p" after I used the move-in-move-out trick to
>     XD> reset the count:
>
>     XD> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 12) |
>     XD> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft
>     XD> $MDNSent)) `----
>
> Thanks. It seems that two different cases have been reported: those
> which are fixed with "M-g" and those (like this case of yours) that are
> not.  I was hoping that the absence of a value for modseq might be a
> clue as to the difference, but apparently not :(

The modseq value comes from the HIGHESTMODSEQ response (see RFC 4551,
3.1.1).  If the HIGHESTMODSEQ response is missing, then the server does
not support the persistent storage of mod-sequences.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Greg Troxel]

Xiyue Deng <manphiz@gmail.com> writes:

> This makes much sense.  I'll try M-g the next time this happens.  Thanks
> everyone!

I had a situation where a message arrived in INBOX.  I could see it in
the filesystem dovecot reads, and in other clients.  It was as if it did
not exist in gnus.

M-g on the group in *Group* buffer seems to have resolved that.
It is interesting that moving out and back resolved what I think is the
same situation in the past.

Just making up stuff without basis:  It's like a missing wakeup in a
synchronization protocol.


> Also I would expect the equivalent of M-g to happen on startup so that
> there are less surprises.

Agreed, but I can see the point of not blowing caches just because of startup.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Hi Greg,

Greg Troxel <gdt@lexort.com> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> This makes much sense.  I'll try M-g the next time this happens.  Thanks
>> everyone!
>
> I had a situation where a message arrived in INBOX.  I could see it in
> the filesystem dovecot reads, and in other clients.  It was as if it did
> not exist in gnus.
>
> M-g on the group in *Group* buffer seems to have resolved that.
> It is interesting that moving out and back resolved what I think is the
> same situation in the past.
>
> Just making up stuff without basis:  It's like a missing wakeup in a
> synchronization protocol.
>
>

Unfortunately, it looks like it doesn't fix it for me, presumably
because I'm using nnimap instead of reading from a local mailbox.
Therefore it feels natural to assume a bug in nnimap, though it's still
years earlier before I can figure it out from the code, if that ever
happens :P

>> Also I would expect the equivalent of M-g to happen on startup so that
>> there are less surprises.
>
> Agreed, but I can see the point of not blowing caches just because of startup.

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Greg Troxel]

Xiyue Deng <manphiz@gmail.com> writes:

> Hi Greg,
>
> Greg Troxel <gdt@lexort.com> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> This makes much sense.  I'll try M-g the next time this happens.  Thanks
>>> everyone!
>>
>> I had a situation where a message arrived in INBOX.  I could see it in
>> the filesystem dovecot reads, and in other clients.  It was as if it did
>> not exist in gnus.
>>
>> M-g on the group in *Group* buffer seems to have resolved that.
>> It is interesting that moving out and back resolved what I think is the
>> same situation in the past.
>>
>> Just making up stuff without basis:  It's like a missing wakeup in a
>> synchronization protocol.
>
> Unfortunately, it looks like it doesn't fix it for me, presumably
> because I'm using nnimap instead of reading from a local mailbox.
> Therefore it feels natural to assume a bug in nnimap, though it's still
> years earlier before I can figure it out from the code, if that ever
> happens :P

This was nnimap in my case, with emacs and dovecot on the same machine
if that matters.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Greg Troxel <gdt@lexort.com> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Hi Greg,
>>
>> Greg Troxel <gdt@lexort.com> writes:
>>
>>> Xiyue Deng <manphiz@gmail.com> writes:
>>>
>>>> This makes much sense.  I'll try M-g the next time this happens.  Thanks
>>>> everyone!
>>>
>>> I had a situation where a message arrived in INBOX.  I could see it in
>>> the filesystem dovecot reads, and in other clients.  It was as if it did
>>> not exist in gnus.
>>>
>>> M-g on the group in *Group* buffer seems to have resolved that.
>>> It is interesting that moving out and back resolved what I think is the
>>> same situation in the past.
>>>
>>> Just making up stuff without basis:  It's like a missing wakeup in a
>>> synchronization protocol.
>>
>> Unfortunately, it looks like it doesn't fix it for me, presumably
>> because I'm using nnimap instead of reading from a local mailbox.
>> Therefore it feels natural to assume a bug in nnimap, though it's still
>> years earlier before I can figure it out from the code, if that ever
>> happens :P
>
> This was nnimap in my case, with emacs and dovecot on the same machine
> if that matters.
>
>

Ah I see.  Then probably the IMAP implementation of Outlook is just bad
;-)

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Greg Troxel]

Xiyue Deng <manphiz@gmail.com> writes:

>> This was nnimap in my case, with emacs and dovecot on the same machine
>> if that matters.
>
> Ah I see.  Then probably the IMAP implementation of Outlook is just bad
> ;-)

I believe there is a gnus bug that I am hitting, and that others are
too.

I am entirely willing to believe the the theory that there is also a bug
in Outlook IMAP....

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Eric Abrahamsen]

Greg Troxel <gdt@lexort.com> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>>> This was nnimap in my case, with emacs and dovecot on the same machine
>>> if that matters.
>>
>> Ah I see.  Then probably the IMAP implementation of Outlook is just bad
>> ;-)
>
> I believe there is a gnus bug that I am hitting, and that others are
> too.
>
> I am entirely willing to believe the the theory that there is also a bug
> in Outlook IMAP....

I was going to say, the IMAP implementation of Outlook is indeed bad!
But that's "good" that you're hitting this issue with Dovecot running
locally, as that's a setup with fewer confounding variables, and also
what I run, so maybe we stand a better chance of reproducing.

Maybe it's time to dive in :( Greg, if you're willing, would you provide
as much details as possible about your setup, and anything/everything
you can remember about what happened/happens before you see this error?

All your nnimap settings are of interest, obviously, but you might as
well send anything that could possibly be relevant, off-list if you'd
prefer.

And if you do see the error crop up again, please let us know what you
were doing immediately before, and if at all possible, leave Gnus in the
broken state until we can do some remote debugging.

Thanks!
Eric

Gnus sometimes fails to see a message that is actually in IMAP

[Greg Troxel]

I have changed the subject, as I am not at all sure that my problem is
the same as "reports new messages but not showing them on IMAP server".
In my case, it seems to me as if there is a filter between gnus and the
IMAP server that redacts a message.


Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I was going to say, the IMAP implementation of Outlook is indeed bad!
> But that's "good" that you're hitting this issue with Dovecot running
> locally, as that's a setup with fewer confounding variables, and also
> what I run, so maybe we stand a better chance of reproducing.

Yes, it does seem more tractable.

One thing I find troublesome about modern gnus is that it is part of
emacs.  I'm currently running 28.2, so that's the gnus I have.  Years
before, when gnus was in a separate repo, I was running up-to-date gnus
with stable emacs.   If I'm confused and we should be debugging gnus
trunk please let me know.

I probably can update to 29.x without trouble, and I should do that.  I
try to stay away from bleeding edge including .0, when the older version
isn't crufty and isn't troublesome, for things I want to use rather than
hack on.  I don't really want to run emacs-current.

> Maybe it's time to dive in :( Greg, if you're willing, would you provide
> as much details as possible about your setup, and anything/everything
> you can remember about what happened/happens before you see this error?

First, I have no perception of anything unusual or remarkable happening
when the problem happens.  Mail arrives, apparently just like any other
message, I see it in K-9 (Android client that seems very well behaved),
and I then go to look for it in gnus (to answer, because I have a
keyboard) and find it's not there.  This is infrequent, maybe once a
month, and there is nothing strange about these messages.  They are from
people I've heard from before, are not necessarily even large, and "more
$filename" in the IMAP dir looks unremarkable.

When this happens, gnus feels entirely normal.  I can still enter INBOX,
read arriving mail, enter with "100 <space>" to see already read mail,
etc.  If I were only using gnus I would have had no idea anything was
wrong, until someone asked me why I didn't respond to some message.

The missing messages are in $HOME/IMAP/cur because they've been read by
another client.  Thunderbird sees the missing messages too.

I have previously tried restarting dovecot and removing dovecot caches,
but this has never fixed the issue.  I am 95% convinced that the is
issue is in gnus.

My setup is:

  - computer runs NetBSD 9, postfix receiving mail from the Internet at
    large, with lots of spam filtering, postfix delivering to $HOME/IMAP
    as maildir, hence into $HOME/IMAP/new.  While I have to mess with
    spam filtering from time to time, it is not flaky - just minor
    misclassification - and the messages in question are 1) not spam and
    2) delivered to INBOX.

  - dovecot 2.3.21 serves $HOME/IMAP.  Many updates have happened since
    I first noticed this -- it's been several years I think (but no
    notes).

  - dovecot has an x509 cert (that validates under pkix rules), and
    listens only on the "implicit TLS" port, meaning that it expects TLS
    negotation right away vs STARTTLS.  (I'm firmly in the "cleartext
    connections should be adminitratively prohibited" camp.)

  - dovecot's uidvalidity as shown in "G p" is a number that when
    interpreted as a timeval is noonish EDT on a Sunday in March of
    2006.  Indeed the IMAP spec says that servers should try not to
    change that.  That's wild, but probably when I installed dovecot
    version 1 on a different computer.  I have upgraded dovecot,
    upgraded the OS version, and migrated to different hardware with a
    different hostname since.  The point is that this problem is almost
    certainly not about gnus' IMAP implemention getting confused about
    uidvalidity changes!

  - emacs with gnus is left running always, and accessed via tmux.  The
    machine is very stable, basically getting rebooted for software
    updates.

  - emacs 28.2 gnus has nnimap configured

    (setq gnus-secondary-select-methods
          '((nnimap "foo.example.com"
                    (nnimap-address "bar.example.com")
                    (nnimap-port 993)
                    (nnimap-stream ssl)
                    (nnimap-authenticator login)
                    (nnir-search-engine imap)
                    )
            (nntp "news.gwene.org")
            )
          )
   where foo is a historical name and bar.example.com is the actual name
   of the host (with emacs and the imap server both).

   The connection goes over IPv6, not ::1, but the actual IP address.
   I would have used localhost, but then the certificate is not valid.

   (Yes, I know the gwene/gmane world is messy and that it's possible I
   should be stopping this or doing it differently, but I don't think it
   is relevant.)

  - I had vestigial config that I just dropped, to add --insecure to
    gnutls-cli, for a historical collection of self-signed certs,
    connecting to localhost, emacs having the wrong path to ssl config,
    and me not straightening out my
    copy-of-config-in-linuxy-pathname-for-emacs.  In the end, it was
    just adding "--insecure" to the gnutls-cli invocation.  I can't
    believe that matters, but mentioning it for completeness.

> And if you do see the error crop up again, please let us know what you
> were doing immediately before, and if at all possible, leave Gnus in the
> broken state until we can do some remote debugging.

Thanks for offering to spend time on this.  I'll write offlist to see
about coordinating.

It does seem like looking at group properties is useful and I'll do that.

I'll at least take better notes.  I think but am not 100% sure that the
broken state survives exiting/restarting gnus, and exiting/restarting
emacs.

My previous workaround was move-out move-back.
Now, M-g on the group seems to fix things.

It would help to understand the big picture of how gnus caches imap
state.  Ideally that would be in the manual; I'll take another look at
that.

It would also be interesting to have a gnus-imap-revalidate function that
for all cached information, fetches it and compares, complaining loudly
if different.  I have used this technique when debugging routing code to
end up with only correct cached information, and didn't think of any
other way to find rare bugs.

Greg

Re: Gnus sometimes fails to see a message that is actually in IMAP

[Eric Abrahamsen]

Greg Troxel <gdt@lexort.com> writes:

Wow, this is a ton of information! Thanks for all the background. Kudos
on using TLS on your own machine, something I have never had the
patience to set up.

> I have changed the subject, as I am not at all sure that my problem is
> the same as "reports new messages but not showing them on IMAP server".
> In my case, it seems to me as if there is a filter between gnus and the
> IMAP server that redacts a message.
>
>
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I was going to say, the IMAP implementation of Outlook is indeed bad!
>> But that's "good" that you're hitting this issue with Dovecot running
>> locally, as that's a setup with fewer confounding variables, and also
>> what I run, so maybe we stand a better chance of reproducing.
>
> Yes, it does seem more tractable.
>
> One thing I find troublesome about modern gnus is that it is part of
> emacs.  I'm currently running 28.2, so that's the gnus I have.  Years
> before, when gnus was in a separate repo, I was running up-to-date gnus
> with stable emacs.   If I'm confused and we should be debugging gnus
> trunk please let me know.

Not confused, of course, but I do think that Emacs 28 is a little bit
too old for this sort of debugging. Ideally we'd be doing it on master,
though 29 is still worthwhile, if you can update to that. There are good
things in the past couple of versions.


[...]

> It would also be interesting to have a gnus-imap-revalidate function that
> for all cached information, fetches it and compares, complaining loudly
> if different.  I have used this technique when debugging routing code to
> end up with only correct cached information, and didn't think of any
> other way to find rare bugs.

This is a really good idea. There's been talk on this list of forcibly
resetting Gnus to the remote server's state at start-up, which would be
something of a band-aid, but this would stand us a better chance of
pinning down the bug. People could run it at Gnus startup and shutdown,
as well as on-demand. I'll look into putting this together.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[James Thomas]

Eric Abrahamsen wrote:

> I've focused on figuring out why the active number goes wrong

On that note, here's something I'd been meaning to debug myself
sometime:

- (Preferably with an empty drafts folder) Compose a message and save it
- Open the drafts folder, press e on the message and then kill the new
  buffer; then (incidentally, if you do '/ N' then this bug does not
  arise) delete the message (B DEL)
- Press q
- The message count is wrong

I don't know if this is related to the above. Btw could you share it if
there's a bug report for the former?

--

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Eric Abrahamsen]

James Thomas <jimjoe@gmx.net> writes:

> Eric Abrahamsen wrote:
>
>> I've focused on figuring out why the active number goes wrong
>
> On that note, here's something I'd been meaning to debug myself
> sometime:
>
> - (Preferably with an empty drafts folder) Compose a message and save it
> - Open the drafts folder, press e on the message and then kill the new
>   buffer; then (incidentally, if you do '/ N' then this bug does not
>   arise) delete the message (B DEL)
> - Press q
> - The message count is wrong

That's a new one for me, but I was able to reproduce it. Would you open
a bug report with that recipe?

> I don't know if this is related to the above. Btw could you share it if
> there's a bug report for the former?

I don't think there is, mostly because it's kind of a vague problem.
I wouldn't be surprised if it ended up being several things at once.