Gnus sometimes reports new messages but not showing them on IMAP server

Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Hi,

I'm having some issues that Gnus sometimes reports that there are new
messages on an IMAP server but it doesn't show them.  In my case it's
using Gmail IMAP, and if I check using other mail clients
(e.g. Thunderbird) or directly check on Gmail website I can seem them.
I'm not sure whether this issue is specific to IMAP either.  As I'm not
sure how to get more info, I didn't use report bug yet, but would like
to ask around and see whether this is a known issue, or otherwise I can
get some suggestions to get more debug info before I formally file a
bug.

Any suggestions are welcome!  TIA!

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Greg Troxel]

Xiyue Deng <manphiz@gmail.com> writes:

> I'm having some issues that Gnus sometimes reports that there are new
> messages on an IMAP server but it doesn't show them.  In my case it's
> using Gmail IMAP, and if I check using other mail clients
> (e.g. Thunderbird) or directly check on Gmail website I can seem them.
> I'm not sure whether this issue is specific to IMAP either.  As I'm not
> sure how to get more info, I didn't use report bug yet, but would like
> to ask around and see whether this is a known issue, or otherwise I can
> get some suggestions to get more debug info before I formally file a
> bug.

I have been having a problem for several years that may be the same
thing.

My mail is in IMAP via dovecot.  Other than the following I have zero
problems with it.

I read mail with gnus (emacs built w/o x11 running in tmux in an xterm,
on NetBSD, currently 28.2 and thus the included gnus), with about 300
folders, about half of which have server-side filing of list mail.  I
also use K-9 Mail on Android (INBOX and maybe 3 more mailboxes, more or
less, ignoring the folders for lists and storage).  Occasionally I use
thunderbird for INBOX, to read something that should have been plain
text and isn't :-) but this issue has happened with no tbird running at
all.

Often, I get a message on my phone that I just leave (marked read) to
deal with later.  Sometimes, maybe 1 in 50, I can't find it in gnus, and
on looking harder find that it is just not showing up there.  Yet K-9
sees it as present and looking in ~/IMAP/cur shows it was well.
Whenever this has happened and I look with Thunderbird, it sees it fine
also.

I have found that if I move the message to another folder (I have TMP
for this very purpose) and then back to INBOX, it can then be found in
gnus.  So something about the message as it sits in the folder was
triggering gnus to miss it, while the very same message refiled does not
trigger this.

I have not been able to determine any pattern of which messages get this
disappearing treatment.  It's not really the message, since moving and
back fixes it.

Early on, I sort of suspected dovecot, but other people don't have
trouble with dovecot, and I have had zero flakiness other than this
specific issue.

I have not perceived "gnus says there are new messages but there
aren't".  I am not at all sure I would have noticed, especially if gnus
said 6 and there were 5.

In your case, if you do "100<spc>" in group, do you see the missing
message?  If not, you should be able to turn on some kind of debugging
to figure out what's going on, but I am unclear on the details.

Greg

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Greg Troxel <gdt@lexort.com> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> I'm having some issues that Gnus sometimes reports that there are new
>> messages on an IMAP server but it doesn't show them.  In my case it's
>> using Gmail IMAP, and if I check using other mail clients
>> (e.g. Thunderbird) or directly check on Gmail website I can seem them.
>> I'm not sure whether this issue is specific to IMAP either.  As I'm not
>> sure how to get more info, I didn't use report bug yet, but would like
>> to ask around and see whether this is a known issue, or otherwise I can
>> get some suggestions to get more debug info before I formally file a
>> bug.
>
> I have been having a problem for several years that may be the same
> thing.
>
> My mail is in IMAP via dovecot.  Other than the following I have zero
> problems with it.
>
> I read mail with gnus (emacs built w/o x11 running in tmux in an xterm,
> on NetBSD, currently 28.2 and thus the included gnus), with about 300
> folders, about half of which have server-side filing of list mail.  I
> also use K-9 Mail on Android (INBOX and maybe 3 more mailboxes, more or
> less, ignoring the folders for lists and storage).  Occasionally I use
> thunderbird for INBOX, to read something that should have been plain
> text and isn't :-) but this issue has happened with no tbird running at
> all.
>
> Often, I get a message on my phone that I just leave (marked read) to
> deal with later.  Sometimes, maybe 1 in 50, I can't find it in gnus, and
> on looking harder find that it is just not showing up there.  Yet K-9
> sees it as present and looking in ~/IMAP/cur shows it was well.
> Whenever this has happened and I look with Thunderbird, it sees it fine
> also.
>

Exactly this symptom.

> I have found that if I move the message to another folder (I have TMP
> for this very purpose) and then back to INBOX, it can then be found in
> gnus.  So something about the message as it sits in the folder was
> triggering gnus to miss it, while the very same message refiled does not
> trigger this.
>

I haven't tried this in this case, but there is actually another issue
that I haven't talked about: sometimes one of the empty folders may
report a number of new messages in it!  In one of the occurrences it
reports 52 new message on an empty folder.  The solution, however, is
similar to what you did but reverse: you move a number of messages from
another folder to it, and the new message count will reduce by the
number of messages you moved!  So in the previous case if you move 10
messages to the folder, the count becomes 42.  You move them out, and
move them back again, 32, etc.

Let me also try your solution in future.

> I have not been able to determine any pattern of which messages get this
> disappearing treatment.  It's not really the message, since moving and
> back fixes it.
>
> Early on, I sort of suspected dovecot, but other people don't have
> trouble with dovecot, and I have had zero flakiness other than this
> specific issue.
>
> I have not perceived "gnus says there are new messages but there
> aren't".  I am not at all sure I would have noticed, especially if gnus
> said 6 and there were 5.
>

I have only encountered such cases when the folder is empty.

> In your case, if you do "100<spc>" in group, do you see the missing
> message?  If not, you should be able to turn on some kind of debugging
> to figure out what's going on, but I am unclear on the details.
>

No, specifying a number didn't work for me, unfortunately.

And I guess to really understand this issue one may have to read the
code of Gnus or IMAP backend or hopefully someone that does may be able
to help.

> Greg
>
>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Apr 11, 2024, Xiyue Deng <manphiz@gmail.com> wrote:

> I'm having some issues that Gnus sometimes reports that there are new
> messages on an IMAP server but it doesn't show them.

On Apr 16, 2024, Greg Troxel <gdt@lexort.com> wrote:

> Often, I get a message on my phone that I just leave (marked read) to
> deal with later.  Sometimes, maybe 1 in 50, I can't find it in gnus, and
> on looking harder find that it is just not showing up there.

For both of you, does hitting `M-g' with point on the group in the
*Group* buffer fix things?

I regularly have similar problems, and I've resorted to binding the
following function to a key:

(defun jdc-gnus-update-IMAP ()
  (interactive)
  ; Only run this in the group buffer or it corrupts other buffers:
  (switch-to-buffer gnus-group-buffer)
  (gnus-topic-jump-to-topic "IMAP groups")
  (gnus-topic-get-new-news-this-topic)
  (gnus-group-list-groups) ; "l", as previous command shows all groups in topic
  )

I think it's doing the equivalent of `M-g' on all groups in the topic
I have that contains my IMAP groups.  For the record, I'm also using
dovecot and k9-mail.

Dan

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Greg Troxel]

Dan Christensen <jdc@uwo.ca> writes:

> On Apr 11, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>
>> I'm having some issues that Gnus sometimes reports that there are new
>> messages on an IMAP server but it doesn't show them.
>
> On Apr 16, 2024, Greg Troxel <gdt@lexort.com> wrote:
>
>> Often, I get a message on my phone that I just leave (marked read) to
>> deal with later.  Sometimes, maybe 1 in 50, I can't find it in gnus, and
>> on looking harder find that it is just not showing up there.
>
> For both of you, does hitting `M-g' with point on the group in the
> *Group* buffer fix things?

I will try that, but I have been exiting the group and ^U SPC to
re-enter and display all, and when a message has gone missing, I have
not even once been able to get gnus to show it.  I have exited and
restarted gnus, and done "g".

> I regularly have similar problems, and I've resorted to binding the
> following function to a key:
>
> (defun jdc-gnus-update-IMAP ()
>   (interactive)
>   ; Only run this in the group buffer or it corrupts other buffers:
>   (switch-to-buffer gnus-group-buffer)
>   (gnus-topic-jump-to-topic "IMAP groups")
>   (gnus-topic-get-new-news-this-topic)
>   (gnus-group-list-groups) ; "l", as previous command shows all groups in topic
>   )
>
> I think it's doing the equivalent of `M-g' on all groups in the topic
> I have that contains my IMAP groups.  For the record, I'm also using
> dovecot and k9-mail.

Are you saying that "g" in the summary does not allow viewing a missing
article, but M-g will?

To me, the problem is that these message are somehow invisible, even
after multiple attempts to fetch/display them.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Apr 18, 2024, Greg Troxel <gdt@lexort.com> wrote:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> For both of you, does hitting `M-g' with point on the group in the
>> *Group* buffer fix things?
>
> I will try that, but I have been exiting the group and ^U SPC to
> re-enter and display all, and when a message has gone missing, I have
> not even once been able to get gnus to show it.  I have exited and
> restarted gnus, and done "g".

Neither of those does the full resync that `M-g' does for IMAP groups.
I don't know what is going on behind the scenes, but `M-g' is necessary
for me to see up-to-date information sometimes.

> Are you saying that "g" in the summary does not allow viewing a missing
> article, but M-g will?

Yes, in many cases.

Dan

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Arash Esbati]

Dan Christensen <jdc@uwo.ca> writes:

> Neither of those does the full resync that `M-g' does for IMAP groups.

I can confirm this.

> I don't know what is going on behind the scenes, but `M-g' is necessary
> for me to see up-to-date information sometimes.

I haven't tried to read/understand the code, but in the Group buffer,
neither 'g' nor 'C-u g' does what 'M-g' does inside the Summary buffer
of an IMAP group.  And I would have expected that 'C-u g' in Group
buffer fits the bill:

,----[ C-h f gnus-group-get-new-news RET ]
| gnus-group-get-new-news is an autoloaded interactive subr-native-elisp
| in ‘gnus-group.el’.
| 
| (gnus-group-get-new-news &optional ARG ONE-LEVEL)
| 
| Get newly arrived articles.
| If ARG is a number, it specifies which levels you are interested in
| re-scanning.  If ARG is non-nil and not a number, this will force
| "hard" re-reading of the active files from all servers.
| If ONE-LEVEL is not nil, then re-scan only the specified level,
| otherwise all levels below ARG will be scanned too.
| 
`----

Hopefully Eric can shed some light on this.

Best, Arash

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Apr 21, 2024, Arash Esbati <arash@gnu.org> wrote:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> I don't know what is going on behind the scenes, but `M-g' is necessary
>> for me to see up-to-date information sometimes.
>
> I haven't tried to read/understand the code, but in the Group buffer,
> neither 'g' nor 'C-u g' does what 'M-g' does inside the Summary buffer
> of an IMAP group.

Just to clarify, I use `M-g' in the Group buffer, but looking at the
code I think using it in the Summary buffer does the same thing.

Aside:  While looking at the code, I think I noticed an error in the doc
string of the function that ends up getting called:

  (defun gnus-group-get-new-news-this-group (&optional n dont-scan)
    "Check for newly arrived news in the current group (and the N-1 next groups).
  The difference between N and the number of newsgroup checked is returned.
  If N is negative, this group and the N-1 previous groups will be checked.
  If DONT-SCAN is non-nil, scan non-activated groups as well."

I think the description of DONT-SCAN is inverted (as its name suggests).
It should say:

  If DONT-SCAN is nil, scan non-activated groups as well.

Or

  If DONT-SCAN is non-nil, then non-activated groups are not scanned.

Dan

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Andreas Schwab]

On Apr 21 2024, Arash Esbati wrote:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Neither of those does the full resync that `M-g' does for IMAP groups.
>
> I can confirm this.
>
>> I don't know what is going on behind the scenes, but `M-g' is necessary
>> for me to see up-to-date information sometimes.
>
> I haven't tried to read/understand the code, but in the Group buffer,
> neither 'g' nor 'C-u g' does what 'M-g' does inside the Summary buffer
> of an IMAP group.

There is also M-g (gnus-group-get-new-news-this-group) in the Group
buffer.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Eric Abrahamsen]

Arash Esbati <arash@gnu.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Neither of those does the full resync that `M-g' does for IMAP groups.
>
> I can confirm this.
>
>> I don't know what is going on behind the scenes, but `M-g' is necessary
>> for me to see up-to-date information sometimes.
>
> I haven't tried to read/understand the code, but in the Group buffer,
> neither 'g' nor 'C-u g' does what 'M-g' does inside the Summary buffer
> of an IMAP group.  And I would have expected that 'C-u g' in Group
> buffer fits the bill:
>
> ,----[ C-h f gnus-group-get-new-news RET ]
> | gnus-group-get-new-news is an autoloaded interactive subr-native-elisp
> | in ‘gnus-group.el’.
> | 
> | (gnus-group-get-new-news &optional ARG ONE-LEVEL)
> | 
> | Get newly arrived articles.
> | If ARG is a number, it specifies which levels you are interested in
> | re-scanning.  If ARG is non-nil and not a number, this will force
> | "hard" re-reading of the active files from all servers.
> | If ONE-LEVEL is not nil, then re-scan only the specified level,
> | otherwise all levels below ARG will be scanned too.
> | 
> `----
>
> Hopefully Eric can shed some light on this.

This is a long-standing bug, probably the most annoying thing about
nnimap. I've made a few attempts at it over the years, but so far
haven't been successful. I believe what it comes down to is that "M-g"
dumps Gnus' "active" number and re-reads it from the IMAP server, while
regular "g" incrementally modifies the active number based on newest
changes. So if anything goes wrong with Gnus' active number, errors will
accumulate from that point until the user does "M-g".

I've focused on figuring out why the active number goes wrong, but in
the interest of pragmatism it might be better just to make sure that
nnimap always does the equivalent of "M-g" at Gnus startup.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Arash Esbati]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> This is a long-standing bug, probably the most annoying thing about
> nnimap. I've made a few attempts at it over the years, but so far
> haven't been successful. I believe what it comes down to is that "M-g"
> dumps Gnus' "active" number and re-reads it from the IMAP server, while
> regular "g" incrementally modifies the active number based on newest
> changes. So if anything goes wrong with Gnus' active number, errors will
> accumulate from that point until the user does "M-g".

Thanks for your response.  I agree, this is indeed annoying -- hitting
on 'M-g' on every IMAP-Group or inside a Group isn't much fun.

> I've focused on figuring out why the active number goes wrong, but in
> the interest of pragmatism it might be better just to make sure that
> nnimap always does the equivalent of "M-g" at Gnus startup.

That might be a good compromise.

Best, Arash

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Greg Troxel]

Arash Esbati <arash@gnu.org> writes:

>> I've focused on figuring out why the active number goes wrong, but in
>> the interest of pragmatism it might be better just to make sure that
>> nnimap always does the equivalent of "M-g" at Gnus startup.
>
> That might be a good compromise.

Agreed.  It seems obvious (said knowing that's dangerous) that on
starting up the program, it's best not to rely on state that could be
wrong.   That's tricky, because mail programs keep messages and don't
refetch them, but relying on counts of unread seems very trouble prone,
vs "I have this message body and the timestamp matches so I won't fetch
it."

I think it would also be good to separate

   check for new

   get all the messages that are there

   really rescan and don't believe state

in terms of being much louder about the differences.  I have the
impression that the assembled gnususers here are at least somewhat hazy
on those boundaries.  At least I am.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Eric Abrahamsen]

Greg Troxel <gdt@lexort.com> writes:

> Arash Esbati <arash@gnu.org> writes:
>
>>> I've focused on figuring out why the active number goes wrong, but in
>>> the interest of pragmatism it might be better just to make sure that
>>> nnimap always does the equivalent of "M-g" at Gnus startup.
>>
>> That might be a good compromise.
>
> Agreed.  It seems obvious (said knowing that's dangerous) that on
> starting up the program, it's best not to rely on state that could be
> wrong.   That's tricky, because mail programs keep messages and don't
> refetch them, but relying on counts of unread seems very trouble prone,
> vs "I have this message body and the timestamp matches so I won't fetch
> it."
>
> I think it would also be good to separate
>
>    check for new
>
>    get all the messages that are there
>
>    really rescan and don't believe state
>
> in terms of being much louder about the differences.  I have the
> impression that the assembled gnususers here are at least somewhat hazy
> on those boundaries.  At least I am.

I think this is an artifact of Gnus starting out as a newsreader, and
adding email capability later. In a NNTP context, the read/unread
message count is entirely private to your local machine, and something
you can manipulate at will. In an IMAP context, it's very much a part of
multi-machine state. Gnus should always defer to the remote server,
particularly at start up. I do wish I knew where the bug was coming
from, though.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Greg Troxel <gdt@lexort.com> writes:
>
>> Arash Esbati <arash@gnu.org> writes:
>>
>>>> I've focused on figuring out why the active number goes wrong, but in
>>>> the interest of pragmatism it might be better just to make sure that
>>>> nnimap always does the equivalent of "M-g" at Gnus startup.
>>>
>>> That might be a good compromise.
>>
>> Agreed.  It seems obvious (said knowing that's dangerous) that on
>> starting up the program, it's best not to rely on state that could be
>> wrong.   That's tricky, because mail programs keep messages and don't
>> refetch them, but relying on counts of unread seems very trouble prone,
>> vs "I have this message body and the timestamp matches so I won't fetch
>> it."
>>
>> I think it would also be good to separate
>>
>>    check for new
>>
>>    get all the messages that are there
>>
>>    really rescan and don't believe state
>>
>> in terms of being much louder about the differences.  I have the
>> impression that the assembled gnususers here are at least somewhat hazy
>> on those boundaries.  At least I am.
>
> I think this is an artifact of Gnus starting out as a newsreader, and
> adding email capability later. In a NNTP context, the read/unread
> message count is entirely private to your local machine, and something
> you can manipulate at will. In an IMAP context, it's very much a part of
> multi-machine state. Gnus should always defer to the remote server,
> particularly at start up. I do wish I knew where the bug was coming
> from, though.
>
>
>

This makes much sense.  I'll try M-g the next time this happens.  Thanks
everyone!

Also I would expect the equivalent of M-g to happen on startup so that
there are less surprises.

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Xiyue Deng <manphiz@gmail.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Greg Troxel <gdt@lexort.com> writes:
>>
>>> Arash Esbati <arash@gnu.org> writes:
>>>
>>>>> I've focused on figuring out why the active number goes wrong, but in
>>>>> the interest of pragmatism it might be better just to make sure that
>>>>> nnimap always does the equivalent of "M-g" at Gnus startup.
>>>>
>>>> That might be a good compromise.
>>>
>>> Agreed.  It seems obvious (said knowing that's dangerous) that on
>>> starting up the program, it's best not to rely on state that could be
>>> wrong.   That's tricky, because mail programs keep messages and don't
>>> refetch them, but relying on counts of unread seems very trouble prone,
>>> vs "I have this message body and the timestamp matches so I won't fetch
>>> it."
>>>
>>> I think it would also be good to separate
>>>
>>>    check for new
>>>
>>>    get all the messages that are there
>>>
>>>    really rescan and don't believe state
>>>
>>> in terms of being much louder about the differences.  I have the
>>> impression that the assembled gnususers here are at least somewhat hazy
>>> on those boundaries.  At least I am.
>>
>> I think this is an artifact of Gnus starting out as a newsreader, and
>> adding email capability later. In a NNTP context, the read/unread
>> message count is entirely private to your local machine, and something
>> you can manipulate at will. In an IMAP context, it's very much a part of
>> multi-machine state. Gnus should always defer to the remote server,
>> particularly at start up. I do wish I knew where the bug was coming
>> from, though.
>>
>>
>>
>
> This makes much sense.  I'll try M-g the next time this happens.  Thanks
> everyone!
>
> Also I would expect the equivalent of M-g to happen on startup so that
> there are less surprises.

So the issue Greg described just happened again that an empty
folder/topic "Trash" from one of my outlook accounts just shown 38
unread messages and selecting it fails with "Can't select group", while
other clients like Thunderbird shows 0 in it.  I have tried "M-g" and it
doesn't help unfortunately :(

Any more ideas for debugging?

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Eric Abrahamsen]

Xiyue Deng <manphiz@gmail.com> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Greg Troxel <gdt@lexort.com> writes:
>>>
>>>> Arash Esbati <arash@gnu.org> writes:
>>>>
>>>>>> I've focused on figuring out why the active number goes wrong, but in
>>>>>> the interest of pragmatism it might be better just to make sure that
>>>>>> nnimap always does the equivalent of "M-g" at Gnus startup.
>>>>>
>>>>> That might be a good compromise.
>>>>
>>>> Agreed.  It seems obvious (said knowing that's dangerous) that on
>>>> starting up the program, it's best not to rely on state that could be
>>>> wrong.   That's tricky, because mail programs keep messages and don't
>>>> refetch them, but relying on counts of unread seems very trouble prone,
>>>> vs "I have this message body and the timestamp matches so I won't fetch
>>>> it."
>>>>
>>>> I think it would also be good to separate
>>>>
>>>>    check for new
>>>>
>>>>    get all the messages that are there
>>>>
>>>>    really rescan and don't believe state
>>>>
>>>> in terms of being much louder about the differences.  I have the
>>>> impression that the assembled gnususers here are at least somewhat hazy
>>>> on those boundaries.  At least I am.
>>>
>>> I think this is an artifact of Gnus starting out as a newsreader, and
>>> adding email capability later. In a NNTP context, the read/unread
>>> message count is entirely private to your local machine, and something
>>> you can manipulate at will. In an IMAP context, it's very much a part of
>>> multi-machine state. Gnus should always defer to the remote server,
>>> particularly at start up. I do wish I knew where the bug was coming
>>> from, though.
>>>
>>>
>>>
>>
>> This makes much sense.  I'll try M-g the next time this happens.  Thanks
>> everyone!
>>
>> Also I would expect the equivalent of M-g to happen on startup so that
>> there are less surprises.
>
> So the issue Greg described just happened again that an empty
> folder/topic "Trash" from one of my outlook accounts just shown 38
> unread messages and selecting it fails with "Can't select group", while
> other clients like Thunderbird shows 0 in it.  I have tried "M-g" and it
> doesn't help unfortunately :(
>
> Any more ideas for debugging?

Maybe move a message into the folder, then move it back out? I'll try to
get to debugging this area of Gnus in the next few weeks...

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>>> Greg Troxel <gdt@lexort.com> writes:
>>>>
>>>>> Arash Esbati <arash@gnu.org> writes:
>>>>>
>>>>>>> I've focused on figuring out why the active number goes wrong, but in
>>>>>>> the interest of pragmatism it might be better just to make sure that
>>>>>>> nnimap always does the equivalent of "M-g" at Gnus startup.
>>>>>>
>>>>>> That might be a good compromise.
>>>>>
>>>>> Agreed.  It seems obvious (said knowing that's dangerous) that on
>>>>> starting up the program, it's best not to rely on state that could be
>>>>> wrong.   That's tricky, because mail programs keep messages and don't
>>>>> refetch them, but relying on counts of unread seems very trouble prone,
>>>>> vs "I have this message body and the timestamp matches so I won't fetch
>>>>> it."
>>>>>
>>>>> I think it would also be good to separate
>>>>>
>>>>>    check for new
>>>>>
>>>>>    get all the messages that are there
>>>>>
>>>>>    really rescan and don't believe state
>>>>>
>>>>> in terms of being much louder about the differences.  I have the
>>>>> impression that the assembled gnususers here are at least somewhat hazy
>>>>> on those boundaries.  At least I am.
>>>>
>>>> I think this is an artifact of Gnus starting out as a newsreader, and
>>>> adding email capability later. In a NNTP context, the read/unread
>>>> message count is entirely private to your local machine, and something
>>>> you can manipulate at will. In an IMAP context, it's very much a part of
>>>> multi-machine state. Gnus should always defer to the remote server,
>>>> particularly at start up. I do wish I knew where the bug was coming
>>>> from, though.
>>>>
>>>>
>>>>
>>>
>>> This makes much sense.  I'll try M-g the next time this happens.  Thanks
>>> everyone!
>>>
>>> Also I would expect the equivalent of M-g to happen on startup so that
>>> there are less surprises.
>>
>> So the issue Greg described just happened again that an empty
>> folder/topic "Trash" from one of my outlook accounts just shown 38
>> unread messages and selecting it fails with "Can't select group", while
>> other clients like Thunderbird shows 0 in it.  I have tried "M-g" and it
>> doesn't help unfortunately :(
>>
>> Any more ideas for debugging?
>
> Maybe move a message into the folder, then move it back out?

Ah I was aware of this workaround.  Just wonder whether there is
metadata/debug info that I can try to get to help the developer such as
you to debug this?

> I'll try to get to debugging this area of Gnus in the next few
> weeks...

Really appreciate this!  Meanwhile probably I should try to submit a bug
for tracking.  Is there a Gnus specific bug tracker or I should just use
report-emacs-bug?

>
>
>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On May  2, 2024, Xiyue Deng <manphiz@gmail.com> wrote:

> Ah I was aware of this workaround.  Just wonder whether there is
> metadata/debug info that I can try to get to help the developer such as
> you to debug this?

You could hit `G p' on the group in the *Group* buffer and show us
what is displayed.  (You can hit `C-c C-k' in that buffer to kill it.)

I sometimes get a situation where the active range starts at 0 instead
of 1, and it's very hard to fix.  I fiddle around with changing things
in that buffer, closing emacs, etc, and can fix it, but I'm not sure
exactly what the recipe is.

Dan

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Hi Dan,

Dan Christensen <jdc@uwo.ca> writes:

> On May  2, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>
>> Ah I was aware of this workaround.  Just wonder whether there is
>> metadata/debug info that I can try to get to help the developer such as
>> you to debug this?
>
> You could hit `G p' on the group in the *Group* buffer and show us
> what is displayed.  (You can hit `C-c C-k' in that buffer to kill it.)
>

Thanks for the tip!  However I don't know what happened the count of
problematic topic was back to 0 since this morning, so I'll probably
have to wait for the next occurrence to try this.

> I sometimes get a situation where the active range starts at 0 instead
> of 1, and it's very hard to fix.  I fiddle around with changing things
> in that buffer, closing emacs, etc, and can fix it, but I'm not sure
> exactly what the recipe is.
>

Noted.  Will keep an eye on this next time.

> Dan
>
>
>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Xiyue Deng <manphiz@gmail.com> writes:

> Hi Dan,
>
> Dan Christensen <jdc@uwo.ca> writes:
>
>> On May  2, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>>
>>> Ah I was aware of this workaround.  Just wonder whether there is
>>> metadata/debug info that I can try to get to help the developer such as
>>> you to debug this?
>>
>> You could hit `G p' on the group in the *Group* buffer and show us
>> what is displayed.  (You can hit `C-c C-k' in that buffer to kill it.)
>>
>
> Thanks for the tip!  However I don't know what happened the count of
> problematic topic was back to 0 since this morning, so I'll probably
> have to wait for the next occurrence to try this.
>

So this happened again in one of my outlook accounts' Sent folder, and
the "G p" information is as below:

,----
| ((modseq)
|  (uidvalidity . "11")
|  (active 1 . 7)
|  (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent))
`----

>> I sometimes get a situation where the active range starts at 0 instead
>> of 1, and it's very hard to fix.  I fiddle around with changing things
>> in that buffer, closing emacs, etc, and can fix it, but I'm not sure
>> exactly what the recipe is.
>>
>
> Noted.  Will keep an eye on this next time.
>

This time the active range does start from 1.  So it seems that it does
think there are 7 active items?

>> Dan
>>
>>
>>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On May  5, 2024, Xiyue Deng <manphiz@gmail.com> wrote:

> So this happened again in one of my outlook accounts' Sent folder, and
> the "G p" information is as below:
>
> ,----
> | ((modseq)
> |  (uidvalidity . "11")
> |  (active 1 . 7)
> |  (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent))
> `----

I guess you are having a different problem than I am.  I assume that
you tried `M-g', and the above doesn't change?  What if you go into
the server buffer with `^' and then hit `C' on the server to close it
and `O' on the server to open it?

Dan

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Dan Christensen <jdc@uwo.ca> writes:

> On May  5, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>
>> So this happened again in one of my outlook accounts' Sent folder, and
>> the "G p" information is as below:
>>
>> ,----
>> | ((modseq)
>> |  (uidvalidity . "11")
>> |  (active 1 . 7)
>> |  (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent))
>> `----
>
> I guess you are having a different problem than I am.  I assume that
> you tried `M-g', and the above doesn't change?

Yes, and it didn't fix it.

> What if you go into the server buffer with `^' and then hit `C' on the
> server to close it and `O' on the server to open it?
>

Tried this as well, and unfortunately it didn't fix it either.

> Dan
>
>
>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Xiyue Deng <manphiz@gmail.com> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> On May  5, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>>
>>> So this happened again in one of my outlook accounts' Sent folder, and
>>> the "G p" information is as below:
>>>
>>> ,----
>>> | ((modseq)
>>> |  (uidvalidity . "11")
>>> |  (active 1 . 7)
>>> |  (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent))
>>> `----
>>
>> I guess you are having a different problem than I am.  I assume that
>> you tried `M-g', and the above doesn't change?
>
> Yes, and it didn't fix it.
>
>> What if you go into the server buffer with `^' and then hit `C' on the
>> server to close it and `O' on the server to open it?
>>
>
> Tried this as well, and unfortunately it didn't fix it either.
>

Just wonder is there anything else that I can try to debug this?  In
worst case I can just use the moving-messages-in-and-out workaround to
reset the count, but wondering whether there is any other things to try.

>> Dan
>>
>>
>>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Xiyue Deng <manphiz@gmail.com> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> On May  5, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>>
>>> So this happened again in one of my outlook accounts' Sent folder, and
>>> the "G p" information is as below:
>>>
>>> ,----
>>> | ((modseq)
>>> |  (uidvalidity . "11")
>>> |  (active 1 . 7)
>>> |  (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent))
>>> `----
>>
>> I guess you are having a different problem than I am.  I assume that
>> you tried `M-g', and the above doesn't change?
>
> Yes, and it didn't fix it.
>
>> What if you go into the server buffer with `^' and then hit `C' on the
>> server to close it and `O' on the server to open it?
>>
>
> Tried this as well, and unfortunately it didn't fix it either.
>

Just wonder is there anything else that I can try to debug this?  In
worst case I can just use the moving-messages-in-and-out workaround to
reset the count, but wondering whether there is any other things to try
and hopefully help identifying the issue from the Gnus side.

>> Dan
>>
>>
>>

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Andrew Cohen]

>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes:

    XD> Xiyue Deng <manphiz@gmail.com> writes:
    >> Dan Christensen <jdc@uwo.ca> writes:
    >> 
    >>> On May  5, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
    >>> 
    >>>> So this happened again in one of my outlook accounts' Sent
    >>>> folder, and the "G p" information is as below:
    >>>> 
    >>>> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 7) |
    >>>> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft
    >>>> $MDNSent)) `----
    >>> 

When the count is restored to normal, does "G p" show modseq still
without a value?

-- 
Andrew Cohen

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Andrew Cohen <acohen@ust.hk> writes:

>>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes:
>
>     XD> Xiyue Deng <manphiz@gmail.com> writes:
>     >> Dan Christensen <jdc@uwo.ca> writes:
>     >> 
>     >>> On May  5, 2024, Xiyue Deng <manphiz@gmail.com> wrote:
>     >>> 
>     >>>> So this happened again in one of my outlook accounts' Sent
>     >>>> folder, and the "G p" information is as below:
>     >>>> 
>     >>>> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 7) |
>     >>>> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft
>     >>>> $MDNSent)) `----
>     >>> 
>
> When the count is restored to normal, does "G p" show modseq still
> without a value?

Yes.  This is "G p" after I used the move-in-move-out trick to reset the
count:

,----
| ((modseq)
|  (uidvalidity . "11")
|  (active 1 . 12)
|  (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent))
`----


-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Andrew Cohen]

>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes:

    >> When the count is restored to normal, does "G p" show modseq
    >> still without a value?

    XD> Yes.  This is "G p" after I used the move-in-move-out trick to
    XD> reset the count:

    XD> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 12) |
    XD> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft
    XD> $MDNSent)) `----

Thanks. It seems that two different cases have been reported: those
which are fixed with "M-g" and those (like this case of yours) that are
not.  I was hoping that the absence of a value for modseq might be a
clue as to the difference, but apparently not :(

The absence of this value suggests that the server doesn't support
QRESYNC. Out of curiosity, what server is this? I think most of the
common ones support it.

-- 
Andrew Cohen

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Andrew Cohen <acohen@ust.hk> writes:

>>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes:
>
>     >> When the count is restored to normal, does "G p" show modseq
>     >> still without a value?
>
>     XD> Yes.  This is "G p" after I used the move-in-move-out trick to
>     XD> reset the count:
>
>     XD> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 12) |
>     XD> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft
>     XD> $MDNSent)) `----
>
> Thanks. It seems that two different cases have been reported: those
> which are fixed with "M-g" and those (like this case of yours) that are
> not.  I was hoping that the absence of a value for modseq might be a
> clue as to the difference, but apparently not :(
>
> The absence of this value suggests that the server doesn't support
> QRESYNC. Out of curiosity, what server is this? I think most of the
> common ones support it.

This is an outlook account.  AAMOF this happened to both of my gmail and
outlook accounts, where empty topic showing unread count like this case
happens in outlook accounts more often.

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Andreas Schwab]

On Mai 10 2024, Andrew Cohen wrote:

>>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes:
>
>     >> When the count is restored to normal, does "G p" show modseq
>     >> still without a value?
>
>     XD> Yes.  This is "G p" after I used the move-in-move-out trick to
>     XD> reset the count:
>
>     XD> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 12) |
>     XD> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft
>     XD> $MDNSent)) `----
>
> Thanks. It seems that two different cases have been reported: those
> which are fixed with "M-g" and those (like this case of yours) that are
> not.  I was hoping that the absence of a value for modseq might be a
> clue as to the difference, but apparently not :(

The modseq value comes from the HIGHESTMODSEQ response (see RFC 4551,
3.1.1).  If the HIGHESTMODSEQ response is missing, then the server does
not support the persistent storage of mod-sequences.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Greg Troxel]

Xiyue Deng <manphiz@gmail.com> writes:

> This makes much sense.  I'll try M-g the next time this happens.  Thanks
> everyone!

I had a situation where a message arrived in INBOX.  I could see it in
the filesystem dovecot reads, and in other clients.  It was as if it did
not exist in gnus.

M-g on the group in *Group* buffer seems to have resolved that.
It is interesting that moving out and back resolved what I think is the
same situation in the past.

Just making up stuff without basis:  It's like a missing wakeup in a
synchronization protocol.


> Also I would expect the equivalent of M-g to happen on startup so that
> there are less surprises.

Agreed, but I can see the point of not blowing caches just because of startup.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Hi Greg,

Greg Troxel <gdt@lexort.com> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> This makes much sense.  I'll try M-g the next time this happens.  Thanks
>> everyone!
>
> I had a situation where a message arrived in INBOX.  I could see it in
> the filesystem dovecot reads, and in other clients.  It was as if it did
> not exist in gnus.
>
> M-g on the group in *Group* buffer seems to have resolved that.
> It is interesting that moving out and back resolved what I think is the
> same situation in the past.
>
> Just making up stuff without basis:  It's like a missing wakeup in a
> synchronization protocol.
>
>

Unfortunately, it looks like it doesn't fix it for me, presumably
because I'm using nnimap instead of reading from a local mailbox.
Therefore it feels natural to assume a bug in nnimap, though it's still
years earlier before I can figure it out from the code, if that ever
happens :P

>> Also I would expect the equivalent of M-g to happen on startup so that
>> there are less surprises.
>
> Agreed, but I can see the point of not blowing caches just because of startup.

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Greg Troxel]

Xiyue Deng <manphiz@gmail.com> writes:

> Hi Greg,
>
> Greg Troxel <gdt@lexort.com> writes:
>
>> Xiyue Deng <manphiz@gmail.com> writes:
>>
>>> This makes much sense.  I'll try M-g the next time this happens.  Thanks
>>> everyone!
>>
>> I had a situation where a message arrived in INBOX.  I could see it in
>> the filesystem dovecot reads, and in other clients.  It was as if it did
>> not exist in gnus.
>>
>> M-g on the group in *Group* buffer seems to have resolved that.
>> It is interesting that moving out and back resolved what I think is the
>> same situation in the past.
>>
>> Just making up stuff without basis:  It's like a missing wakeup in a
>> synchronization protocol.
>
> Unfortunately, it looks like it doesn't fix it for me, presumably
> because I'm using nnimap instead of reading from a local mailbox.
> Therefore it feels natural to assume a bug in nnimap, though it's still
> years earlier before I can figure it out from the code, if that ever
> happens :P

This was nnimap in my case, with emacs and dovecot on the same machine
if that matters.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Xiyue Deng]

Greg Troxel <gdt@lexort.com> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>> Hi Greg,
>>
>> Greg Troxel <gdt@lexort.com> writes:
>>
>>> Xiyue Deng <manphiz@gmail.com> writes:
>>>
>>>> This makes much sense.  I'll try M-g the next time this happens.  Thanks
>>>> everyone!
>>>
>>> I had a situation where a message arrived in INBOX.  I could see it in
>>> the filesystem dovecot reads, and in other clients.  It was as if it did
>>> not exist in gnus.
>>>
>>> M-g on the group in *Group* buffer seems to have resolved that.
>>> It is interesting that moving out and back resolved what I think is the
>>> same situation in the past.
>>>
>>> Just making up stuff without basis:  It's like a missing wakeup in a
>>> synchronization protocol.
>>
>> Unfortunately, it looks like it doesn't fix it for me, presumably
>> because I'm using nnimap instead of reading from a local mailbox.
>> Therefore it feels natural to assume a bug in nnimap, though it's still
>> years earlier before I can figure it out from the code, if that ever
>> happens :P
>
> This was nnimap in my case, with emacs and dovecot on the same machine
> if that matters.
>
>

Ah I see.  Then probably the IMAP implementation of Outlook is just bad
;-)

-- 
Xiyue Deng

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Greg Troxel]

Xiyue Deng <manphiz@gmail.com> writes:

>> This was nnimap in my case, with emacs and dovecot on the same machine
>> if that matters.
>
> Ah I see.  Then probably the IMAP implementation of Outlook is just bad
> ;-)

I believe there is a gnus bug that I am hitting, and that others are
too.

I am entirely willing to believe the the theory that there is also a bug
in Outlook IMAP....

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Eric Abrahamsen]

Greg Troxel <gdt@lexort.com> writes:

> Xiyue Deng <manphiz@gmail.com> writes:
>
>>> This was nnimap in my case, with emacs and dovecot on the same machine
>>> if that matters.
>>
>> Ah I see.  Then probably the IMAP implementation of Outlook is just bad
>> ;-)
>
> I believe there is a gnus bug that I am hitting, and that others are
> too.
>
> I am entirely willing to believe the the theory that there is also a bug
> in Outlook IMAP....

I was going to say, the IMAP implementation of Outlook is indeed bad!
But that's "good" that you're hitting this issue with Dovecot running
locally, as that's a setup with fewer confounding variables, and also
what I run, so maybe we stand a better chance of reproducing.

Maybe it's time to dive in :( Greg, if you're willing, would you provide
as much details as possible about your setup, and anything/everything
you can remember about what happened/happens before you see this error?

All your nnimap settings are of interest, obviously, but you might as
well send anything that could possibly be relevant, off-list if you'd
prefer.

And if you do see the error crop up again, please let us know what you
were doing immediately before, and if at all possible, leave Gnus in the
broken state until we can do some remote debugging.

Thanks!
Eric

Gnus sometimes fails to see a message that is actually in IMAP

[Greg Troxel]

I have changed the subject, as I am not at all sure that my problem is
the same as "reports new messages but not showing them on IMAP server".
In my case, it seems to me as if there is a filter between gnus and the
IMAP server that redacts a message.


Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I was going to say, the IMAP implementation of Outlook is indeed bad!
> But that's "good" that you're hitting this issue with Dovecot running
> locally, as that's a setup with fewer confounding variables, and also
> what I run, so maybe we stand a better chance of reproducing.

Yes, it does seem more tractable.

One thing I find troublesome about modern gnus is that it is part of
emacs.  I'm currently running 28.2, so that's the gnus I have.  Years
before, when gnus was in a separate repo, I was running up-to-date gnus
with stable emacs.   If I'm confused and we should be debugging gnus
trunk please let me know.

I probably can update to 29.x without trouble, and I should do that.  I
try to stay away from bleeding edge including .0, when the older version
isn't crufty and isn't troublesome, for things I want to use rather than
hack on.  I don't really want to run emacs-current.

> Maybe it's time to dive in :( Greg, if you're willing, would you provide
> as much details as possible about your setup, and anything/everything
> you can remember about what happened/happens before you see this error?

First, I have no perception of anything unusual or remarkable happening
when the problem happens.  Mail arrives, apparently just like any other
message, I see it in K-9 (Android client that seems very well behaved),
and I then go to look for it in gnus (to answer, because I have a
keyboard) and find it's not there.  This is infrequent, maybe once a
month, and there is nothing strange about these messages.  They are from
people I've heard from before, are not necessarily even large, and "more
$filename" in the IMAP dir looks unremarkable.

When this happens, gnus feels entirely normal.  I can still enter INBOX,
read arriving mail, enter with "100 <space>" to see already read mail,
etc.  If I were only using gnus I would have had no idea anything was
wrong, until someone asked me why I didn't respond to some message.

The missing messages are in $HOME/IMAP/cur because they've been read by
another client.  Thunderbird sees the missing messages too.

I have previously tried restarting dovecot and removing dovecot caches,
but this has never fixed the issue.  I am 95% convinced that the is
issue is in gnus.

My setup is:

  - computer runs NetBSD 9, postfix receiving mail from the Internet at
    large, with lots of spam filtering, postfix delivering to $HOME/IMAP
    as maildir, hence into $HOME/IMAP/new.  While I have to mess with
    spam filtering from time to time, it is not flaky - just minor
    misclassification - and the messages in question are 1) not spam and
    2) delivered to INBOX.

  - dovecot 2.3.21 serves $HOME/IMAP.  Many updates have happened since
    I first noticed this -- it's been several years I think (but no
    notes).

  - dovecot has an x509 cert (that validates under pkix rules), and
    listens only on the "implicit TLS" port, meaning that it expects TLS
    negotation right away vs STARTTLS.  (I'm firmly in the "cleartext
    connections should be adminitratively prohibited" camp.)

  - dovecot's uidvalidity as shown in "G p" is a number that when
    interpreted as a timeval is noonish EDT on a Sunday in March of
    2006.  Indeed the IMAP spec says that servers should try not to
    change that.  That's wild, but probably when I installed dovecot
    version 1 on a different computer.  I have upgraded dovecot,
    upgraded the OS version, and migrated to different hardware with a
    different hostname since.  The point is that this problem is almost
    certainly not about gnus' IMAP implemention getting confused about
    uidvalidity changes!

  - emacs with gnus is left running always, and accessed via tmux.  The
    machine is very stable, basically getting rebooted for software
    updates.

  - emacs 28.2 gnus has nnimap configured

    (setq gnus-secondary-select-methods
          '((nnimap "foo.example.com"
                    (nnimap-address "bar.example.com")
                    (nnimap-port 993)
                    (nnimap-stream ssl)
                    (nnimap-authenticator login)
                    (nnir-search-engine imap)
                    )
            (nntp "news.gwene.org")
            )
          )
   where foo is a historical name and bar.example.com is the actual name
   of the host (with emacs and the imap server both).

   The connection goes over IPv6, not ::1, but the actual IP address.
   I would have used localhost, but then the certificate is not valid.

   (Yes, I know the gwene/gmane world is messy and that it's possible I
   should be stopping this or doing it differently, but I don't think it
   is relevant.)

  - I had vestigial config that I just dropped, to add --insecure to
    gnutls-cli, for a historical collection of self-signed certs,
    connecting to localhost, emacs having the wrong path to ssl config,
    and me not straightening out my
    copy-of-config-in-linuxy-pathname-for-emacs.  In the end, it was
    just adding "--insecure" to the gnutls-cli invocation.  I can't
    believe that matters, but mentioning it for completeness.

> And if you do see the error crop up again, please let us know what you
> were doing immediately before, and if at all possible, leave Gnus in the
> broken state until we can do some remote debugging.

Thanks for offering to spend time on this.  I'll write offlist to see
about coordinating.

It does seem like looking at group properties is useful and I'll do that.

I'll at least take better notes.  I think but am not 100% sure that the
broken state survives exiting/restarting gnus, and exiting/restarting
emacs.

My previous workaround was move-out move-back.
Now, M-g on the group seems to fix things.

It would help to understand the big picture of how gnus caches imap
state.  Ideally that would be in the manual; I'll take another look at
that.

It would also be interesting to have a gnus-imap-revalidate function that
for all cached information, fetches it and compares, complaining loudly
if different.  I have used this technique when debugging routing code to
end up with only correct cached information, and didn't think of any
other way to find rare bugs.

Greg

Re: Gnus sometimes fails to see a message that is actually in IMAP

[Eric Abrahamsen]

Greg Troxel <gdt@lexort.com> writes:

Wow, this is a ton of information! Thanks for all the background. Kudos
on using TLS on your own machine, something I have never had the
patience to set up.

> I have changed the subject, as I am not at all sure that my problem is
> the same as "reports new messages but not showing them on IMAP server".
> In my case, it seems to me as if there is a filter between gnus and the
> IMAP server that redacts a message.
>
>
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I was going to say, the IMAP implementation of Outlook is indeed bad!
>> But that's "good" that you're hitting this issue with Dovecot running
>> locally, as that's a setup with fewer confounding variables, and also
>> what I run, so maybe we stand a better chance of reproducing.
>
> Yes, it does seem more tractable.
>
> One thing I find troublesome about modern gnus is that it is part of
> emacs.  I'm currently running 28.2, so that's the gnus I have.  Years
> before, when gnus was in a separate repo, I was running up-to-date gnus
> with stable emacs.   If I'm confused and we should be debugging gnus
> trunk please let me know.

Not confused, of course, but I do think that Emacs 28 is a little bit
too old for this sort of debugging. Ideally we'd be doing it on master,
though 29 is still worthwhile, if you can update to that. There are good
things in the past couple of versions.


[...]

> It would also be interesting to have a gnus-imap-revalidate function that
> for all cached information, fetches it and compares, complaining loudly
> if different.  I have used this technique when debugging routing code to
> end up with only correct cached information, and didn't think of any
> other way to find rare bugs.

This is a really good idea. There's been talk on this list of forcibly
resetting Gnus to the remote server's state at start-up, which would be
something of a band-aid, but this would stand us a better chance of
pinning down the bug. People could run it at Gnus startup and shutdown,
as well as on-demand. I'll look into putting this together.

Re: Gnus sometimes fails to see a message that is actually in IMAP

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Greg Troxel <gdt@lexort.com> writes:
>
> Wow, this is a ton of information! Thanks for all the background. Kudos
> on using TLS on your own machine, something I have never had the
> patience to set up.
>
>> I have changed the subject, as I am not at all sure that my problem is
>> the same as "reports new messages but not showing them on IMAP server".
>> In my case, it seems to me as if there is a filter between gnus and the
>> IMAP server that redacts a message.

Brief update -- when I started looking further into this, I realized
that I have the same issue with the INBOX of one of my main IMAP
accounts. I'd just not noticed because I have 486 unread messages in
that folder :( Well, 483 according to Gnus.

This account includes a remote Dovecot on the server that actually
receives mail, and two local machines that each have their own local
Dovecot, and Emacs/Gnus. The remote and local Dovecots are synced with
mbsync, and both remote and local report 486 unread messages when
checked with doveadm.

My other local machine is currently not booting, but I'll be very
curious to see if Gnus on that machine is wrong in the same way
(a deterministic bug) or not (space gremlins).

Otherwise, the difference between "g" and "M-g", for IMAP servers that
support qresync, is that "g" goes the qresync route:

EXAMINE <group> (QRESYNC (<uidvalidity> <modseq>))

while "M-g" does a full fetch of flags:

"UID FETCH 1:* FLAGS"

So that's a pretty strong indicator that there's a bug in the qresync
branch of `nnimap-update-qresync-info'. Some have reported bugs that
*weren't* resolved by "M-g", so there will likely be a few things to
fix, but one thing at a time.

I'd originally planned to provide a function people could run after
updating, to see if the numbers had gone out of whack, but now I think I
can do a bit better; maybe a tweaked version of
`nnimap-update-qresync-info' that would do a bit of its own
error-checking.

Eric

Re: Gnus sometimes fails to see a message that is actually in IMAP

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Greg Troxel <gdt@lexort.com> writes:
>>
>> Wow, this is a ton of information! Thanks for all the background. Kudos
>> on using TLS on your own machine, something I have never had the
>> patience to set up.
>>
>>> I have changed the subject, as I am not at all sure that my problem is
>>> the same as "reports new messages but not showing them on IMAP server".
>>> In my case, it seems to me as if there is a filter between gnus and the
>>> IMAP server that redacts a message.
>
> Brief update -- when I started looking further into this, I realized
> that I have the same issue with the INBOX of one of my main IMAP
> accounts. I'd just not noticed because I have 486 unread messages in
> that folder :( Well, 483 according to Gnus.
>
> This account includes a remote Dovecot on the server that actually
> receives mail, and two local machines that each have their own local
> Dovecot, and Emacs/Gnus. The remote and local Dovecots are synced with
> mbsync, and both remote and local report 486 unread messages when
> checked with doveadm.
>
> My other local machine is currently not booting, but I'll be very
> curious to see if Gnus on that machine is wrong in the same way
> (a deterministic bug) or not (space gremlins).
>
> Otherwise, the difference between "g" and "M-g", for IMAP servers that
> support qresync, is that "g" goes the qresync route:
>
> EXAMINE <group> (QRESYNC (<uidvalidity> <modseq>))
>
> while "M-g" does a full fetch of flags:
>
> "UID FETCH 1:* FLAGS"
>
> So that's a pretty strong indicator that there's a bug in the qresync
> branch of `nnimap-update-qresync-info'. Some have reported bugs that
> *weren't* resolved by "M-g", so there will likely be a few things to
> fix, but one thing at a time.

I'm currently looking with deep suspicion at the very beginning of that
function:

  ;; Add all the vanished articles to the list of read articles.
  (setf (gnus-info-read info)
        (range-add-list
         (range-add-list
          (range-concat (gnus-info-read info) vanished)
	  (cdr (assq '%Flagged flags)))
	 (cdr (assq '%Seen flags))))

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[James Thomas]

Eric Abrahamsen wrote:

> I've focused on figuring out why the active number goes wrong

On that note, here's something I'd been meaning to debug myself
sometime:

- (Preferably with an empty drafts folder) Compose a message and save it
- Open the drafts folder, press e on the message and then kill the new
  buffer; then (incidentally, if you do '/ N' then this bug does not
  arise) delete the message (B DEL)
- Press q
- The message count is wrong

I don't know if this is related to the above. Btw could you share it if
there's a bug report for the former?

--

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[Eric Abrahamsen]

James Thomas <jimjoe@gmx.net> writes:

> Eric Abrahamsen wrote:
>
>> I've focused on figuring out why the active number goes wrong
>
> On that note, here's something I'd been meaning to debug myself
> sometime:
>
> - (Preferably with an empty drafts folder) Compose a message and save it
> - Open the drafts folder, press e on the message and then kill the new
>   buffer; then (incidentally, if you do '/ N' then this bug does not
>   arise) delete the message (B DEL)
> - Press q
> - The message count is wrong

That's a new one for me, but I was able to reproduce it. Would you open
a bug report with that recipe?

> I don't know if this is related to the above. Btw could you share it if
> there's a bug report for the former?

I don't think there is, mostly because it's kind of a vague problem.
I wouldn't be surprised if it ended up being several things at once.

Re: Gnus sometimes reports new messages but not showing them on IMAP server

[James Thomas]

Hi,

Xiyue Deng wrote:

> Hi,
>
> i'm having some issues that Gnus sometimes reports that there are new
> messages on an IMAP server but it doesn't show them.  In my case it's
> using Gmail IMAP, and if I check using other mail clients
> (e.g. Thunderbird) or directly check on Gmail website I can seem them.
> I'm not sure whether this issue is specific to IMAP either.  As I'm not
> sure how to get more info, I didn't use report bug yet, but would like
> to ask around and see whether this is a known issue, or otherwise I can
> get some suggestions to get more debug info before I formally file a
> bug.
>
> Any suggestions are welcome!  TIA!

James Thomas wrote:

> I haven't tested this patch (don't really know how to), but this could
> very well be the cause of this long-standing bug:

Could you check if this patch (the 1st one, which swaps the 0 and 1),
makes any difference?

Regards,
James

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[James Thomas]

(Forgot to cc this here:)

James Thomas wrote:

> Dan Christensen wrote:
>
>> On Sep 11, 2024, James Thomas via "Bug reports for GNU Emacs, the
>> Swiss army knife of text editors" <bug-gnu-emacs@gnu.org> wrote:
>>
>>> James Thomas wrote:
>>>
>>>> I haven't tested this patch (don't really know how to), but this could
>>>> very well be the cause of this long-standing bug:
>>>
>>> Could you check if this patch (the 1st one, which swaps the 0 and 1),
>>> makes any difference?
>>
>> I've noticed a long-standing bug, and tried this patch.  Here's the bug:
>> If I use `B m' or `B c' to move or copy a message from an nnimap group
>> to a non-existent nnimap group, Gnus asks if I want to create the group.
>> I say yes, and then it gets created with some faulty parameters that
>> make the *Group* buffer show odd numbers:
>>
>>   ("nnimap+rocky:testmove1" 3
>>    ((1 . 1))
>>    ((unexist 0) (seen 1)) "nnimap:rocky"
>>    ((modseq . "3") (uidvalidity . "1549221829") (active 0 . 1) (permanent-flags %Answered %Flagged %Deleted %Seen %Draft %*)))))
>>
>> Note that article 0 is mentioned twice there.  If I close Gnus, get
>> rid of the "(unexist 0)", and change the active range to 1 . 1, then
>> things are good again.
>>
>> I can consistently reproduce this.
>>
>> I wondered if the patch in this thread would help, but after applying
>> it, I'm unable to create the new group in this way.  After hitting `y'
>> to agree to creating a new group, I get:
>>
>> Debugger entered--Lisp error: (error "Couldn’t create group nnimap+rocky:testmove4")
>>   signal(error ("Couldn’t create group nnimap+rocky:testmove4"))
>>   error("Couldn't create group %s" "nnimap+rocky:testmove4")
>>   gnus-read-move-group-name("Copy" "nnimap+rocky:testmove1" (50086) "nn:")
>>   gnus-summary-move-article(nil nil nil copy)
>>   gnus-summary-copy-article(nil)
>>   funcall-interactively(gnus-summary-copy-article nil)
>>   call-interactively(gnus-summary-copy-article nil nil)
>>   command-execute(gnus-summary-copy-article)
>>
>> So this patch makes Gnus worse for me, rather than better.  I hope this
>> reproducible strategy helps pin down a solution.
>
> Sorry! I got confused again... Please try the other patch in that bug
> report (the one that swaps the car and cdr). I find that it works with
> your use case!
>
> --

--

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

On Sep 12, 2024, James Thomas <jimjoe@gmx.net> wrote:

>> Sorry! I got confused again... Please try the other patch in that bug
>> report (the one that swaps the car and cdr). I find that it works with
>> your use case!

Yes, that patch fixes the problem I've had for ages.  But I don't think
that it's correct.  Here's what is done in nnml.el:

	  (nnheader-insert "211 %d %d %d %s\n"
			   (max (1+ (- (cdr active) (car active))) 0)
			   (car active) (cdr active) group)))))))

Notice that the car comes before the cdr, and that you have to add 1
to the difference between the cdr and the car, since if the active
range is 2 . 3, then there are 2 active articles, which is one more
than 3 - 2.  So I suspect that the patch attached below might be
correct.  Someone who knows more about what the "211" line should
contain should review this.

My patch also fixes the problem I've had.  With either patch, after
moving or copying an article to a new group, the new group has (active 1
. 1) as expected, with no unexist entry.

But the two patches would behave differently when active was already set.

Dan


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnimap.patch --]
[-- Type: text/x-diff, Size: 577 bytes --]

diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
index 17a55f98..cdd9f01f 100644
--- a/lisp/gnus/nnimap.el
+++ b/lisp/gnus/nnimap.el
@@ -918,10 +918,10 @@ nnimap-request-group
 	    (nnimap-finish-retrieve-group-infos server info sequences
 						t)
 	    (setq active (nth 2 (assoc group nnimap-current-infos)))))
-	(setq active (or active '(0 . 1)))
+	(setq active (or active '(1 . 0)))
 	(erase-buffer)
 	(insert (format "211 %d %d %d %S\n"
-			(- (cdr active) (car active))
+			(max (1+ (- (cdr active) (car active))) 0)
 			(car active)
 			(cdr active)
 			group))

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

An alternative to

  (max (1+ (- (cdr active) (car active))) 0)

would be (range-length active), I believe.  (Untested.)  That function
assume that the range is well-formed, so doesn't use max to avoid
negative numbers, but that's probably fine.  (range-length '(1 . 0))
does correctly return 0.

Dan

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Adam Sjøgren]

Dan writes:

> Someone who knows more about what the "211" line should contain
> should review this.

rfc 3977 (NNTP) says:

  "Responses
     211 number low high group     Group successfully selected
      [...]

   Parameters
     group     Name of newsgroup
     number    Estimated number of articles in the group
     low       Reported low water mark
     high      Reported high water mark"

 · https://datatracker.ietf.org/doc/rfc3977/


  Best regards,

    Adam

-- 
 "Are you William Blake?"                                   Adam Sjøgren
 "Yes, do you know my poetry?"                         asjo@koldfront.dk

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Sep 15, 2024, James Thomas <jimjoe@gmx.net> wrote:

> Dan Christensen wrote:
>
>>   (max (1+ (- (cdr active) (car active))) 0)
>
> Just wanna chime in with what's in (info "(gnus) Required Back End
> Functions"):
>
>   Gnus just discards the total-number, so whether one should take the
>   bother to generate it properly (if that is a problem) is left as an
>   exercise to the reader.

That's interesting.  I wonder if it is still correct.

As I mentioned in an earlier message in this bug report, without that
line of the patch, Gnus gives me an error.  Without that line, Gnus
would calculate the total number of articles as -1, and this line
from gnus-start.el would fail to match because of the minus sign:

    (when (looking-at "[0-9]+ [0-9]+ \\([0-9]+\\) [0-9]+")

I'm not sure if this is the only place that would be affected.  Also,
I see that many other parts of Gnus do a correct calculation of the
total number of articles, so I guess we should follow suit.

Dan

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[James Thomas]

Dan Christensen wrote:

> On Sep 15, 2024, James Thomas <jimjoe@gmx.net> wrote:
>
>> Dan Christensen wrote:
>>
>>>   (max (1+ (- (cdr active) (car active))) 0)
>>
>> Just wanna chime in with what's in (info "(gnus) Required Back End
>> Functions"):
>>
>>   Gnus just discards the total-number, so whether one should take the
>>   bother to generate it properly (if that is a problem) is left as an
>>   exercise to the reader.
>
> That's interesting.  I wonder if it is still correct.
>
> As I mentioned in an earlier message in this bug report, without that
> line of the patch, Gnus gives me an error.  Without that line, Gnus
> would calculate the total number of articles as -1, and this line
> from gnus-start.el would fail to match because of the minus sign:
>
>     (when (looking-at "[0-9]+ [0-9]+ \\([0-9]+\\) [0-9]+")

Of course, a non-numeric character would still be wanton.

> I see that many other parts of Gnus do a correct calculation of the
> total number of articles

Wrt this function? Maybe they didn't realize this. :-)

Regards,
James

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Sep 16, 2024, James Thomas <jimjoe@gmx.net> wrote:

> Dan Christensen wrote:
>
>> I see that many other parts of Gnus do a correct calculation of the
>> total number of articles
>
> Wrt this function?

Yes, for example, the code from nnml-request-group that I posted
earlier in the thread, and other places that have "211".

In any case, the correct calculation is simple, so I think it's
good to do it correctly.

Maintainers, should we go ahead and merge this?

Dan

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[James Thomas]

Dan Christensen wrote:

> On Sep 16, 2024, James Thomas <jimjoe@gmx.net> wrote:
>
>> Dan Christensen wrote:
>>
>>> I see that many other parts of Gnus do a correct calculation of the
>>> total number of articles
>>
>> Wrt this function?
>
> Yes, for example, the code from nnml-request-group that I posted
> earlier in the thread, and other places that have "211".

No, I mean something that _uses_ that value? The only function I can
find is 'gnus-browse-foreign-server' which doesn't.

Anyway not a big deal. I guess.

Regards,
James

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

[-- Attachment #1: Type: text/plain, Size: 2338 bytes --]

On Sep 16, 2024, James Thomas <jimjoe@gmx.net> wrote:

> Wait! I think we've miscommunicated: I'd meant the _other_ patch, the
> one in (gnus-summary-goto-article "<86zfoqpobd.fsf@gmx.net>"). I use
> that and it's working fine here.

I believe you are talking about the patch to nnimap-request-group that
swaps the car and the cdr:

 	(insert (format "211 %d %d %d %S\n"
 			(- (cdr active) (car active))
-			(car active)
 			(cdr active)
+			(car active)
 			group))

After that patch, the relevant part of nnimap-request-group would look
like:

	(setq active (or active '(0 . 1)))
	(erase-buffer)
	(insert (format "211 %d %d %d %S\n"
			(- (cdr active) (car active))
			(cdr active)
			(car active)
			group))

This code has two bugs that (mostly) cancel each other out.  First, when
the group is empty, it represents that with the active range (0 . 1).
But that active range represents a group with two articles, numbered 0
and 1.  If you look throughout Gnus, the empty group is always
represented by the range (1 . 0).

Second, this code has the car and the cdr in the wrong order.
Everywhere else in Gnus that creates a 211 line from an active
range puts the car before the cdr.  In the case of an empty group,
these two bugs cancel.  But if active already had a non-nil value,
then the car and the cdr would be wrong.  And even if the active
range is always nil at this point, I don't approve of code that
has two bugs that cancel.

After my proposed patch (attached again), the code looks like

	(setq active (or active '(1 . 0)))
	(erase-buffer)
	(insert (format "211 %d %d %d %S\n"
			(max (1+ (- (cdr active) (car active))) 0)
			(car active)
			(cdr active)
			group))

Now the active range is correctly set to (1 . 0) for an empty group.
This alone doesn't work, as the expression (- (cdr active) (car active))
in the current code would then give a negative value for the number of
articles, which breaks at least one other place in Gnus.  So we use
the calculation that nnml-request-group uses to get the correct number
of articles in the range (by adding 1), and also force negative values
to 0, just in case active contains some bogus data.

I think this is the correct way to proceed, in line with how all
the other parts of Gnus create the 211 line.

Dan


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnimap.patch --]
[-- Type: text/x-diff; name="nnimap.patch", Size: 594 bytes --]

diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
index 17a55f98..cdd9f01f 100644
--- a/lisp/gnus/nnimap.el
+++ b/lisp/gnus/nnimap.el
@@ -918,10 +918,10 @@ nnimap-request-group
 	    (nnimap-finish-retrieve-group-infos server info sequences
 						t)
 	    (setq active (nth 2 (assoc group nnimap-current-infos)))))
-	(setq active (or active '(0 . 1)))
+	(setq active (or active '(1 . 0)))
 	(erase-buffer)
 	(insert (format "211 %d %d %d %S\n"
-			(- (cdr active) (car active))
+			(max (1+ (- (cdr active) (car active))) 0)
 			(car active)
 			(cdr active)
 			group))

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Sep 18, 2024, James Thomas <jimjoe@gmx.net> wrote:

> James Thomas wrote:
>
> In fact, is the max needed? Wouldn't this work as well?

This is about the line

  (max (1+ (- (cdr active) (car active))) 0)

The max is there in case active contains a pair like (10 . 8), for which
(1+ (- (cdr active) (car active))) evaluates to -1.  While this seems
unlikely to happen, at least two other backends use the above code to
guard against it, and it seems wise to be generous in what we accept.

This is especially true because gnus-parse-active does not accept
negative numbers.

Dan

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[James Thomas]

Dan Christensen wrote:

> On Sep 18, 2024, James Thomas wrote:
>
>> James Thomas wrote:
>>
>> In fact, is the max needed? Wouldn't this work as well?
>
> This is about the line
>
>   (max (1+ (- (cdr active) (car active))) 0)
>
> The max is there in case active contains a pair like (10 . 8), for which
> (1+ (- (cdr active) (car active))) evaluates to -1.  While this seems
> unlikely to happen, at least two other backends use the above code to
> guard against it, and it seems wise to be generous in what we accept.
>
> This is especially true because gnus-parse-active does not accept
> negative numbers.

Cool, thanks!

--

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

[-- Attachment #1: Type: text/plain, Size: 156 bytes --]

So how do we go about getting this merged?  I've copied Eric
on this and have attached the patch again (since there are
several in the thread).

Dan


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnimap.patch --]
[-- Type: text/x-diff; name="nnimap.patch", Size: 594 bytes --]

diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
index 17a55f98..cdd9f01f 100644
--- a/lisp/gnus/nnimap.el
+++ b/lisp/gnus/nnimap.el
@@ -918,10 +918,10 @@ nnimap-request-group
 	    (nnimap-finish-retrieve-group-infos server info sequences
 						t)
 	    (setq active (nth 2 (assoc group nnimap-current-infos)))))
-	(setq active (or active '(0 . 1)))
+	(setq active (or active '(1 . 0)))
 	(erase-buffer)
 	(insert (format "211 %d %d %d %S\n"
-			(- (cdr active) (car active))
+			(max (1+ (- (cdr active) (car active))) 0)
 			(car active)
 			(cdr active)
 			group))

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Sep 24, 2024, Dan Christensen <jdc@uwo.ca> wrote:

> So how do we go about getting this merged?

I think someone should apply my nnimap.patch and close this bug report.

The existing code is clearly wrong, and triggers a reproducible bug.

The patch is clearly correct, mirroring what is done elsewhere in Gnus,
and fixing the reproducible bug.  I explained in detail in this thread
why it is correct.  I've also been using the patched code for around
two weeks without issue.

Dan

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Stefan Kangas]

Dan Christensen <jdc@uwo.ca> writes:

> On Sep 24, 2024, Dan Christensen <jdc@uwo.ca> wrote:
>
>> So how do we go about getting this merged?

I think we're just waiting for someone familiar with Gnus to take a look
at it.

Eric, or someone else who knows Gnus, could you please review this
patch?

> I think someone should apply my nnimap.patch and close this bug report.
>
> The existing code is clearly wrong, and triggers a reproducible bug.
>
> The patch is clearly correct, mirroring what is done elsewhere in Gnus,
> and fixing the reproducible bug.  I explained in detail in this thread
> why it is correct.  I've also been using the patched code for around
> two weeks without issue.

All of us are volunteers here, so it's unfortunately not guaranteed that
we can get to patches in anything that resembles a timely manner.  We
also don't have too many Gnus hackers on board.  While we do our best, I
can unfortunately only recommend patience here.

Please feel free to ping us again if there has been no progress within
another 1-2 months or so.  Ideally, this will be reviewed by someone
familiar with Gnus, but if it comes to it, I could take a cursory look
and push it if nothing stands out as obviously incorrect.

Thanks again for the patch, and for trying to move things forward.

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Oct  1, 2024, Stefan Kangas <stefankangas@gmail.com> wrote:

>> On Sep 24, 2024, Dan Christensen <jdc@uwo.ca> wrote:
>>
>>> So how do we go about getting this merged?
>
> I think we're just waiting for someone familiar with Gnus to take a
> look at it.

Ok, thanks for explaining more about the process.  I was hoping that
since the fix is minor, it could go in now, but I'm fine with waiting
for a review.

I also re-read my last message and realize that the tone came across
wrong.  I was trying to concisely summarize the situation, but was too
curt.

Best,

Dan

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Robert Pluim]

>>>>> On Tue, 1 Oct 2024 22:25:24 +0000, Dan Christensen <jdc@uwo.ca> said:

    Dan> On Oct  1, 2024, Stefan Kangas <stefankangas@gmail.com> wrote:
    >>> On Sep 24, 2024, Dan Christensen <jdc@uwo.ca> wrote:
    >>> 
    >>>> So how do we go about getting this merged?
    >> 
    >> I think we're just waiting for someone familiar with Gnus to take a
    >> look at it.

    Dan> Ok, thanks for explaining more about the process.  I was hoping that
    Dan> since the fix is minor, it could go in now, but I'm fine with waiting
    Dan> for a review.

Iʼve reviewed it, and it looks ok. Since I uses Gnus + nnimap here,
Iʼll apply it and see if it breaks anything.

Robert
-- 

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Robert Pluim]

>>>>> On Wed, 02 Oct 2024 18:55:28 +0200, Robert Pluim <rpluim@gmail.com> said:

>>>>> On Tue, 1 Oct 2024 22:25:24 +0000, Dan Christensen <jdc@uwo.ca> said:
    Dan> On Oct  1, 2024, Stefan Kangas <stefankangas@gmail.com> wrote:
    >>>> On Sep 24, 2024, Dan Christensen <jdc@uwo.ca> wrote:
    >>>> 
    >>>>> So how do we go about getting this merged?
    >>> 
    >>> I think we're just waiting for someone familiar with Gnus to take a
    >>> look at it.

    Dan> Ok, thanks for explaining more about the process.  I was hoping that
    Dan> since the fix is minor, it could go in now, but I'm fine with waiting
    Dan> for a review.

    Robert> Iʼve reviewed it, and it looks ok. Since I uses Gnus + nnimap here,
    Robert> Iʼll apply it and see if it breaks anything.

It doesnʼt appear to have broken anything here, so I think we can
apply it. I think itʼs small enough not to need copyright assignment,
but Iʼll leave it to the maintainers to confirm.

Robert
-- 

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Oct 11, 2024, Robert Pluim <rpluim@gmail.com> wrote:

> It doesnʼt appear to have broken anything here, so I think we can
> apply it. I think itʼs small enough not to need copyright assignment,
> but Iʼll leave it to the maintainers to confirm.

Thanks, Robert!  I submitted the copyright paperwork many years ago,
so we should be good to go.

Dan

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Robert Pluim]

>>>>> On Fri, 11 Oct 2024 13:45:05 +0000, Dan Christensen <jdc@uwo.ca> said:

    Dan> On Oct 11, 2024, Robert Pluim <rpluim@gmail.com> wrote:
    >> It doesnʼt appear to have broken anything here, so I think we can
    >> apply it. I think itʼs small enough not to need copyright assignment,
    >> but Iʼll leave it to the maintainers to confirm.

    Dan> Thanks, Robert!  I submitted the copyright paperwork many years ago,
    Dan> so we should be good to go.

Hmm, youʼre in etc/AUTHORS, but there are no commits with 'jdc@uwo.ca'
on them. Probably a victim of the Great Gnus Merge a few years ago :-)

Robert
-- 

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Stefan Kangas <stefankangas@gmail.com>,
>     Eric Abrahamsen <eric@ericabrahamsen.net>,  Eli Zaretskii
>  <eliz@gnu.org>,  <ding@gnus.org>,  <72949@debbugs.gnu.org>
> Date: Fri, 11 Oct 2024 15:36:43 +0200
> 
> >>>>> On Wed, 02 Oct 2024 18:55:28 +0200, Robert Pluim <rpluim@gmail.com> said:
> 
> >>>>> On Tue, 1 Oct 2024 22:25:24 +0000, Dan Christensen <jdc@uwo.ca> said:
>     Dan> On Oct  1, 2024, Stefan Kangas <stefankangas@gmail.com> wrote:
>     >>>> On Sep 24, 2024, Dan Christensen <jdc@uwo.ca> wrote:
>     >>>> 
>     >>>>> So how do we go about getting this merged?
>     >>> 
>     >>> I think we're just waiting for someone familiar with Gnus to take a
>     >>> look at it.
> 
>     Dan> Ok, thanks for explaining more about the process.  I was hoping that
>     Dan> since the fix is minor, it could go in now, but I'm fine with waiting
>     Dan> for a review.
> 
>     Robert> Iʼve reviewed it, and it looks ok. Since I uses Gnus + nnimap here,
>     Robert> Iʼll apply it and see if it breaks anything.
> 
> It doesnʼt appear to have broken anything here, so I think we can
> apply it. I think itʼs small enough not to need copyright assignment,
> but Iʼll leave it to the maintainers to confirm.

It's small enough, yes.  Just please don't forget the
Copyright-paperwork-exempt thingy.

Thanks.

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Robert Pluim]

tags 72949 fixed
close 72949 31.1
quit

>>>>> On Fri, 11 Oct 2024 18:39:33 +0300, Eli Zaretskii <eliz@gnu.org> said:

    Eli> It's small enough, yes.  Just please don't forget the
    Eli> Copyright-paperwork-exempt thingy.

Given Danʼs comments about assignment, Iʼm not 100% sure itʼs
necessary, but I put it in anyway.

Closing.
Committed as 5c129791c5b

Robert
-- 

Re: bug#72949: Gnus sometimes reports new messages but not showing them on IMAP server

[Dan Christensen]

On Sep 15, 2024, Andreas Schwab <schwab@linux-m68k.org> wrote:

> On Sep 14 2024, Dan Christensen wrote:
>
>> In particular, for an empty group, the "low" value will be 1 while the
>> "high" value will be 0.
>
>    If the group is empty, one of the following three situations will
>    occur.  Clients MUST accept all three cases; servers MUST NOT
>    represent an empty group in any other way.
>
>    o  The high water mark will be one less than the low water mark, and
>       the estimated article count will be zero.  Servers SHOULD use this
>       method to show an empty group.  This is the only time that the
>       high water mark can be less than the low water mark.

Thanks, that again confirms that my proposed patch is good.

Dan

PS: Some messages went to the bug tracker but not the ding list, so
I'm including the link to the bug report here for those on the ding
list who want to see the missing messages:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72949