* Gnus sometimes reports new messages but not showing them on IMAP server @ 2024-04-12 0:57 Xiyue Deng 2024-04-17 0:03 ` Greg Troxel 2024-04-17 19:51 ` Dan Christensen 0 siblings, 2 replies; 41+ messages in thread From: Xiyue Deng @ 2024-04-12 0:57 UTC (permalink / raw) To: ding Hi, I'm having some issues that Gnus sometimes reports that there are new messages on an IMAP server but it doesn't show them. In my case it's using Gmail IMAP, and if I check using other mail clients (e.g. Thunderbird) or directly check on Gmail website I can seem them. I'm not sure whether this issue is specific to IMAP either. As I'm not sure how to get more info, I didn't use report bug yet, but would like to ask around and see whether this is a known issue, or otherwise I can get some suggestions to get more debug info before I formally file a bug. Any suggestions are welcome! TIA! -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-12 0:57 Gnus sometimes reports new messages but not showing them on IMAP server Xiyue Deng @ 2024-04-17 0:03 ` Greg Troxel 2024-04-17 6:47 ` Xiyue Deng 2024-04-17 19:51 ` Dan Christensen 1 sibling, 1 reply; 41+ messages in thread From: Greg Troxel @ 2024-04-17 0:03 UTC (permalink / raw) To: Xiyue Deng; +Cc: ding Xiyue Deng <manphiz@gmail.com> writes: > I'm having some issues that Gnus sometimes reports that there are new > messages on an IMAP server but it doesn't show them. In my case it's > using Gmail IMAP, and if I check using other mail clients > (e.g. Thunderbird) or directly check on Gmail website I can seem them. > I'm not sure whether this issue is specific to IMAP either. As I'm not > sure how to get more info, I didn't use report bug yet, but would like > to ask around and see whether this is a known issue, or otherwise I can > get some suggestions to get more debug info before I formally file a > bug. I have been having a problem for several years that may be the same thing. My mail is in IMAP via dovecot. Other than the following I have zero problems with it. I read mail with gnus (emacs built w/o x11 running in tmux in an xterm, on NetBSD, currently 28.2 and thus the included gnus), with about 300 folders, about half of which have server-side filing of list mail. I also use K-9 Mail on Android (INBOX and maybe 3 more mailboxes, more or less, ignoring the folders for lists and storage). Occasionally I use thunderbird for INBOX, to read something that should have been plain text and isn't :-) but this issue has happened with no tbird running at all. Often, I get a message on my phone that I just leave (marked read) to deal with later. Sometimes, maybe 1 in 50, I can't find it in gnus, and on looking harder find that it is just not showing up there. Yet K-9 sees it as present and looking in ~/IMAP/cur shows it was well. Whenever this has happened and I look with Thunderbird, it sees it fine also. I have found that if I move the message to another folder (I have TMP for this very purpose) and then back to INBOX, it can then be found in gnus. So something about the message as it sits in the folder was triggering gnus to miss it, while the very same message refiled does not trigger this. I have not been able to determine any pattern of which messages get this disappearing treatment. It's not really the message, since moving and back fixes it. Early on, I sort of suspected dovecot, but other people don't have trouble with dovecot, and I have had zero flakiness other than this specific issue. I have not perceived "gnus says there are new messages but there aren't". I am not at all sure I would have noticed, especially if gnus said 6 and there were 5. In your case, if you do "100<spc>" in group, do you see the missing message? If not, you should be able to turn on some kind of debugging to figure out what's going on, but I am unclear on the details. Greg ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-17 0:03 ` Greg Troxel @ 2024-04-17 6:47 ` Xiyue Deng 0 siblings, 0 replies; 41+ messages in thread From: Xiyue Deng @ 2024-04-17 6:47 UTC (permalink / raw) To: ding Greg Troxel <gdt@lexort.com> writes: > Xiyue Deng <manphiz@gmail.com> writes: > >> I'm having some issues that Gnus sometimes reports that there are new >> messages on an IMAP server but it doesn't show them. In my case it's >> using Gmail IMAP, and if I check using other mail clients >> (e.g. Thunderbird) or directly check on Gmail website I can seem them. >> I'm not sure whether this issue is specific to IMAP either. As I'm not >> sure how to get more info, I didn't use report bug yet, but would like >> to ask around and see whether this is a known issue, or otherwise I can >> get some suggestions to get more debug info before I formally file a >> bug. > > I have been having a problem for several years that may be the same > thing. > > My mail is in IMAP via dovecot. Other than the following I have zero > problems with it. > > I read mail with gnus (emacs built w/o x11 running in tmux in an xterm, > on NetBSD, currently 28.2 and thus the included gnus), with about 300 > folders, about half of which have server-side filing of list mail. I > also use K-9 Mail on Android (INBOX and maybe 3 more mailboxes, more or > less, ignoring the folders for lists and storage). Occasionally I use > thunderbird for INBOX, to read something that should have been plain > text and isn't :-) but this issue has happened with no tbird running at > all. > > Often, I get a message on my phone that I just leave (marked read) to > deal with later. Sometimes, maybe 1 in 50, I can't find it in gnus, and > on looking harder find that it is just not showing up there. Yet K-9 > sees it as present and looking in ~/IMAP/cur shows it was well. > Whenever this has happened and I look with Thunderbird, it sees it fine > also. > Exactly this symptom. > I have found that if I move the message to another folder (I have TMP > for this very purpose) and then back to INBOX, it can then be found in > gnus. So something about the message as it sits in the folder was > triggering gnus to miss it, while the very same message refiled does not > trigger this. > I haven't tried this in this case, but there is actually another issue that I haven't talked about: sometimes one of the empty folders may report a number of new messages in it! In one of the occurrences it reports 52 new message on an empty folder. The solution, however, is similar to what you did but reverse: you move a number of messages from another folder to it, and the new message count will reduce by the number of messages you moved! So in the previous case if you move 10 messages to the folder, the count becomes 42. You move them out, and move them back again, 32, etc. Let me also try your solution in future. > I have not been able to determine any pattern of which messages get this > disappearing treatment. It's not really the message, since moving and > back fixes it. > > Early on, I sort of suspected dovecot, but other people don't have > trouble with dovecot, and I have had zero flakiness other than this > specific issue. > > I have not perceived "gnus says there are new messages but there > aren't". I am not at all sure I would have noticed, especially if gnus > said 6 and there were 5. > I have only encountered such cases when the folder is empty. > In your case, if you do "100<spc>" in group, do you see the missing > message? If not, you should be able to turn on some kind of debugging > to figure out what's going on, but I am unclear on the details. > No, specifying a number didn't work for me, unfortunately. And I guess to really understand this issue one may have to read the code of Gnus or IMAP backend or hopefully someone that does may be able to help. > Greg > > -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-12 0:57 Gnus sometimes reports new messages but not showing them on IMAP server Xiyue Deng 2024-04-17 0:03 ` Greg Troxel @ 2024-04-17 19:51 ` Dan Christensen 2024-04-18 12:13 ` Greg Troxel 1 sibling, 1 reply; 41+ messages in thread From: Dan Christensen @ 2024-04-17 19:51 UTC (permalink / raw) To: ding On Apr 11, 2024, Xiyue Deng <manphiz@gmail.com> wrote: > I'm having some issues that Gnus sometimes reports that there are new > messages on an IMAP server but it doesn't show them. On Apr 16, 2024, Greg Troxel <gdt@lexort.com> wrote: > Often, I get a message on my phone that I just leave (marked read) to > deal with later. Sometimes, maybe 1 in 50, I can't find it in gnus, and > on looking harder find that it is just not showing up there. For both of you, does hitting `M-g' with point on the group in the *Group* buffer fix things? I regularly have similar problems, and I've resorted to binding the following function to a key: (defun jdc-gnus-update-IMAP () (interactive) ; Only run this in the group buffer or it corrupts other buffers: (switch-to-buffer gnus-group-buffer) (gnus-topic-jump-to-topic "IMAP groups") (gnus-topic-get-new-news-this-topic) (gnus-group-list-groups) ; "l", as previous command shows all groups in topic ) I think it's doing the equivalent of `M-g' on all groups in the topic I have that contains my IMAP groups. For the record, I'm also using dovecot and k9-mail. Dan ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-17 19:51 ` Dan Christensen @ 2024-04-18 12:13 ` Greg Troxel 2024-04-19 22:55 ` Dan Christensen 0 siblings, 1 reply; 41+ messages in thread From: Greg Troxel @ 2024-04-18 12:13 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > On Apr 11, 2024, Xiyue Deng <manphiz@gmail.com> wrote: > >> I'm having some issues that Gnus sometimes reports that there are new >> messages on an IMAP server but it doesn't show them. > > On Apr 16, 2024, Greg Troxel <gdt@lexort.com> wrote: > >> Often, I get a message on my phone that I just leave (marked read) to >> deal with later. Sometimes, maybe 1 in 50, I can't find it in gnus, and >> on looking harder find that it is just not showing up there. > > For both of you, does hitting `M-g' with point on the group in the > *Group* buffer fix things? I will try that, but I have been exiting the group and ^U SPC to re-enter and display all, and when a message has gone missing, I have not even once been able to get gnus to show it. I have exited and restarted gnus, and done "g". > I regularly have similar problems, and I've resorted to binding the > following function to a key: > > (defun jdc-gnus-update-IMAP () > (interactive) > ; Only run this in the group buffer or it corrupts other buffers: > (switch-to-buffer gnus-group-buffer) > (gnus-topic-jump-to-topic "IMAP groups") > (gnus-topic-get-new-news-this-topic) > (gnus-group-list-groups) ; "l", as previous command shows all groups in topic > ) > > I think it's doing the equivalent of `M-g' on all groups in the topic > I have that contains my IMAP groups. For the record, I'm also using > dovecot and k9-mail. Are you saying that "g" in the summary does not allow viewing a missing article, but M-g will? To me, the problem is that these message are somehow invisible, even after multiple attempts to fetch/display them. ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-18 12:13 ` Greg Troxel @ 2024-04-19 22:55 ` Dan Christensen 2024-04-21 12:08 ` Arash Esbati 0 siblings, 1 reply; 41+ messages in thread From: Dan Christensen @ 2024-04-19 22:55 UTC (permalink / raw) To: ding On Apr 18, 2024, Greg Troxel <gdt@lexort.com> wrote: > Dan Christensen <jdc@uwo.ca> writes: > >> For both of you, does hitting `M-g' with point on the group in the >> *Group* buffer fix things? > > I will try that, but I have been exiting the group and ^U SPC to > re-enter and display all, and when a message has gone missing, I have > not even once been able to get gnus to show it. I have exited and > restarted gnus, and done "g". Neither of those does the full resync that `M-g' does for IMAP groups. I don't know what is going on behind the scenes, but `M-g' is necessary for me to see up-to-date information sometimes. > Are you saying that "g" in the summary does not allow viewing a missing > article, but M-g will? Yes, in many cases. Dan ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-19 22:55 ` Dan Christensen @ 2024-04-21 12:08 ` Arash Esbati 2024-04-21 12:43 ` Dan Christensen ` (2 more replies) 0 siblings, 3 replies; 41+ messages in thread From: Arash Esbati @ 2024-04-21 12:08 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > Neither of those does the full resync that `M-g' does for IMAP groups. I can confirm this. > I don't know what is going on behind the scenes, but `M-g' is necessary > for me to see up-to-date information sometimes. I haven't tried to read/understand the code, but in the Group buffer, neither 'g' nor 'C-u g' does what 'M-g' does inside the Summary buffer of an IMAP group. And I would have expected that 'C-u g' in Group buffer fits the bill: ,----[ C-h f gnus-group-get-new-news RET ] | gnus-group-get-new-news is an autoloaded interactive subr-native-elisp | in ‘gnus-group.el’. | | (gnus-group-get-new-news &optional ARG ONE-LEVEL) | | Get newly arrived articles. | If ARG is a number, it specifies which levels you are interested in | re-scanning. If ARG is non-nil and not a number, this will force | "hard" re-reading of the active files from all servers. | If ONE-LEVEL is not nil, then re-scan only the specified level, | otherwise all levels below ARG will be scanned too. | `---- Hopefully Eric can shed some light on this. Best, Arash ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-21 12:08 ` Arash Esbati @ 2024-04-21 12:43 ` Dan Christensen 2024-04-21 13:14 ` Andreas Schwab 2024-04-21 16:18 ` Eric Abrahamsen 2 siblings, 0 replies; 41+ messages in thread From: Dan Christensen @ 2024-04-21 12:43 UTC (permalink / raw) To: ding On Apr 21, 2024, Arash Esbati <arash@gnu.org> wrote: > Dan Christensen <jdc@uwo.ca> writes: > >> I don't know what is going on behind the scenes, but `M-g' is necessary >> for me to see up-to-date information sometimes. > > I haven't tried to read/understand the code, but in the Group buffer, > neither 'g' nor 'C-u g' does what 'M-g' does inside the Summary buffer > of an IMAP group. Just to clarify, I use `M-g' in the Group buffer, but looking at the code I think using it in the Summary buffer does the same thing. Aside: While looking at the code, I think I noticed an error in the doc string of the function that ends up getting called: (defun gnus-group-get-new-news-this-group (&optional n dont-scan) "Check for newly arrived news in the current group (and the N-1 next groups). The difference between N and the number of newsgroup checked is returned. If N is negative, this group and the N-1 previous groups will be checked. If DONT-SCAN is non-nil, scan non-activated groups as well." I think the description of DONT-SCAN is inverted (as its name suggests). It should say: If DONT-SCAN is nil, scan non-activated groups as well. Or If DONT-SCAN is non-nil, then non-activated groups are not scanned. Dan ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-21 12:08 ` Arash Esbati 2024-04-21 12:43 ` Dan Christensen @ 2024-04-21 13:14 ` Andreas Schwab 2024-04-21 16:18 ` Eric Abrahamsen 2 siblings, 0 replies; 41+ messages in thread From: Andreas Schwab @ 2024-04-21 13:14 UTC (permalink / raw) To: ding On Apr 21 2024, Arash Esbati wrote: > Dan Christensen <jdc@uwo.ca> writes: > >> Neither of those does the full resync that `M-g' does for IMAP groups. > > I can confirm this. > >> I don't know what is going on behind the scenes, but `M-g' is necessary >> for me to see up-to-date information sometimes. > > I haven't tried to read/understand the code, but in the Group buffer, > neither 'g' nor 'C-u g' does what 'M-g' does inside the Summary buffer > of an IMAP group. There is also M-g (gnus-group-get-new-news-this-group) in the Group buffer. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different." ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-21 12:08 ` Arash Esbati 2024-04-21 12:43 ` Dan Christensen 2024-04-21 13:14 ` Andreas Schwab @ 2024-04-21 16:18 ` Eric Abrahamsen 2024-04-21 18:16 ` Arash Esbati 2024-04-24 15:23 ` Gnus sometimes reports new messages but not showing them on IMAP server James Thomas 2 siblings, 2 replies; 41+ messages in thread From: Eric Abrahamsen @ 2024-04-21 16:18 UTC (permalink / raw) To: ding Arash Esbati <arash@gnu.org> writes: > Dan Christensen <jdc@uwo.ca> writes: > >> Neither of those does the full resync that `M-g' does for IMAP groups. > > I can confirm this. > >> I don't know what is going on behind the scenes, but `M-g' is necessary >> for me to see up-to-date information sometimes. > > I haven't tried to read/understand the code, but in the Group buffer, > neither 'g' nor 'C-u g' does what 'M-g' does inside the Summary buffer > of an IMAP group. And I would have expected that 'C-u g' in Group > buffer fits the bill: > > ,----[ C-h f gnus-group-get-new-news RET ] > | gnus-group-get-new-news is an autoloaded interactive subr-native-elisp > | in ‘gnus-group.el’. > | > | (gnus-group-get-new-news &optional ARG ONE-LEVEL) > | > | Get newly arrived articles. > | If ARG is a number, it specifies which levels you are interested in > | re-scanning. If ARG is non-nil and not a number, this will force > | "hard" re-reading of the active files from all servers. > | If ONE-LEVEL is not nil, then re-scan only the specified level, > | otherwise all levels below ARG will be scanned too. > | > `---- > > Hopefully Eric can shed some light on this. This is a long-standing bug, probably the most annoying thing about nnimap. I've made a few attempts at it over the years, but so far haven't been successful. I believe what it comes down to is that "M-g" dumps Gnus' "active" number and re-reads it from the IMAP server, while regular "g" incrementally modifies the active number based on newest changes. So if anything goes wrong with Gnus' active number, errors will accumulate from that point until the user does "M-g". I've focused on figuring out why the active number goes wrong, but in the interest of pragmatism it might be better just to make sure that nnimap always does the equivalent of "M-g" at Gnus startup. ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-21 16:18 ` Eric Abrahamsen @ 2024-04-21 18:16 ` Arash Esbati 2024-04-22 0:00 ` Greg Troxel 2024-04-24 15:23 ` Gnus sometimes reports new messages but not showing them on IMAP server James Thomas 1 sibling, 1 reply; 41+ messages in thread From: Arash Esbati @ 2024-04-21 18:16 UTC (permalink / raw) To: ding Eric Abrahamsen <eric@ericabrahamsen.net> writes: > This is a long-standing bug, probably the most annoying thing about > nnimap. I've made a few attempts at it over the years, but so far > haven't been successful. I believe what it comes down to is that "M-g" > dumps Gnus' "active" number and re-reads it from the IMAP server, while > regular "g" incrementally modifies the active number based on newest > changes. So if anything goes wrong with Gnus' active number, errors will > accumulate from that point until the user does "M-g". Thanks for your response. I agree, this is indeed annoying -- hitting on 'M-g' on every IMAP-Group or inside a Group isn't much fun. > I've focused on figuring out why the active number goes wrong, but in > the interest of pragmatism it might be better just to make sure that > nnimap always does the equivalent of "M-g" at Gnus startup. That might be a good compromise. Best, Arash ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-21 18:16 ` Arash Esbati @ 2024-04-22 0:00 ` Greg Troxel 2024-04-22 2:35 ` Eric Abrahamsen 0 siblings, 1 reply; 41+ messages in thread From: Greg Troxel @ 2024-04-22 0:00 UTC (permalink / raw) To: ding Arash Esbati <arash@gnu.org> writes: >> I've focused on figuring out why the active number goes wrong, but in >> the interest of pragmatism it might be better just to make sure that >> nnimap always does the equivalent of "M-g" at Gnus startup. > > That might be a good compromise. Agreed. It seems obvious (said knowing that's dangerous) that on starting up the program, it's best not to rely on state that could be wrong. That's tricky, because mail programs keep messages and don't refetch them, but relying on counts of unread seems very trouble prone, vs "I have this message body and the timestamp matches so I won't fetch it." I think it would also be good to separate check for new get all the messages that are there really rescan and don't believe state in terms of being much louder about the differences. I have the impression that the assembled gnususers here are at least somewhat hazy on those boundaries. At least I am. ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-22 0:00 ` Greg Troxel @ 2024-04-22 2:35 ` Eric Abrahamsen 2024-04-27 19:46 ` Xiyue Deng 0 siblings, 1 reply; 41+ messages in thread From: Eric Abrahamsen @ 2024-04-22 2:35 UTC (permalink / raw) To: ding Greg Troxel <gdt@lexort.com> writes: > Arash Esbati <arash@gnu.org> writes: > >>> I've focused on figuring out why the active number goes wrong, but in >>> the interest of pragmatism it might be better just to make sure that >>> nnimap always does the equivalent of "M-g" at Gnus startup. >> >> That might be a good compromise. > > Agreed. It seems obvious (said knowing that's dangerous) that on > starting up the program, it's best not to rely on state that could be > wrong. That's tricky, because mail programs keep messages and don't > refetch them, but relying on counts of unread seems very trouble prone, > vs "I have this message body and the timestamp matches so I won't fetch > it." > > I think it would also be good to separate > > check for new > > get all the messages that are there > > really rescan and don't believe state > > in terms of being much louder about the differences. I have the > impression that the assembled gnususers here are at least somewhat hazy > on those boundaries. At least I am. I think this is an artifact of Gnus starting out as a newsreader, and adding email capability later. In a NNTP context, the read/unread message count is entirely private to your local machine, and something you can manipulate at will. In an IMAP context, it's very much a part of multi-machine state. Gnus should always defer to the remote server, particularly at start up. I do wish I knew where the bug was coming from, though. ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-22 2:35 ` Eric Abrahamsen @ 2024-04-27 19:46 ` Xiyue Deng 2024-05-02 5:47 ` Xiyue Deng 2024-05-10 0:43 ` Greg Troxel 0 siblings, 2 replies; 41+ messages in thread From: Xiyue Deng @ 2024-04-27 19:46 UTC (permalink / raw) To: ding Eric Abrahamsen <eric@ericabrahamsen.net> writes: > Greg Troxel <gdt@lexort.com> writes: > >> Arash Esbati <arash@gnu.org> writes: >> >>>> I've focused on figuring out why the active number goes wrong, but in >>>> the interest of pragmatism it might be better just to make sure that >>>> nnimap always does the equivalent of "M-g" at Gnus startup. >>> >>> That might be a good compromise. >> >> Agreed. It seems obvious (said knowing that's dangerous) that on >> starting up the program, it's best not to rely on state that could be >> wrong. That's tricky, because mail programs keep messages and don't >> refetch them, but relying on counts of unread seems very trouble prone, >> vs "I have this message body and the timestamp matches so I won't fetch >> it." >> >> I think it would also be good to separate >> >> check for new >> >> get all the messages that are there >> >> really rescan and don't believe state >> >> in terms of being much louder about the differences. I have the >> impression that the assembled gnususers here are at least somewhat hazy >> on those boundaries. At least I am. > > I think this is an artifact of Gnus starting out as a newsreader, and > adding email capability later. In a NNTP context, the read/unread > message count is entirely private to your local machine, and something > you can manipulate at will. In an IMAP context, it's very much a part of > multi-machine state. Gnus should always defer to the remote server, > particularly at start up. I do wish I knew where the bug was coming > from, though. > > > This makes much sense. I'll try M-g the next time this happens. Thanks everyone! Also I would expect the equivalent of M-g to happen on startup so that there are less surprises. -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-27 19:46 ` Xiyue Deng @ 2024-05-02 5:47 ` Xiyue Deng 2024-05-02 15:08 ` Eric Abrahamsen 2024-05-10 0:43 ` Greg Troxel 1 sibling, 1 reply; 41+ messages in thread From: Xiyue Deng @ 2024-05-02 5:47 UTC (permalink / raw) To: ding Xiyue Deng <manphiz@gmail.com> writes: > Eric Abrahamsen <eric@ericabrahamsen.net> writes: > >> Greg Troxel <gdt@lexort.com> writes: >> >>> Arash Esbati <arash@gnu.org> writes: >>> >>>>> I've focused on figuring out why the active number goes wrong, but in >>>>> the interest of pragmatism it might be better just to make sure that >>>>> nnimap always does the equivalent of "M-g" at Gnus startup. >>>> >>>> That might be a good compromise. >>> >>> Agreed. It seems obvious (said knowing that's dangerous) that on >>> starting up the program, it's best not to rely on state that could be >>> wrong. That's tricky, because mail programs keep messages and don't >>> refetch them, but relying on counts of unread seems very trouble prone, >>> vs "I have this message body and the timestamp matches so I won't fetch >>> it." >>> >>> I think it would also be good to separate >>> >>> check for new >>> >>> get all the messages that are there >>> >>> really rescan and don't believe state >>> >>> in terms of being much louder about the differences. I have the >>> impression that the assembled gnususers here are at least somewhat hazy >>> on those boundaries. At least I am. >> >> I think this is an artifact of Gnus starting out as a newsreader, and >> adding email capability later. In a NNTP context, the read/unread >> message count is entirely private to your local machine, and something >> you can manipulate at will. In an IMAP context, it's very much a part of >> multi-machine state. Gnus should always defer to the remote server, >> particularly at start up. I do wish I knew where the bug was coming >> from, though. >> >> >> > > This makes much sense. I'll try M-g the next time this happens. Thanks > everyone! > > Also I would expect the equivalent of M-g to happen on startup so that > there are less surprises. So the issue Greg described just happened again that an empty folder/topic "Trash" from one of my outlook accounts just shown 38 unread messages and selecting it fails with "Can't select group", while other clients like Thunderbird shows 0 in it. I have tried "M-g" and it doesn't help unfortunately :( Any more ideas for debugging? -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-02 5:47 ` Xiyue Deng @ 2024-05-02 15:08 ` Eric Abrahamsen 2024-05-02 17:24 ` Xiyue Deng 0 siblings, 1 reply; 41+ messages in thread From: Eric Abrahamsen @ 2024-05-02 15:08 UTC (permalink / raw) To: ding Xiyue Deng <manphiz@gmail.com> writes: > Xiyue Deng <manphiz@gmail.com> writes: > >> Eric Abrahamsen <eric@ericabrahamsen.net> writes: >> >>> Greg Troxel <gdt@lexort.com> writes: >>> >>>> Arash Esbati <arash@gnu.org> writes: >>>> >>>>>> I've focused on figuring out why the active number goes wrong, but in >>>>>> the interest of pragmatism it might be better just to make sure that >>>>>> nnimap always does the equivalent of "M-g" at Gnus startup. >>>>> >>>>> That might be a good compromise. >>>> >>>> Agreed. It seems obvious (said knowing that's dangerous) that on >>>> starting up the program, it's best not to rely on state that could be >>>> wrong. That's tricky, because mail programs keep messages and don't >>>> refetch them, but relying on counts of unread seems very trouble prone, >>>> vs "I have this message body and the timestamp matches so I won't fetch >>>> it." >>>> >>>> I think it would also be good to separate >>>> >>>> check for new >>>> >>>> get all the messages that are there >>>> >>>> really rescan and don't believe state >>>> >>>> in terms of being much louder about the differences. I have the >>>> impression that the assembled gnususers here are at least somewhat hazy >>>> on those boundaries. At least I am. >>> >>> I think this is an artifact of Gnus starting out as a newsreader, and >>> adding email capability later. In a NNTP context, the read/unread >>> message count is entirely private to your local machine, and something >>> you can manipulate at will. In an IMAP context, it's very much a part of >>> multi-machine state. Gnus should always defer to the remote server, >>> particularly at start up. I do wish I knew where the bug was coming >>> from, though. >>> >>> >>> >> >> This makes much sense. I'll try M-g the next time this happens. Thanks >> everyone! >> >> Also I would expect the equivalent of M-g to happen on startup so that >> there are less surprises. > > So the issue Greg described just happened again that an empty > folder/topic "Trash" from one of my outlook accounts just shown 38 > unread messages and selecting it fails with "Can't select group", while > other clients like Thunderbird shows 0 in it. I have tried "M-g" and it > doesn't help unfortunately :( > > Any more ideas for debugging? Maybe move a message into the folder, then move it back out? I'll try to get to debugging this area of Gnus in the next few weeks... ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-02 15:08 ` Eric Abrahamsen @ 2024-05-02 17:24 ` Xiyue Deng 2024-05-02 17:41 ` Dan Christensen 0 siblings, 1 reply; 41+ messages in thread From: Xiyue Deng @ 2024-05-02 17:24 UTC (permalink / raw) To: ding Eric Abrahamsen <eric@ericabrahamsen.net> writes: > Xiyue Deng <manphiz@gmail.com> writes: > >> Xiyue Deng <manphiz@gmail.com> writes: >> >>> Eric Abrahamsen <eric@ericabrahamsen.net> writes: >>> >>>> Greg Troxel <gdt@lexort.com> writes: >>>> >>>>> Arash Esbati <arash@gnu.org> writes: >>>>> >>>>>>> I've focused on figuring out why the active number goes wrong, but in >>>>>>> the interest of pragmatism it might be better just to make sure that >>>>>>> nnimap always does the equivalent of "M-g" at Gnus startup. >>>>>> >>>>>> That might be a good compromise. >>>>> >>>>> Agreed. It seems obvious (said knowing that's dangerous) that on >>>>> starting up the program, it's best not to rely on state that could be >>>>> wrong. That's tricky, because mail programs keep messages and don't >>>>> refetch them, but relying on counts of unread seems very trouble prone, >>>>> vs "I have this message body and the timestamp matches so I won't fetch >>>>> it." >>>>> >>>>> I think it would also be good to separate >>>>> >>>>> check for new >>>>> >>>>> get all the messages that are there >>>>> >>>>> really rescan and don't believe state >>>>> >>>>> in terms of being much louder about the differences. I have the >>>>> impression that the assembled gnususers here are at least somewhat hazy >>>>> on those boundaries. At least I am. >>>> >>>> I think this is an artifact of Gnus starting out as a newsreader, and >>>> adding email capability later. In a NNTP context, the read/unread >>>> message count is entirely private to your local machine, and something >>>> you can manipulate at will. In an IMAP context, it's very much a part of >>>> multi-machine state. Gnus should always defer to the remote server, >>>> particularly at start up. I do wish I knew where the bug was coming >>>> from, though. >>>> >>>> >>>> >>> >>> This makes much sense. I'll try M-g the next time this happens. Thanks >>> everyone! >>> >>> Also I would expect the equivalent of M-g to happen on startup so that >>> there are less surprises. >> >> So the issue Greg described just happened again that an empty >> folder/topic "Trash" from one of my outlook accounts just shown 38 >> unread messages and selecting it fails with "Can't select group", while >> other clients like Thunderbird shows 0 in it. I have tried "M-g" and it >> doesn't help unfortunately :( >> >> Any more ideas for debugging? > > Maybe move a message into the folder, then move it back out? Ah I was aware of this workaround. Just wonder whether there is metadata/debug info that I can try to get to help the developer such as you to debug this? > I'll try to get to debugging this area of Gnus in the next few > weeks... Really appreciate this! Meanwhile probably I should try to submit a bug for tracking. Is there a Gnus specific bug tracker or I should just use report-emacs-bug? > > > -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-02 17:24 ` Xiyue Deng @ 2024-05-02 17:41 ` Dan Christensen 2024-05-02 18:41 ` Xiyue Deng 0 siblings, 1 reply; 41+ messages in thread From: Dan Christensen @ 2024-05-02 17:41 UTC (permalink / raw) To: ding On May 2, 2024, Xiyue Deng <manphiz@gmail.com> wrote: > Ah I was aware of this workaround. Just wonder whether there is > metadata/debug info that I can try to get to help the developer such as > you to debug this? You could hit `G p' on the group in the *Group* buffer and show us what is displayed. (You can hit `C-c C-k' in that buffer to kill it.) I sometimes get a situation where the active range starts at 0 instead of 1, and it's very hard to fix. I fiddle around with changing things in that buffer, closing emacs, etc, and can fix it, but I'm not sure exactly what the recipe is. Dan ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-02 17:41 ` Dan Christensen @ 2024-05-02 18:41 ` Xiyue Deng 2024-05-05 20:18 ` Xiyue Deng 0 siblings, 1 reply; 41+ messages in thread From: Xiyue Deng @ 2024-05-02 18:41 UTC (permalink / raw) To: ding Hi Dan, Dan Christensen <jdc@uwo.ca> writes: > On May 2, 2024, Xiyue Deng <manphiz@gmail.com> wrote: > >> Ah I was aware of this workaround. Just wonder whether there is >> metadata/debug info that I can try to get to help the developer such as >> you to debug this? > > You could hit `G p' on the group in the *Group* buffer and show us > what is displayed. (You can hit `C-c C-k' in that buffer to kill it.) > Thanks for the tip! However I don't know what happened the count of problematic topic was back to 0 since this morning, so I'll probably have to wait for the next occurrence to try this. > I sometimes get a situation where the active range starts at 0 instead > of 1, and it's very hard to fix. I fiddle around with changing things > in that buffer, closing emacs, etc, and can fix it, but I'm not sure > exactly what the recipe is. > Noted. Will keep an eye on this next time. > Dan > > > -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-02 18:41 ` Xiyue Deng @ 2024-05-05 20:18 ` Xiyue Deng 2024-05-05 21:01 ` Dan Christensen 0 siblings, 1 reply; 41+ messages in thread From: Xiyue Deng @ 2024-05-05 20:18 UTC (permalink / raw) To: ding Xiyue Deng <manphiz@gmail.com> writes: > Hi Dan, > > Dan Christensen <jdc@uwo.ca> writes: > >> On May 2, 2024, Xiyue Deng <manphiz@gmail.com> wrote: >> >>> Ah I was aware of this workaround. Just wonder whether there is >>> metadata/debug info that I can try to get to help the developer such as >>> you to debug this? >> >> You could hit `G p' on the group in the *Group* buffer and show us >> what is displayed. (You can hit `C-c C-k' in that buffer to kill it.) >> > > Thanks for the tip! However I don't know what happened the count of > problematic topic was back to 0 since this morning, so I'll probably > have to wait for the next occurrence to try this. > So this happened again in one of my outlook accounts' Sent folder, and the "G p" information is as below: ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 7) | (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent)) `---- >> I sometimes get a situation where the active range starts at 0 instead >> of 1, and it's very hard to fix. I fiddle around with changing things >> in that buffer, closing emacs, etc, and can fix it, but I'm not sure >> exactly what the recipe is. >> > > Noted. Will keep an eye on this next time. > This time the active range does start from 1. So it seems that it does think there are 7 active items? >> Dan >> >> >> -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-05 20:18 ` Xiyue Deng @ 2024-05-05 21:01 ` Dan Christensen 2024-05-05 22:10 ` Xiyue Deng 0 siblings, 1 reply; 41+ messages in thread From: Dan Christensen @ 2024-05-05 21:01 UTC (permalink / raw) To: ding On May 5, 2024, Xiyue Deng <manphiz@gmail.com> wrote: > So this happened again in one of my outlook accounts' Sent folder, and > the "G p" information is as below: > > ,---- > | ((modseq) > | (uidvalidity . "11") > | (active 1 . 7) > | (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent)) > `---- I guess you are having a different problem than I am. I assume that you tried `M-g', and the above doesn't change? What if you go into the server buffer with `^' and then hit `C' on the server to close it and `O' on the server to open it? Dan ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-05 21:01 ` Dan Christensen @ 2024-05-05 22:10 ` Xiyue Deng 2024-05-08 18:36 ` Xiyue Deng 2024-05-08 18:41 ` Xiyue Deng 0 siblings, 2 replies; 41+ messages in thread From: Xiyue Deng @ 2024-05-05 22:10 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > On May 5, 2024, Xiyue Deng <manphiz@gmail.com> wrote: > >> So this happened again in one of my outlook accounts' Sent folder, and >> the "G p" information is as below: >> >> ,---- >> | ((modseq) >> | (uidvalidity . "11") >> | (active 1 . 7) >> | (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent)) >> `---- > > I guess you are having a different problem than I am. I assume that > you tried `M-g', and the above doesn't change? Yes, and it didn't fix it. > What if you go into the server buffer with `^' and then hit `C' on the > server to close it and `O' on the server to open it? > Tried this as well, and unfortunately it didn't fix it either. > Dan > > > -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-05 22:10 ` Xiyue Deng @ 2024-05-08 18:36 ` Xiyue Deng 2024-05-08 18:41 ` Xiyue Deng 1 sibling, 0 replies; 41+ messages in thread From: Xiyue Deng @ 2024-05-08 18:36 UTC (permalink / raw) To: ding Xiyue Deng <manphiz@gmail.com> writes: > Dan Christensen <jdc@uwo.ca> writes: > >> On May 5, 2024, Xiyue Deng <manphiz@gmail.com> wrote: >> >>> So this happened again in one of my outlook accounts' Sent folder, and >>> the "G p" information is as below: >>> >>> ,---- >>> | ((modseq) >>> | (uidvalidity . "11") >>> | (active 1 . 7) >>> | (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent)) >>> `---- >> >> I guess you are having a different problem than I am. I assume that >> you tried `M-g', and the above doesn't change? > > Yes, and it didn't fix it. > >> What if you go into the server buffer with `^' and then hit `C' on the >> server to close it and `O' on the server to open it? >> > > Tried this as well, and unfortunately it didn't fix it either. > Just wonder is there anything else that I can try to debug this? In worst case I can just use the moving-messages-in-and-out workaround to reset the count, but wondering whether there is any other things to try. >> Dan >> >> >> -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-05 22:10 ` Xiyue Deng 2024-05-08 18:36 ` Xiyue Deng @ 2024-05-08 18:41 ` Xiyue Deng 2024-05-09 23:11 ` Andrew Cohen 1 sibling, 1 reply; 41+ messages in thread From: Xiyue Deng @ 2024-05-08 18:41 UTC (permalink / raw) To: ding Xiyue Deng <manphiz@gmail.com> writes: > Dan Christensen <jdc@uwo.ca> writes: > >> On May 5, 2024, Xiyue Deng <manphiz@gmail.com> wrote: >> >>> So this happened again in one of my outlook accounts' Sent folder, and >>> the "G p" information is as below: >>> >>> ,---- >>> | ((modseq) >>> | (uidvalidity . "11") >>> | (active 1 . 7) >>> | (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent)) >>> `---- >> >> I guess you are having a different problem than I am. I assume that >> you tried `M-g', and the above doesn't change? > > Yes, and it didn't fix it. > >> What if you go into the server buffer with `^' and then hit `C' on the >> server to close it and `O' on the server to open it? >> > > Tried this as well, and unfortunately it didn't fix it either. > Just wonder is there anything else that I can try to debug this? In worst case I can just use the moving-messages-in-and-out workaround to reset the count, but wondering whether there is any other things to try and hopefully help identifying the issue from the Gnus side. >> Dan >> >> >> -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-08 18:41 ` Xiyue Deng @ 2024-05-09 23:11 ` Andrew Cohen 2024-05-10 7:45 ` Xiyue Deng 0 siblings, 1 reply; 41+ messages in thread From: Andrew Cohen @ 2024-05-09 23:11 UTC (permalink / raw) To: ding >>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes: XD> Xiyue Deng <manphiz@gmail.com> writes: >> Dan Christensen <jdc@uwo.ca> writes: >> >>> On May 5, 2024, Xiyue Deng <manphiz@gmail.com> wrote: >>> >>>> So this happened again in one of my outlook accounts' Sent >>>> folder, and the "G p" information is as below: >>>> >>>> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 7) | >>>> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft >>>> $MDNSent)) `---- >>> When the count is restored to normal, does "G p" show modseq still without a value? -- Andrew Cohen ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-09 23:11 ` Andrew Cohen @ 2024-05-10 7:45 ` Xiyue Deng 2024-05-10 8:16 ` Andrew Cohen 0 siblings, 1 reply; 41+ messages in thread From: Xiyue Deng @ 2024-05-10 7:45 UTC (permalink / raw) To: ding Andrew Cohen <acohen@ust.hk> writes: >>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes: > > XD> Xiyue Deng <manphiz@gmail.com> writes: > >> Dan Christensen <jdc@uwo.ca> writes: > >> > >>> On May 5, 2024, Xiyue Deng <manphiz@gmail.com> wrote: > >>> > >>>> So this happened again in one of my outlook accounts' Sent > >>>> folder, and the "G p" information is as below: > >>>> > >>>> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 7) | > >>>> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft > >>>> $MDNSent)) `---- > >>> > > When the count is restored to normal, does "G p" show modseq still > without a value? Yes. This is "G p" after I used the move-in-move-out trick to reset the count: ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 12) | (permanent-flags %Seen %Answered %Flagged %Deleted %Draft $MDNSent)) `---- -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-10 7:45 ` Xiyue Deng @ 2024-05-10 8:16 ` Andrew Cohen 2024-05-10 8:26 ` Xiyue Deng 2024-05-10 9:13 ` Andreas Schwab 0 siblings, 2 replies; 41+ messages in thread From: Andrew Cohen @ 2024-05-10 8:16 UTC (permalink / raw) To: ding >>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes: >> When the count is restored to normal, does "G p" show modseq >> still without a value? XD> Yes. This is "G p" after I used the move-in-move-out trick to XD> reset the count: XD> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 12) | XD> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft XD> $MDNSent)) `---- Thanks. It seems that two different cases have been reported: those which are fixed with "M-g" and those (like this case of yours) that are not. I was hoping that the absence of a value for modseq might be a clue as to the difference, but apparently not :( The absence of this value suggests that the server doesn't support QRESYNC. Out of curiosity, what server is this? I think most of the common ones support it. -- Andrew Cohen ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-10 8:16 ` Andrew Cohen @ 2024-05-10 8:26 ` Xiyue Deng 2024-05-10 9:13 ` Andreas Schwab 1 sibling, 0 replies; 41+ messages in thread From: Xiyue Deng @ 2024-05-10 8:26 UTC (permalink / raw) To: ding Andrew Cohen <acohen@ust.hk> writes: >>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes: > > >> When the count is restored to normal, does "G p" show modseq > >> still without a value? > > XD> Yes. This is "G p" after I used the move-in-move-out trick to > XD> reset the count: > > XD> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 12) | > XD> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft > XD> $MDNSent)) `---- > > Thanks. It seems that two different cases have been reported: those > which are fixed with "M-g" and those (like this case of yours) that are > not. I was hoping that the absence of a value for modseq might be a > clue as to the difference, but apparently not :( > > The absence of this value suggests that the server doesn't support > QRESYNC. Out of curiosity, what server is this? I think most of the > common ones support it. This is an outlook account. AAMOF this happened to both of my gmail and outlook accounts, where empty topic showing unread count like this case happens in outlook accounts more often. -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-10 8:16 ` Andrew Cohen 2024-05-10 8:26 ` Xiyue Deng @ 2024-05-10 9:13 ` Andreas Schwab 1 sibling, 0 replies; 41+ messages in thread From: Andreas Schwab @ 2024-05-10 9:13 UTC (permalink / raw) To: Andrew Cohen; +Cc: ding On Mai 10 2024, Andrew Cohen wrote: >>>>>> "XD" == Xiyue Deng <manphiz@gmail.com> writes: > > >> When the count is restored to normal, does "G p" show modseq > >> still without a value? > > XD> Yes. This is "G p" after I used the move-in-move-out trick to > XD> reset the count: > > XD> ,---- | ((modseq) | (uidvalidity . "11") | (active 1 . 12) | > XD> (permanent-flags %Seen %Answered %Flagged %Deleted %Draft > XD> $MDNSent)) `---- > > Thanks. It seems that two different cases have been reported: those > which are fixed with "M-g" and those (like this case of yours) that are > not. I was hoping that the absence of a value for modseq might be a > clue as to the difference, but apparently not :( The modseq value comes from the HIGHESTMODSEQ response (see RFC 4551, 3.1.1). If the HIGHESTMODSEQ response is missing, then the server does not support the persistent storage of mod-sequences. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different." ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-27 19:46 ` Xiyue Deng 2024-05-02 5:47 ` Xiyue Deng @ 2024-05-10 0:43 ` Greg Troxel 2024-05-10 1:20 ` Xiyue Deng 1 sibling, 1 reply; 41+ messages in thread From: Greg Troxel @ 2024-05-10 0:43 UTC (permalink / raw) To: Xiyue Deng; +Cc: ding Xiyue Deng <manphiz@gmail.com> writes: > This makes much sense. I'll try M-g the next time this happens. Thanks > everyone! I had a situation where a message arrived in INBOX. I could see it in the filesystem dovecot reads, and in other clients. It was as if it did not exist in gnus. M-g on the group in *Group* buffer seems to have resolved that. It is interesting that moving out and back resolved what I think is the same situation in the past. Just making up stuff without basis: It's like a missing wakeup in a synchronization protocol. > Also I would expect the equivalent of M-g to happen on startup so that > there are less surprises. Agreed, but I can see the point of not blowing caches just because of startup. ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-10 0:43 ` Greg Troxel @ 2024-05-10 1:20 ` Xiyue Deng 2024-05-10 12:24 ` Greg Troxel 0 siblings, 1 reply; 41+ messages in thread From: Xiyue Deng @ 2024-05-10 1:20 UTC (permalink / raw) To: Greg Troxel; +Cc: ding Hi Greg, Greg Troxel <gdt@lexort.com> writes: > Xiyue Deng <manphiz@gmail.com> writes: > >> This makes much sense. I'll try M-g the next time this happens. Thanks >> everyone! > > I had a situation where a message arrived in INBOX. I could see it in > the filesystem dovecot reads, and in other clients. It was as if it did > not exist in gnus. > > M-g on the group in *Group* buffer seems to have resolved that. > It is interesting that moving out and back resolved what I think is the > same situation in the past. > > Just making up stuff without basis: It's like a missing wakeup in a > synchronization protocol. > > Unfortunately, it looks like it doesn't fix it for me, presumably because I'm using nnimap instead of reading from a local mailbox. Therefore it feels natural to assume a bug in nnimap, though it's still years earlier before I can figure it out from the code, if that ever happens :P >> Also I would expect the equivalent of M-g to happen on startup so that >> there are less surprises. > > Agreed, but I can see the point of not blowing caches just because of startup. -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-10 1:20 ` Xiyue Deng @ 2024-05-10 12:24 ` Greg Troxel 2024-05-10 20:06 ` Xiyue Deng 0 siblings, 1 reply; 41+ messages in thread From: Greg Troxel @ 2024-05-10 12:24 UTC (permalink / raw) To: Xiyue Deng; +Cc: ding Xiyue Deng <manphiz@gmail.com> writes: > Hi Greg, > > Greg Troxel <gdt@lexort.com> writes: > >> Xiyue Deng <manphiz@gmail.com> writes: >> >>> This makes much sense. I'll try M-g the next time this happens. Thanks >>> everyone! >> >> I had a situation where a message arrived in INBOX. I could see it in >> the filesystem dovecot reads, and in other clients. It was as if it did >> not exist in gnus. >> >> M-g on the group in *Group* buffer seems to have resolved that. >> It is interesting that moving out and back resolved what I think is the >> same situation in the past. >> >> Just making up stuff without basis: It's like a missing wakeup in a >> synchronization protocol. > > Unfortunately, it looks like it doesn't fix it for me, presumably > because I'm using nnimap instead of reading from a local mailbox. > Therefore it feels natural to assume a bug in nnimap, though it's still > years earlier before I can figure it out from the code, if that ever > happens :P This was nnimap in my case, with emacs and dovecot on the same machine if that matters. ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-10 12:24 ` Greg Troxel @ 2024-05-10 20:06 ` Xiyue Deng 2024-05-11 19:08 ` Greg Troxel 0 siblings, 1 reply; 41+ messages in thread From: Xiyue Deng @ 2024-05-10 20:06 UTC (permalink / raw) To: Greg Troxel; +Cc: ding Greg Troxel <gdt@lexort.com> writes: > Xiyue Deng <manphiz@gmail.com> writes: > >> Hi Greg, >> >> Greg Troxel <gdt@lexort.com> writes: >> >>> Xiyue Deng <manphiz@gmail.com> writes: >>> >>>> This makes much sense. I'll try M-g the next time this happens. Thanks >>>> everyone! >>> >>> I had a situation where a message arrived in INBOX. I could see it in >>> the filesystem dovecot reads, and in other clients. It was as if it did >>> not exist in gnus. >>> >>> M-g on the group in *Group* buffer seems to have resolved that. >>> It is interesting that moving out and back resolved what I think is the >>> same situation in the past. >>> >>> Just making up stuff without basis: It's like a missing wakeup in a >>> synchronization protocol. >> >> Unfortunately, it looks like it doesn't fix it for me, presumably >> because I'm using nnimap instead of reading from a local mailbox. >> Therefore it feels natural to assume a bug in nnimap, though it's still >> years earlier before I can figure it out from the code, if that ever >> happens :P > > This was nnimap in my case, with emacs and dovecot on the same machine > if that matters. > > Ah I see. Then probably the IMAP implementation of Outlook is just bad ;-) -- Xiyue Deng ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-10 20:06 ` Xiyue Deng @ 2024-05-11 19:08 ` Greg Troxel 2024-05-11 22:33 ` Eric Abrahamsen 0 siblings, 1 reply; 41+ messages in thread From: Greg Troxel @ 2024-05-11 19:08 UTC (permalink / raw) To: Xiyue Deng; +Cc: ding Xiyue Deng <manphiz@gmail.com> writes: >> This was nnimap in my case, with emacs and dovecot on the same machine >> if that matters. > > Ah I see. Then probably the IMAP implementation of Outlook is just bad > ;-) I believe there is a gnus bug that I am hitting, and that others are too. I am entirely willing to believe the the theory that there is also a bug in Outlook IMAP.... ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-05-11 19:08 ` Greg Troxel @ 2024-05-11 22:33 ` Eric Abrahamsen 2024-05-12 11:14 ` Gnus sometimes fails to see a message that is actually in IMAP Greg Troxel 0 siblings, 1 reply; 41+ messages in thread From: Eric Abrahamsen @ 2024-05-11 22:33 UTC (permalink / raw) To: ding Greg Troxel <gdt@lexort.com> writes: > Xiyue Deng <manphiz@gmail.com> writes: > >>> This was nnimap in my case, with emacs and dovecot on the same machine >>> if that matters. >> >> Ah I see. Then probably the IMAP implementation of Outlook is just bad >> ;-) > > I believe there is a gnus bug that I am hitting, and that others are > too. > > I am entirely willing to believe the the theory that there is also a bug > in Outlook IMAP.... I was going to say, the IMAP implementation of Outlook is indeed bad! But that's "good" that you're hitting this issue with Dovecot running locally, as that's a setup with fewer confounding variables, and also what I run, so maybe we stand a better chance of reproducing. Maybe it's time to dive in :( Greg, if you're willing, would you provide as much details as possible about your setup, and anything/everything you can remember about what happened/happens before you see this error? All your nnimap settings are of interest, obviously, but you might as well send anything that could possibly be relevant, off-list if you'd prefer. And if you do see the error crop up again, please let us know what you were doing immediately before, and if at all possible, leave Gnus in the broken state until we can do some remote debugging. Thanks! Eric ^ permalink raw reply [flat|nested] 41+ messages in thread
* Gnus sometimes fails to see a message that is actually in IMAP 2024-05-11 22:33 ` Eric Abrahamsen @ 2024-05-12 11:14 ` Greg Troxel 2024-05-13 14:39 ` Eric Abrahamsen 0 siblings, 1 reply; 41+ messages in thread From: Greg Troxel @ 2024-05-12 11:14 UTC (permalink / raw) To: Eric Abrahamsen; +Cc: ding I have changed the subject, as I am not at all sure that my problem is the same as "reports new messages but not showing them on IMAP server". In my case, it seems to me as if there is a filter between gnus and the IMAP server that redacts a message. Eric Abrahamsen <eric@ericabrahamsen.net> writes: > I was going to say, the IMAP implementation of Outlook is indeed bad! > But that's "good" that you're hitting this issue with Dovecot running > locally, as that's a setup with fewer confounding variables, and also > what I run, so maybe we stand a better chance of reproducing. Yes, it does seem more tractable. One thing I find troublesome about modern gnus is that it is part of emacs. I'm currently running 28.2, so that's the gnus I have. Years before, when gnus was in a separate repo, I was running up-to-date gnus with stable emacs. If I'm confused and we should be debugging gnus trunk please let me know. I probably can update to 29.x without trouble, and I should do that. I try to stay away from bleeding edge including .0, when the older version isn't crufty and isn't troublesome, for things I want to use rather than hack on. I don't really want to run emacs-current. > Maybe it's time to dive in :( Greg, if you're willing, would you provide > as much details as possible about your setup, and anything/everything > you can remember about what happened/happens before you see this error? First, I have no perception of anything unusual or remarkable happening when the problem happens. Mail arrives, apparently just like any other message, I see it in K-9 (Android client that seems very well behaved), and I then go to look for it in gnus (to answer, because I have a keyboard) and find it's not there. This is infrequent, maybe once a month, and there is nothing strange about these messages. They are from people I've heard from before, are not necessarily even large, and "more $filename" in the IMAP dir looks unremarkable. When this happens, gnus feels entirely normal. I can still enter INBOX, read arriving mail, enter with "100 <space>" to see already read mail, etc. If I were only using gnus I would have had no idea anything was wrong, until someone asked me why I didn't respond to some message. The missing messages are in $HOME/IMAP/cur because they've been read by another client. Thunderbird sees the missing messages too. I have previously tried restarting dovecot and removing dovecot caches, but this has never fixed the issue. I am 95% convinced that the is issue is in gnus. My setup is: - computer runs NetBSD 9, postfix receiving mail from the Internet at large, with lots of spam filtering, postfix delivering to $HOME/IMAP as maildir, hence into $HOME/IMAP/new. While I have to mess with spam filtering from time to time, it is not flaky - just minor misclassification - and the messages in question are 1) not spam and 2) delivered to INBOX. - dovecot 2.3.21 serves $HOME/IMAP. Many updates have happened since I first noticed this -- it's been several years I think (but no notes). - dovecot has an x509 cert (that validates under pkix rules), and listens only on the "implicit TLS" port, meaning that it expects TLS negotation right away vs STARTTLS. (I'm firmly in the "cleartext connections should be adminitratively prohibited" camp.) - dovecot's uidvalidity as shown in "G p" is a number that when interpreted as a timeval is noonish EDT on a Sunday in March of 2006. Indeed the IMAP spec says that servers should try not to change that. That's wild, but probably when I installed dovecot version 1 on a different computer. I have upgraded dovecot, upgraded the OS version, and migrated to different hardware with a different hostname since. The point is that this problem is almost certainly not about gnus' IMAP implemention getting confused about uidvalidity changes! - emacs with gnus is left running always, and accessed via tmux. The machine is very stable, basically getting rebooted for software updates. - emacs 28.2 gnus has nnimap configured (setq gnus-secondary-select-methods '((nnimap "foo.example.com" (nnimap-address "bar.example.com") (nnimap-port 993) (nnimap-stream ssl) (nnimap-authenticator login) (nnir-search-engine imap) ) (nntp "news.gwene.org") ) ) where foo is a historical name and bar.example.com is the actual name of the host (with emacs and the imap server both). The connection goes over IPv6, not ::1, but the actual IP address. I would have used localhost, but then the certificate is not valid. (Yes, I know the gwene/gmane world is messy and that it's possible I should be stopping this or doing it differently, but I don't think it is relevant.) - I had vestigial config that I just dropped, to add --insecure to gnutls-cli, for a historical collection of self-signed certs, connecting to localhost, emacs having the wrong path to ssl config, and me not straightening out my copy-of-config-in-linuxy-pathname-for-emacs. In the end, it was just adding "--insecure" to the gnutls-cli invocation. I can't believe that matters, but mentioning it for completeness. > And if you do see the error crop up again, please let us know what you > were doing immediately before, and if at all possible, leave Gnus in the > broken state until we can do some remote debugging. Thanks for offering to spend time on this. I'll write offlist to see about coordinating. It does seem like looking at group properties is useful and I'll do that. I'll at least take better notes. I think but am not 100% sure that the broken state survives exiting/restarting gnus, and exiting/restarting emacs. My previous workaround was move-out move-back. Now, M-g on the group seems to fix things. It would help to understand the big picture of how gnus caches imap state. Ideally that would be in the manual; I'll take another look at that. It would also be interesting to have a gnus-imap-revalidate function that for all cached information, fetches it and compares, complaining loudly if different. I have used this technique when debugging routing code to end up with only correct cached information, and didn't think of any other way to find rare bugs. Greg ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes fails to see a message that is actually in IMAP 2024-05-12 11:14 ` Gnus sometimes fails to see a message that is actually in IMAP Greg Troxel @ 2024-05-13 14:39 ` Eric Abrahamsen 2024-05-24 15:58 ` Eric Abrahamsen 0 siblings, 1 reply; 41+ messages in thread From: Eric Abrahamsen @ 2024-05-13 14:39 UTC (permalink / raw) To: ding Greg Troxel <gdt@lexort.com> writes: Wow, this is a ton of information! Thanks for all the background. Kudos on using TLS on your own machine, something I have never had the patience to set up. > I have changed the subject, as I am not at all sure that my problem is > the same as "reports new messages but not showing them on IMAP server". > In my case, it seems to me as if there is a filter between gnus and the > IMAP server that redacts a message. > > > Eric Abrahamsen <eric@ericabrahamsen.net> writes: > >> I was going to say, the IMAP implementation of Outlook is indeed bad! >> But that's "good" that you're hitting this issue with Dovecot running >> locally, as that's a setup with fewer confounding variables, and also >> what I run, so maybe we stand a better chance of reproducing. > > Yes, it does seem more tractable. > > One thing I find troublesome about modern gnus is that it is part of > emacs. I'm currently running 28.2, so that's the gnus I have. Years > before, when gnus was in a separate repo, I was running up-to-date gnus > with stable emacs. If I'm confused and we should be debugging gnus > trunk please let me know. Not confused, of course, but I do think that Emacs 28 is a little bit too old for this sort of debugging. Ideally we'd be doing it on master, though 29 is still worthwhile, if you can update to that. There are good things in the past couple of versions. [...] > It would also be interesting to have a gnus-imap-revalidate function that > for all cached information, fetches it and compares, complaining loudly > if different. I have used this technique when debugging routing code to > end up with only correct cached information, and didn't think of any > other way to find rare bugs. This is a really good idea. There's been talk on this list of forcibly resetting Gnus to the remote server's state at start-up, which would be something of a band-aid, but this would stand us a better chance of pinning down the bug. People could run it at Gnus startup and shutdown, as well as on-demand. I'll look into putting this together. ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes fails to see a message that is actually in IMAP 2024-05-13 14:39 ` Eric Abrahamsen @ 2024-05-24 15:58 ` Eric Abrahamsen 2024-05-24 17:29 ` Eric Abrahamsen 0 siblings, 1 reply; 41+ messages in thread From: Eric Abrahamsen @ 2024-05-24 15:58 UTC (permalink / raw) To: ding; +Cc: Greg Troxel, Andrew Cohen Eric Abrahamsen <eric@ericabrahamsen.net> writes: > Greg Troxel <gdt@lexort.com> writes: > > Wow, this is a ton of information! Thanks for all the background. Kudos > on using TLS on your own machine, something I have never had the > patience to set up. > >> I have changed the subject, as I am not at all sure that my problem is >> the same as "reports new messages but not showing them on IMAP server". >> In my case, it seems to me as if there is a filter between gnus and the >> IMAP server that redacts a message. Brief update -- when I started looking further into this, I realized that I have the same issue with the INBOX of one of my main IMAP accounts. I'd just not noticed because I have 486 unread messages in that folder :( Well, 483 according to Gnus. This account includes a remote Dovecot on the server that actually receives mail, and two local machines that each have their own local Dovecot, and Emacs/Gnus. The remote and local Dovecots are synced with mbsync, and both remote and local report 486 unread messages when checked with doveadm. My other local machine is currently not booting, but I'll be very curious to see if Gnus on that machine is wrong in the same way (a deterministic bug) or not (space gremlins). Otherwise, the difference between "g" and "M-g", for IMAP servers that support qresync, is that "g" goes the qresync route: EXAMINE <group> (QRESYNC (<uidvalidity> <modseq>)) while "M-g" does a full fetch of flags: "UID FETCH 1:* FLAGS" So that's a pretty strong indicator that there's a bug in the qresync branch of `nnimap-update-qresync-info'. Some have reported bugs that *weren't* resolved by "M-g", so there will likely be a few things to fix, but one thing at a time. I'd originally planned to provide a function people could run after updating, to see if the numbers had gone out of whack, but now I think I can do a bit better; maybe a tweaked version of `nnimap-update-qresync-info' that would do a bit of its own error-checking. Eric ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes fails to see a message that is actually in IMAP 2024-05-24 15:58 ` Eric Abrahamsen @ 2024-05-24 17:29 ` Eric Abrahamsen 0 siblings, 0 replies; 41+ messages in thread From: Eric Abrahamsen @ 2024-05-24 17:29 UTC (permalink / raw) To: ding; +Cc: Greg Troxel, Andrew Cohen Eric Abrahamsen <eric@ericabrahamsen.net> writes: > Eric Abrahamsen <eric@ericabrahamsen.net> writes: > >> Greg Troxel <gdt@lexort.com> writes: >> >> Wow, this is a ton of information! Thanks for all the background. Kudos >> on using TLS on your own machine, something I have never had the >> patience to set up. >> >>> I have changed the subject, as I am not at all sure that my problem is >>> the same as "reports new messages but not showing them on IMAP server". >>> In my case, it seems to me as if there is a filter between gnus and the >>> IMAP server that redacts a message. > > Brief update -- when I started looking further into this, I realized > that I have the same issue with the INBOX of one of my main IMAP > accounts. I'd just not noticed because I have 486 unread messages in > that folder :( Well, 483 according to Gnus. > > This account includes a remote Dovecot on the server that actually > receives mail, and two local machines that each have their own local > Dovecot, and Emacs/Gnus. The remote and local Dovecots are synced with > mbsync, and both remote and local report 486 unread messages when > checked with doveadm. > > My other local machine is currently not booting, but I'll be very > curious to see if Gnus on that machine is wrong in the same way > (a deterministic bug) or not (space gremlins). > > Otherwise, the difference between "g" and "M-g", for IMAP servers that > support qresync, is that "g" goes the qresync route: > > EXAMINE <group> (QRESYNC (<uidvalidity> <modseq>)) > > while "M-g" does a full fetch of flags: > > "UID FETCH 1:* FLAGS" > > So that's a pretty strong indicator that there's a bug in the qresync > branch of `nnimap-update-qresync-info'. Some have reported bugs that > *weren't* resolved by "M-g", so there will likely be a few things to > fix, but one thing at a time. I'm currently looking with deep suspicion at the very beginning of that function: ;; Add all the vanished articles to the list of read articles. (setf (gnus-info-read info) (range-add-list (range-add-list (range-concat (gnus-info-read info) vanished) (cdr (assq '%Flagged flags))) (cdr (assq '%Seen flags)))) ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-21 16:18 ` Eric Abrahamsen 2024-04-21 18:16 ` Arash Esbati @ 2024-04-24 15:23 ` James Thomas 2024-04-25 2:59 ` Eric Abrahamsen 1 sibling, 1 reply; 41+ messages in thread From: James Thomas @ 2024-04-24 15:23 UTC (permalink / raw) To: Eric Abrahamsen; +Cc: ding Eric Abrahamsen wrote: > I've focused on figuring out why the active number goes wrong On that note, here's something I'd been meaning to debug myself sometime: - (Preferably with an empty drafts folder) Compose a message and save it - Open the drafts folder, press e on the message and then kill the new buffer; then (incidentally, if you do '/ N' then this bug does not arise) delete the message (B DEL) - Press q - The message count is wrong I don't know if this is related to the above. Btw could you share it if there's a bug report for the former? -- ^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: Gnus sometimes reports new messages but not showing them on IMAP server 2024-04-24 15:23 ` Gnus sometimes reports new messages but not showing them on IMAP server James Thomas @ 2024-04-25 2:59 ` Eric Abrahamsen 0 siblings, 0 replies; 41+ messages in thread From: Eric Abrahamsen @ 2024-04-25 2:59 UTC (permalink / raw) To: ding James Thomas <jimjoe@gmx.net> writes: > Eric Abrahamsen wrote: > >> I've focused on figuring out why the active number goes wrong > > On that note, here's something I'd been meaning to debug myself > sometime: > > - (Preferably with an empty drafts folder) Compose a message and save it > - Open the drafts folder, press e on the message and then kill the new > buffer; then (incidentally, if you do '/ N' then this bug does not > arise) delete the message (B DEL) > - Press q > - The message count is wrong That's a new one for me, but I was able to reproduce it. Would you open a bug report with that recipe? > I don't know if this is related to the above. Btw could you share it if > there's a bug report for the former? I don't think there is, mostly because it's kind of a vague problem. I wouldn't be surprised if it ended up being several things at once. ^ permalink raw reply [flat|nested] 41+ messages in thread
end of thread, other threads:[~2024-05-24 17:30 UTC | newest] Thread overview: 41+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-04-12 0:57 Gnus sometimes reports new messages but not showing them on IMAP server Xiyue Deng 2024-04-17 0:03 ` Greg Troxel 2024-04-17 6:47 ` Xiyue Deng 2024-04-17 19:51 ` Dan Christensen 2024-04-18 12:13 ` Greg Troxel 2024-04-19 22:55 ` Dan Christensen 2024-04-21 12:08 ` Arash Esbati 2024-04-21 12:43 ` Dan Christensen 2024-04-21 13:14 ` Andreas Schwab 2024-04-21 16:18 ` Eric Abrahamsen 2024-04-21 18:16 ` Arash Esbati 2024-04-22 0:00 ` Greg Troxel 2024-04-22 2:35 ` Eric Abrahamsen 2024-04-27 19:46 ` Xiyue Deng 2024-05-02 5:47 ` Xiyue Deng 2024-05-02 15:08 ` Eric Abrahamsen 2024-05-02 17:24 ` Xiyue Deng 2024-05-02 17:41 ` Dan Christensen 2024-05-02 18:41 ` Xiyue Deng 2024-05-05 20:18 ` Xiyue Deng 2024-05-05 21:01 ` Dan Christensen 2024-05-05 22:10 ` Xiyue Deng 2024-05-08 18:36 ` Xiyue Deng 2024-05-08 18:41 ` Xiyue Deng 2024-05-09 23:11 ` Andrew Cohen 2024-05-10 7:45 ` Xiyue Deng 2024-05-10 8:16 ` Andrew Cohen 2024-05-10 8:26 ` Xiyue Deng 2024-05-10 9:13 ` Andreas Schwab 2024-05-10 0:43 ` Greg Troxel 2024-05-10 1:20 ` Xiyue Deng 2024-05-10 12:24 ` Greg Troxel 2024-05-10 20:06 ` Xiyue Deng 2024-05-11 19:08 ` Greg Troxel 2024-05-11 22:33 ` Eric Abrahamsen 2024-05-12 11:14 ` Gnus sometimes fails to see a message that is actually in IMAP Greg Troxel 2024-05-13 14:39 ` Eric Abrahamsen 2024-05-24 15:58 ` Eric Abrahamsen 2024-05-24 17:29 ` Eric Abrahamsen 2024-04-24 15:23 ` Gnus sometimes reports new messages but not showing them on IMAP server James Thomas 2024-04-25 2:59 ` Eric Abrahamsen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).