Gnus sometimes fails to see a message that is actually in IMAP

[Greg Troxel <gdt@lexort.com>]

I have changed the subject, as I am not at all sure that my problem is
the same as "reports new messages but not showing them on IMAP server".
In my case, it seems to me as if there is a filter between gnus and the
IMAP server that redacts a message.


Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I was going to say, the IMAP implementation of Outlook is indeed bad!
> But that's "good" that you're hitting this issue with Dovecot running
> locally, as that's a setup with fewer confounding variables, and also
> what I run, so maybe we stand a better chance of reproducing.

Yes, it does seem more tractable.

One thing I find troublesome about modern gnus is that it is part of
emacs.  I'm currently running 28.2, so that's the gnus I have.  Years
before, when gnus was in a separate repo, I was running up-to-date gnus
with stable emacs.   If I'm confused and we should be debugging gnus
trunk please let me know.

I probably can update to 29.x without trouble, and I should do that.  I
try to stay away from bleeding edge including .0, when the older version
isn't crufty and isn't troublesome, for things I want to use rather than
hack on.  I don't really want to run emacs-current.

> Maybe it's time to dive in :( Greg, if you're willing, would you provide
> as much details as possible about your setup, and anything/everything
> you can remember about what happened/happens before you see this error?

First, I have no perception of anything unusual or remarkable happening
when the problem happens.  Mail arrives, apparently just like any other
message, I see it in K-9 (Android client that seems very well behaved),
and I then go to look for it in gnus (to answer, because I have a
keyboard) and find it's not there.  This is infrequent, maybe once a
month, and there is nothing strange about these messages.  They are from
people I've heard from before, are not necessarily even large, and "more
$filename" in the IMAP dir looks unremarkable.

When this happens, gnus feels entirely normal.  I can still enter INBOX,
read arriving mail, enter with "100 <space>" to see already read mail,
etc.  If I were only using gnus I would have had no idea anything was
wrong, until someone asked me why I didn't respond to some message.

The missing messages are in $HOME/IMAP/cur because they've been read by
another client.  Thunderbird sees the missing messages too.

I have previously tried restarting dovecot and removing dovecot caches,
but this has never fixed the issue.  I am 95% convinced that the is
issue is in gnus.

My setup is:

  - computer runs NetBSD 9, postfix receiving mail from the Internet at
    large, with lots of spam filtering, postfix delivering to $HOME/IMAP
    as maildir, hence into $HOME/IMAP/new.  While I have to mess with
    spam filtering from time to time, it is not flaky - just minor
    misclassification - and the messages in question are 1) not spam and
    2) delivered to INBOX.

  - dovecot 2.3.21 serves $HOME/IMAP.  Many updates have happened since
    I first noticed this -- it's been several years I think (but no
    notes).

  - dovecot has an x509 cert (that validates under pkix rules), and
    listens only on the "implicit TLS" port, meaning that it expects TLS
    negotation right away vs STARTTLS.  (I'm firmly in the "cleartext
    connections should be adminitratively prohibited" camp.)

  - dovecot's uidvalidity as shown in "G p" is a number that when
    interpreted as a timeval is noonish EDT on a Sunday in March of
    2006.  Indeed the IMAP spec says that servers should try not to
    change that.  That's wild, but probably when I installed dovecot
    version 1 on a different computer.  I have upgraded dovecot,
    upgraded the OS version, and migrated to different hardware with a
    different hostname since.  The point is that this problem is almost
    certainly not about gnus' IMAP implemention getting confused about
    uidvalidity changes!

  - emacs with gnus is left running always, and accessed via tmux.  The
    machine is very stable, basically getting rebooted for software
    updates.

  - emacs 28.2 gnus has nnimap configured

    (setq gnus-secondary-select-methods
          '((nnimap "foo.example.com"
                    (nnimap-address "bar.example.com")
                    (nnimap-port 993)
                    (nnimap-stream ssl)
                    (nnimap-authenticator login)
                    (nnir-search-engine imap)
                    )
            (nntp "news.gwene.org")
            )
          )
   where foo is a historical name and bar.example.com is the actual name
   of the host (with emacs and the imap server both).

   The connection goes over IPv6, not ::1, but the actual IP address.
   I would have used localhost, but then the certificate is not valid.

   (Yes, I know the gwene/gmane world is messy and that it's possible I
   should be stopping this or doing it differently, but I don't think it
   is relevant.)

  - I had vestigial config that I just dropped, to add --insecure to
    gnutls-cli, for a historical collection of self-signed certs,
    connecting to localhost, emacs having the wrong path to ssl config,
    and me not straightening out my
    copy-of-config-in-linuxy-pathname-for-emacs.  In the end, it was
    just adding "--insecure" to the gnutls-cli invocation.  I can't
    believe that matters, but mentioning it for completeness.

> And if you do see the error crop up again, please let us know what you
> were doing immediately before, and if at all possible, leave Gnus in the
> broken state until we can do some remote debugging.

Thanks for offering to spend time on this.  I'll write offlist to see
about coordinating.

It does seem like looking at group properties is useful and I'll do that.

I'll at least take better notes.  I think but am not 100% sure that the
broken state survives exiting/restarting gnus, and exiting/restarting
emacs.

My previous workaround was move-out move-back.
Now, M-g on the group seems to fix things.

It would help to understand the big picture of how gnus caches imap
state.  Ideally that would be in the manual; I'll take another look at
that.

It would also be interesting to have a gnus-imap-revalidate function that
for all cached information, fetches it and compares, complaining loudly
if different.  I have used this technique when debugging routing code to
end up with only correct cached information, and didn't think of any
other way to find rare bugs.

Greg