From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-2.1 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL autolearn=ham autolearn_force=no version=3.4.4 Received: from mx1.math.uh.edu (mx1.math.uh.edu [129.7.128.32]) by inbox.vuxu.org (Postfix) with ESMTP id 999E022825 for ; Sun, 12 May 2024 13:15:09 +0200 (CEST) Received: from lists1.math.uh.edu ([129.7.128.208]) by mx1.math.uh.edu with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1s67Aa-00000006iYt-1nFm for ml@inbox.vuxu.org; Sun, 12 May 2024 06:15:08 -0500 Received: from lists1.math.uh.edu ([127.0.0.1] helo=lists.math.uh.edu) by lists1.math.uh.edu with smtp (Exim 4.97.1) (envelope-from ) id 1s67Aa-00000004Psf-0eSD for ml@inbox.vuxu.org; Sun, 12 May 2024 06:15:08 -0500 Received: from mx1.math.uh.edu ([129.7.128.32]) by lists1.math.uh.edu with esmtp (Exim 4.97.1) (envelope-from ) id 1s67AW-00000004PsW-3SjM for ding@lists.math.uh.edu; Sun, 12 May 2024 06:15:05 -0500 Received: from quimby.gnus.org ([95.216.78.240]) by mx1.math.uh.edu with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1s67AS-00000006iY8-30Ys for ding@lists.math.uh.edu; Sun, 12 May 2024 06:15:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Content-Type:MIME-Version:Message-ID:Date:References:Subject: Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=d/5LJZ9E/rjIRgAIPHVdbnUu15R9QHPVW/fVGWreP1E=; b=qR6ZCelRXekiAOymNCSu/oucfN NrBUfjZ+kk+T6PUvkMwHM7yMBgs2+gNu+xAx3lorXkvSda7O05S2Zp3/WCPb0y8deNESMb76CMSLD V0N/HnENdjcvHvFCQTne1qmEbq2PWqEzDzfm+D2Xx8QIpaI6NrE6i5Nitj1YoAeAHjoc=; Received: from s1.lexort.com ([71.19.148.97]) by quimby.gnus.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1s67AK-00042O-HW for ding@gnus.org; Sun, 12 May 2024 13:14:55 +0200 Received: by s1.lexort.com (Postfix, from userid 10853) id EF0834106FC; Sun, 12 May 2024 07:14:48 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lexort.com; s=mail; t=1715512488; bh=e2L+YauQyuqwePLYTySj0FeYwu25LrtIKgnzd+JBwH0=; h=From:To:Cc:Subject:References:Date; b=UXWtWzy4d0CXlWwuRbi4CRtOdRNvWxRikQ+/Zrslt8/+0A61e4ogdO/gPIj/beB4C aemX6XNJMIpQZUbEjLC26ua6YmDmGuCFYqQm7XCXcSHtgjVcEO7xN8z+ZRKRDHMyE2 TKK9DIn8uKPbxJJ3t26I7Kzt/2KxPxKl8M8AGZvE= From: Greg Troxel To: Eric Abrahamsen Cc: ding@gnus.org Subject: Gnus sometimes fails to see a message that is actually in IMAP References: <87h6g7mm8r.fsf@debian-hx90.lan> <878r1blqcg.fsf@uwo.ca> <87ttjx55ey.fsf@uwo.ca> <875xwalmf4.fsf@ericabrahamsen.net> <87wmoqjf92.fsf@ericabrahamsen.net> <874jbmk2ql.fsf@debian-hx90.lan> <87msoyfoo4.fsf@debian-hx90.lan> <87plttctyy.fsf@debian-hx90.lan> <87msow9dwx.fsf@ericabrahamsen.net> OpenPGP: id=098ED60E Date: Sun, 12 May 2024 07:14:48 -0400 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain List-ID: Precedence: bulk I have changed the subject, as I am not at all sure that my problem is the same as "reports new messages but not showing them on IMAP server". In my case, it seems to me as if there is a filter between gnus and the IMAP server that redacts a message. Eric Abrahamsen writes: > I was going to say, the IMAP implementation of Outlook is indeed bad! > But that's "good" that you're hitting this issue with Dovecot running > locally, as that's a setup with fewer confounding variables, and also > what I run, so maybe we stand a better chance of reproducing. Yes, it does seem more tractable. One thing I find troublesome about modern gnus is that it is part of emacs. I'm currently running 28.2, so that's the gnus I have. Years before, when gnus was in a separate repo, I was running up-to-date gnus with stable emacs. If I'm confused and we should be debugging gnus trunk please let me know. I probably can update to 29.x without trouble, and I should do that. I try to stay away from bleeding edge including .0, when the older version isn't crufty and isn't troublesome, for things I want to use rather than hack on. I don't really want to run emacs-current. > Maybe it's time to dive in :( Greg, if you're willing, would you provide > as much details as possible about your setup, and anything/everything > you can remember about what happened/happens before you see this error? First, I have no perception of anything unusual or remarkable happening when the problem happens. Mail arrives, apparently just like any other message, I see it in K-9 (Android client that seems very well behaved), and I then go to look for it in gnus (to answer, because I have a keyboard) and find it's not there. This is infrequent, maybe once a month, and there is nothing strange about these messages. They are from people I've heard from before, are not necessarily even large, and "more $filename" in the IMAP dir looks unremarkable. When this happens, gnus feels entirely normal. I can still enter INBOX, read arriving mail, enter with "100 " to see already read mail, etc. If I were only using gnus I would have had no idea anything was wrong, until someone asked me why I didn't respond to some message. The missing messages are in $HOME/IMAP/cur because they've been read by another client. Thunderbird sees the missing messages too. I have previously tried restarting dovecot and removing dovecot caches, but this has never fixed the issue. I am 95% convinced that the is issue is in gnus. My setup is: - computer runs NetBSD 9, postfix receiving mail from the Internet at large, with lots of spam filtering, postfix delivering to $HOME/IMAP as maildir, hence into $HOME/IMAP/new. While I have to mess with spam filtering from time to time, it is not flaky - just minor misclassification - and the messages in question are 1) not spam and 2) delivered to INBOX. - dovecot 2.3.21 serves $HOME/IMAP. Many updates have happened since I first noticed this -- it's been several years I think (but no notes). - dovecot has an x509 cert (that validates under pkix rules), and listens only on the "implicit TLS" port, meaning that it expects TLS negotation right away vs STARTTLS. (I'm firmly in the "cleartext connections should be adminitratively prohibited" camp.) - dovecot's uidvalidity as shown in "G p" is a number that when interpreted as a timeval is noonish EDT on a Sunday in March of 2006. Indeed the IMAP spec says that servers should try not to change that. That's wild, but probably when I installed dovecot version 1 on a different computer. I have upgraded dovecot, upgraded the OS version, and migrated to different hardware with a different hostname since. The point is that this problem is almost certainly not about gnus' IMAP implemention getting confused about uidvalidity changes! - emacs with gnus is left running always, and accessed via tmux. The machine is very stable, basically getting rebooted for software updates. - emacs 28.2 gnus has nnimap configured (setq gnus-secondary-select-methods '((nnimap "foo.example.com" (nnimap-address "bar.example.com") (nnimap-port 993) (nnimap-stream ssl) (nnimap-authenticator login) (nnir-search-engine imap) ) (nntp "news.gwene.org") ) ) where foo is a historical name and bar.example.com is the actual name of the host (with emacs and the imap server both). The connection goes over IPv6, not ::1, but the actual IP address. I would have used localhost, but then the certificate is not valid. (Yes, I know the gwene/gmane world is messy and that it's possible I should be stopping this or doing it differently, but I don't think it is relevant.) - I had vestigial config that I just dropped, to add --insecure to gnutls-cli, for a historical collection of self-signed certs, connecting to localhost, emacs having the wrong path to ssl config, and me not straightening out my copy-of-config-in-linuxy-pathname-for-emacs. In the end, it was just adding "--insecure" to the gnutls-cli invocation. I can't believe that matters, but mentioning it for completeness. > And if you do see the error crop up again, please let us know what you > were doing immediately before, and if at all possible, leave Gnus in the > broken state until we can do some remote debugging. Thanks for offering to spend time on this. I'll write offlist to see about coordinating. It does seem like looking at group properties is useful and I'll do that. I'll at least take better notes. I think but am not 100% sure that the broken state survives exiting/restarting gnus, and exiting/restarting emacs. My previous workaround was move-out move-back. Now, M-g on the group seems to fix things. It would help to understand the big picture of how gnus caches imap state. Ideally that would be in the manual; I'll take another look at that. It would also be interesting to have a gnus-imap-revalidate function that for all cached information, fetches it and compares, complaining loudly if different. I have used this technique when debugging routing code to end up with only correct cached information, and didn't think of any other way to find rare bugs. Greg