* Debugging extremely long IMAP folder exit times @ 2019-07-18 18:34 Jason L Tibbitts III 2019-07-18 20:31 ` Andreas Schwab 0 siblings, 1 reply; 10+ messages in thread From: Jason L Tibbitts III @ 2019-07-18 18:34 UTC (permalink / raw) To: ding Occasionally a folder on an IMAP server will get into a state where it takes ages for gnus to leave it. I have one folder which before today was quick to exit, but today suddenly takes over two minutes. Gnus shows "Expiring articles..." in the minibuffer. If I C-u RET to enter the group, gnus thinks there are over 364000 messages in the group. There are actually only about 9000 messages there, according to another IMAP client. My guess is that gnus is just iterating over 364k entries to see what it should expire. Note that when I run gnus-group-expire-articles on the group from the summary buffer, it doesn't take very long at all. Here is what's in newsrc.eld for the group. I was worried there would be a huge set of ranges in there but the data is quite simple. ("fail2ban" 3 ((1 . 364570)) ((unexist) (seen (364431 . 364570))) nil ((modseq) (uidvalidity . "1535374911") (active 1 . 364570) (permanent-flags %Answered %Flagged %Draft %Deleted %Seen %*) total-expire (expiry-wait . 7))) Any hints on understanding what gnus is doing here, or suggestions on how to speed it up? I used to know quite a bit about Emacs and Gnus internals but it has been many years now since I tried to do anything interesting. I am running emacs 26.2; gnus-version says "Gnus v5.13". Thanks, - J< ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Debugging extremely long IMAP folder exit times 2019-07-18 18:34 Debugging extremely long IMAP folder exit times Jason L Tibbitts III @ 2019-07-18 20:31 ` Andreas Schwab 2019-07-18 22:15 ` Jason L Tibbitts III 0 siblings, 1 reply; 10+ messages in thread From: Andreas Schwab @ 2019-07-18 20:31 UTC (permalink / raw) To: Jason L Tibbitts III; +Cc: ding On Jul 18 2019, Jason L Tibbitts III <tibbs@math.uh.edu> wrote: > Here is what's in newsrc.eld for the group. I was worried there would > be a huge set of ranges in there but the data is quite simple. > > ("fail2ban" 3 ((1 . 364570)) ((unexist) (seen (364431 . 364570))) nil This is your problem. Since the unexist list is empty, gnus doesn't know that the articles 1 ... 354000 (approx.) don't exist. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different." ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Debugging extremely long IMAP folder exit times 2019-07-18 20:31 ` Andreas Schwab @ 2019-07-18 22:15 ` Jason L Tibbitts III 2019-07-22 17:08 ` Jason L Tibbitts III 0 siblings, 1 reply; 10+ messages in thread From: Jason L Tibbitts III @ 2019-07-18 22:15 UTC (permalink / raw) To: Andreas Schwab; +Cc: ding >>>>> "AS" == Andreas Schwab <schwab@linux-m68k.org> writes: AS> This is your problem. Since the unexist list is empty, gnus doesn't AS> know that the articles 1 ... 354000 (approx.) don't exist. OK, so... I didn't do anything specific to make that happen. How would I fix it? I note that after I sent my message, exiting that group/folder somehow became quite fast for a while. But I quit gnus and closed emacs, and after restarting it again took minutes to exit the group. - J< ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Debugging extremely long IMAP folder exit times 2019-07-18 22:15 ` Jason L Tibbitts III @ 2019-07-22 17:08 ` Jason L Tibbitts III 2019-07-22 17:54 ` Eric Abrahamsen 0 siblings, 1 reply; 10+ messages in thread From: Jason L Tibbitts III @ 2019-07-22 17:08 UTC (permalink / raw) To: ding To answer my own question, using M-c on the folder (from the Group buffer) was sufficient to get things back into a sane state. Fortunately I had no marks or anything in that folder to lose. If I had, I suppose the best option would have been to move the ticket messages to another folder first. - J< ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Debugging extremely long IMAP folder exit times 2019-07-22 17:08 ` Jason L Tibbitts III @ 2019-07-22 17:54 ` Eric Abrahamsen 2019-07-22 18:44 ` Jason L Tibbitts III 0 siblings, 1 reply; 10+ messages in thread From: Eric Abrahamsen @ 2019-07-22 17:54 UTC (permalink / raw) To: ding Jason L Tibbitts III <tibbs@math.uh.edu> writes: > To answer my own question, using M-c on the folder (from the Group > buffer) was sufficient to get things back into a sane state. > Fortunately I had no marks or anything in that folder to lose. If I > had, I suppose the best option would have been to move the ticket > messages to another folder first. Ticked is equivalent to IMAP flagged, so that data would also be preserved in the server, and wouldn't need to be backed up (I think). I was just looking to see if there's a "unilaterally reset group marks from IMAP server flags" command, but at first glance there doesn't appear to be. That would be a nice thing to have. In theory, Gnus doesn't really need to keep marks that are redundant to server flags in the first place, but that would take some tweaking. It would also be nice to know how your 'unexist flags got out of whack to begin with. Much to do. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Debugging extremely long IMAP folder exit times 2019-07-22 17:54 ` Eric Abrahamsen @ 2019-07-22 18:44 ` Jason L Tibbitts III 2019-07-22 23:25 ` Eric Abrahamsen 0 siblings, 1 reply; 10+ messages in thread From: Jason L Tibbitts III @ 2019-07-22 18:44 UTC (permalink / raw) To: Eric Abrahamsen; +Cc: ding >>>>> "EA" == Eric Abrahamsen <eric@ericabrahamsen.net> writes: EA> Ticked is equivalent to IMAP flagged, so that data would also be EA> preserved in the server, and wouldn't need to be backed up (I EA> think). I would think so, but when I did this Gnus lost track of which messages I had read and (in IMAP at least) that's also server-side state. So I really have no idea what might be preserved after a cal to gnus-group-clear-data. The documentation does imply that it clears all marks, so I figured it actively cleared everything on the server as well. EA> In theory, Gnus doesn't really need to keep marks that are EA> redundant to server flags in the first place, but that would take EA> some tweaking. Yes, I don't think the original design anticipated having another entity which could keep track of all mailbox state. As an aside, I wonder how difficult it would be to support what will be rfc8621 (JMAP-Mail). It's certainly a good bit simpler than IMAP in some ways. The fact that it supports sending mail (just have a message stored somewhere on the server and provide a request to send it along with some metadata) is a rather different paradigm from the current way it's all set up. My guess it that it needs quite a bit of work by people smarter/better at elisp than I am but maybe someone's already playing with it. EA> It would also be nice to know how your 'unexist flags got out of EA> whack to begin with. I honestly have no idea. I do generally have more than one IMAP client connected to the server at once, but I was only using one of them within a few hours of this one folder becoming slow. This also isn't the first time I've had the issue, but it's the first time in a few months. - J< ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Debugging extremely long IMAP folder exit times 2019-07-22 18:44 ` Jason L Tibbitts III @ 2019-07-22 23:25 ` Eric Abrahamsen 2019-07-24 23:54 ` Jason L Tibbitts III 0 siblings, 1 reply; 10+ messages in thread From: Eric Abrahamsen @ 2019-07-22 23:25 UTC (permalink / raw) To: ding Jason L Tibbitts III <tibbs@math.uh.edu> writes: >>>>>> "EA" == Eric Abrahamsen <eric@ericabrahamsen.net> writes: > > EA> Ticked is equivalent to IMAP flagged, so that data would also be > EA> preserved in the server, and wouldn't need to be backed up (I > EA> think). > > I would think so, but when I did this Gnus lost track of which messages > I had read and (in IMAP at least) that's also server-side state. So I > really have no idea what might be preserved after a cal to > gnus-group-clear-data. The documentation does imply that it clears all > marks, so I figured it actively cleared everything on the server as > well. Oh, I guess you're right, it removes marks from the backend as well. In the past I think I've "dealt with" this by shutting down Gnus, going into my newsrc.eld, completely removing the entry for the imap group, and then restarting Gnus and re-subscribing to the group. > EA> In theory, Gnus doesn't really need to keep marks that are > EA> redundant to server flags in the first place, but that would take > EA> some tweaking. > > Yes, I don't think the original design anticipated having another entity > which could keep track of all mailbox state. Nope, but it would be nice if it could deal with that. > As an aside, I wonder how difficult it would be to support what will be > rfc8621 (JMAP-Mail). It's certainly a good bit simpler than IMAP in > some ways. The fact that it supports sending mail (just have a message > stored somewhere on the server and provide a request to send it along > with some metadata) is a rather different paradigm from the current way > it's all set up. > > My guess it that it needs quite a bit of work by people smarter/better > at elisp than I am but maybe someone's already playing with it. I don't think anyone is, but it's definitely been in the back of my mind when thinking about refactoring Gnus' backends. It would be fun to have Gnus be the first client outside of Fastmail's own software to support JMAP. It looks like they've just published the first RFC for the storage part of JMAP, but not yet the mail-related part. I've thought about creating light connections between messages under composition and Gnus servers, so that the server itself could have a chance to handle the message -- that would be useful for per-server message drafts, but could also handle server's like JMAP that do the sending themselves. > EA> It would also be nice to know how your 'unexist flags got out of > EA> whack to begin with. > > I honestly have no idea. I do generally have more than one IMAP client > connected to the server at once, but I was only using one of them within > a few hours of this one folder becoming slow. This also isn't the first > time I've had the issue, but it's the first time in a few months. Who knows... I access imap accounts from two Gnus installations and one k9 android app, and nothing's ever gone wonky. Bit of a mystery. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Debugging extremely long IMAP folder exit times 2019-07-22 23:25 ` Eric Abrahamsen @ 2019-07-24 23:54 ` Jason L Tibbitts III 2019-07-25 20:09 ` Dan Christensen 0 siblings, 1 reply; 10+ messages in thread From: Jason L Tibbitts III @ 2019-07-24 23:54 UTC (permalink / raw) To: ding I think this is another instance of the same issue. Fortunately the count is small enough that it doesn't take all that long to exit the group. ("fedora-package-review" 3 ((1 . 146459)) ((unexist) (seen (140473 . 145786) (145837 . 146143) (146170 . 146459)) (killed (146173 . 146174) (146176 . 146188) (146190 . 146203) (146205 . 146228) (146231 . 146288) (146291 . 146292) (146294 . 146298) (146300 . 146321) 146323 (146325 . 146326) (146328 . 146341) (146343 . 146355) (146357 . 146367) 146369 (146371 . 146384) (146386 . 146389) 146391 (146393 . 146394) (146397 . 146414) 146417 (146419 . 146433) (146435 . 146459))) nil ((modseq) (uidvalidity . "1422948496") (active 1 . 146459) (permanent-flags %Answered %Flagged %Draft %Deleted %Seen %*) total-expire (expiry-wait . 3))) I guess I'll clear the data on that one as well. - J< ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Debugging extremely long IMAP folder exit times 2019-07-24 23:54 ` Jason L Tibbitts III @ 2019-07-25 20:09 ` Dan Christensen 2019-07-25 20:27 ` Jason L Tibbitts III 0 siblings, 1 reply; 10+ messages in thread From: Dan Christensen @ 2019-07-25 20:09 UTC (permalink / raw) To: ding On Jul 24, 2019, Jason L Tibbitts III <tibbs@math.uh.edu> wrote: > I think this is another instance of the same issue. Fortunately the > count is small enough that it doesn't take all that long to exit the group. ... > I guess I'll clear the data on that one as well. If you haven't yet cleared the data, it would be interesting to see if there are fixes that don't clear the remote marks and flags (even though you don't seem to have any in this case). 1) Does `M-g' on the group fix the problem? (I often have to do that on IMAP groups to make Gnus fully re-read the server info.) 2) What if you `C-k' the group, exit Gnus, restart emacs and gnus, and then subscribe to the group again? Dan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Debugging extremely long IMAP folder exit times 2019-07-25 20:09 ` Dan Christensen @ 2019-07-25 20:27 ` Jason L Tibbitts III 0 siblings, 0 replies; 10+ messages in thread From: Jason L Tibbitts III @ 2019-07-25 20:27 UTC (permalink / raw) To: ding >>>>> "DC" == Dan Christensen <jdc@uwo.ca> writes: DC> If you haven't yet cleared the data, it would be interesting to see DC> if there are fixes that don't clear the remote marks and flags (even DC> though you don't seem to have any in this case). I haven't yet, but somehow the problem has fixed itself in the meantime. Just checked .newsrc.eld and the unexist info is filled in. So when I have this problem again, I will go ahead and try both of your suggestions and see if they help. - J< ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2019-07-25 20:27 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-07-18 18:34 Debugging extremely long IMAP folder exit times Jason L Tibbitts III 2019-07-18 20:31 ` Andreas Schwab 2019-07-18 22:15 ` Jason L Tibbitts III 2019-07-22 17:08 ` Jason L Tibbitts III 2019-07-22 17:54 ` Eric Abrahamsen 2019-07-22 18:44 ` Jason L Tibbitts III 2019-07-22 23:25 ` Eric Abrahamsen 2019-07-24 23:54 ` Jason L Tibbitts III 2019-07-25 20:09 ` Dan Christensen 2019-07-25 20:27 ` Jason L Tibbitts III
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).