Debugging extremely long IMAP folder exit times

Debugging extremely long IMAP folder exit times

[Jason L Tibbitts III]

Occasionally a folder on an IMAP server will get into a state where it
takes ages for gnus to leave it.  I have one folder which before today
was quick to exit, but today suddenly takes over two minutes.  Gnus
shows "Expiring articles..." in the minibuffer.

If I C-u RET to enter the group, gnus thinks there are over 364000
messages in the group.  There are actually only about 9000 messages
there, according to another IMAP client.  My guess is that gnus is just
iterating over 364k entries to see what it should expire.

Note that when I run gnus-group-expire-articles on the group from the
summary buffer, it doesn't take very long at all.

Here is what's in newsrc.eld for the group.  I was worried there would
be a huge set of ranges in there but the data is quite simple.

("fail2ban" 3 ((1 . 364570)) ((unexist) (seen (364431 . 364570))) nil
((modseq) (uidvalidity . "1535374911") (active 1 . 364570)
(permanent-flags %Answered %Flagged %Draft %Deleted %Seen %*)
total-expire (expiry-wait . 7)))

Any hints on understanding what gnus is doing here, or suggestions on
how to speed it up?  I used to know quite a bit about Emacs and Gnus
internals but it has been many years now since I tried to do anything
interesting.

I am running emacs 26.2; gnus-version says "Gnus v5.13".

Thanks,

 - J<

Re: Debugging extremely long IMAP folder exit times

[Andreas Schwab]

On Jul 18 2019, Jason L Tibbitts III <tibbs@math.uh.edu> wrote:

> Here is what's in newsrc.eld for the group.  I was worried there would
> be a huge set of ranges in there but the data is quite simple.
>
> ("fail2ban" 3 ((1 . 364570)) ((unexist) (seen (364431 . 364570))) nil

This is your problem.  Since the unexist list is empty, gnus doesn't
know that the articles 1 ... 354000 (approx.) don't exist.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Debugging extremely long IMAP folder exit times

[Jason L Tibbitts III]

>>>>> "AS" == Andreas Schwab <schwab@linux-m68k.org> writes:

AS> This is your problem.  Since the unexist list is empty, gnus doesn't
AS> know that the articles 1 ... 354000 (approx.) don't exist.

OK, so... I didn't do anything specific to make that happen.  How would
I fix it?

I note that after I sent my message, exiting that group/folder somehow
became quite fast for a while.  But I quit gnus and closed emacs, and
after restarting it again took minutes to exit the group.

 - J<

Re: Debugging extremely long IMAP folder exit times

[Jason L Tibbitts III]

To answer my own question, using M-c on the folder (from the Group
buffer) was sufficient to get things back into a sane state.
Fortunately I had no marks or anything in that folder to lose.  If I
had, I suppose the best option would have been to move the ticket
messages to another folder first.

 - J<

Re: Debugging extremely long IMAP folder exit times

[Eric Abrahamsen]

Jason L Tibbitts III <tibbs@math.uh.edu> writes:

> To answer my own question, using M-c on the folder (from the Group
> buffer) was sufficient to get things back into a sane state.
> Fortunately I had no marks or anything in that folder to lose.  If I
> had, I suppose the best option would have been to move the ticket
> messages to another folder first.

Ticked is equivalent to IMAP flagged, so that data would also be
preserved in the server, and wouldn't need to be backed up (I think).

I was just looking to see if there's a "unilaterally reset group marks
from IMAP server flags" command, but at first glance there doesn't
appear to be. That would be a nice thing to have. In theory, Gnus
doesn't really need to keep marks that are redundant to server flags in
the first place, but that would take some tweaking. It would also be
nice to know how your 'unexist flags got out of whack to begin with.

Much to do.

Re: Debugging extremely long IMAP folder exit times

[Jason L Tibbitts III]

>>>>> "EA" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

EA> Ticked is equivalent to IMAP flagged, so that data would also be
EA> preserved in the server, and wouldn't need to be backed up (I
EA> think).

I would think so, but when I did this Gnus lost track of which messages
I had read and (in IMAP at least) that's also server-side state.  So I
really have no idea what might be preserved after a cal to
gnus-group-clear-data.  The documentation does imply that it clears all
marks, so I figured it actively cleared everything on the server as
well.

EA> In theory, Gnus doesn't really need to keep marks that are
EA> redundant to server flags in the first place, but that would take
EA> some tweaking.

Yes, I don't think the original design anticipated having another entity
which could keep track of all mailbox state.

As an aside, I wonder how difficult it would be to support what will be
rfc8621 (JMAP-Mail).  It's certainly a good bit simpler than IMAP in
some ways.  The fact that it supports sending mail (just have a message
stored somewhere on the server and provide a request to send it along
with some metadata) is a rather different paradigm from the current way
it's all set up.

My guess it that it needs quite a bit of work by people smarter/better
at elisp than I am but maybe someone's already playing with it.

EA> It would also be nice to know how your 'unexist flags got out of
EA> whack to begin with.

I honestly have no idea.  I do generally have more than one IMAP client
connected to the server at once, but I was only using one of them within
a few hours of this one folder becoming slow.  This also isn't the first
time I've had the issue, but it's the first time in a few months.

 - J<

Re: Debugging extremely long IMAP folder exit times

[Eric Abrahamsen]

Jason L Tibbitts III <tibbs@math.uh.edu> writes:

>>>>>> "EA" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
> EA> Ticked is equivalent to IMAP flagged, so that data would also be
> EA> preserved in the server, and wouldn't need to be backed up (I
> EA> think).
>
> I would think so, but when I did this Gnus lost track of which messages
> I had read and (in IMAP at least) that's also server-side state.  So I
> really have no idea what might be preserved after a cal to
> gnus-group-clear-data.  The documentation does imply that it clears all
> marks, so I figured it actively cleared everything on the server as
> well.

Oh, I guess you're right, it removes marks from the backend as well. In
the past I think I've "dealt with" this by shutting down Gnus, going
into my newsrc.eld, completely removing the entry for the imap group,
and then restarting Gnus and re-subscribing to the group.

> EA> In theory, Gnus doesn't really need to keep marks that are
> EA> redundant to server flags in the first place, but that would take
> EA> some tweaking.
>
> Yes, I don't think the original design anticipated having another entity
> which could keep track of all mailbox state.

Nope, but it would be nice if it could deal with that.

> As an aside, I wonder how difficult it would be to support what will be
> rfc8621 (JMAP-Mail).  It's certainly a good bit simpler than IMAP in
> some ways.  The fact that it supports sending mail (just have a message
> stored somewhere on the server and provide a request to send it along
> with some metadata) is a rather different paradigm from the current way
> it's all set up.
>
> My guess it that it needs quite a bit of work by people smarter/better
> at elisp than I am but maybe someone's already playing with it.

I don't think anyone is, but it's definitely been in the back of my mind
when thinking about refactoring Gnus' backends. It would be fun to have
Gnus be the first client outside of Fastmail's own software to support
JMAP. It looks like they've just published the first RFC for the storage
part of JMAP, but not yet the mail-related part.

I've thought about creating light connections between messages under
composition and Gnus servers, so that the server itself could have a
chance to handle the message -- that would be useful for per-server
message drafts, but could also handle server's like JMAP that do the
sending themselves.

> EA> It would also be nice to know how your 'unexist flags got out of
> EA> whack to begin with.
>
> I honestly have no idea.  I do generally have more than one IMAP client
> connected to the server at once, but I was only using one of them within
> a few hours of this one folder becoming slow.  This also isn't the first
> time I've had the issue, but it's the first time in a few months.

Who knows... I access imap accounts from two Gnus installations and one
k9 android app, and nothing's ever gone wonky. Bit of a mystery.

Re: Debugging extremely long IMAP folder exit times

[Jason L Tibbitts III]

I think this is another instance of the same issue.  Fortunately the
count is small enough that it doesn't take all that long to exit the group.

("fedora-package-review" 3 ((1 . 146459)) ((unexist) (seen (140473 . 145786) (145837 . 146143) (146170 . 146459)) (killed (146173 . 146174) (146176 . 146188) (146190 . 146203) (146205 . 146228) (146231 . 146288) (146291 . 146292) (146294 . 146298) (146300 . 146321) 146323 (146325 . 146326) (146328 . 146341) (146343 . 146355) (146357 . 146367) 146369 (146371 . 146384) (146386 . 146389) 146391 (146393 . 146394) (146397 . 146414) 146417 (146419 . 146433) (146435 . 146459))) nil ((modseq) (uidvalidity . "1422948496") (active 1 . 146459) (permanent-flags %Answered %Flagged %Draft %Deleted %Seen %*) total-expire (expiry-wait . 3))) 

I guess I'll clear the data on that one as well.

 - J<

Re: Debugging extremely long IMAP folder exit times

[Dan Christensen]

On Jul 24, 2019, Jason L Tibbitts III <tibbs@math.uh.edu> wrote:

> I think this is another instance of the same issue.  Fortunately the
> count is small enough that it doesn't take all that long to exit the group.

...

> I guess I'll clear the data on that one as well.

If you haven't yet cleared the data, it would be interesting to see if
there are fixes that don't clear the remote marks and flags (even though
you don't seem to have any in this case).

1) Does `M-g' on the group fix the problem?  (I often have to do that on
IMAP groups to make Gnus fully re-read the server info.)

2) What if you `C-k' the group, exit Gnus, restart emacs and gnus, and
then subscribe to the group again?

Dan

Re: Debugging extremely long IMAP folder exit times

[Jason L Tibbitts III]

>>>>> "DC" == Dan Christensen <jdc@uwo.ca> writes:

DC> If you haven't yet cleared the data, it would be interesting to see
DC> if there are fixes that don't clear the remote marks and flags (even
DC> though you don't seem to have any in this case).

I haven't yet, but somehow the problem has fixed itself in the
meantime.  Just checked .newsrc.eld and the unexist info is filled in.

So when I have this problem again, I will go ahead and try both of your
suggestions and see if they help.

 - J<