davmail hangs *after* retrieving pop email from Exchange server

davmail hangs *after* retrieving pop email from Exchange server

[Eric S Fraga]

Hello,

TL;DR: how do I figure out what gnus is doing when it hangs after
downloading emails via davmail from an Exchange server?

Longer story: due to the introduction of multi-factor authentication
(MFA) in our institution, I have spent the past 2 days trying to find a
solution that allows me to continue using gnus.

For the record, one interesting albeit less than satisfying solution is
to run Thunderbird with pop access to the exchange server.  Thunderbird
supports OAuth2 access.  Then you can point gnus to the mbox format file
Thunderbird creates to store the downloaded emails.  This does mean
having to have Thunderbird running...

Davmail is the most often suggested solution.  It took some time to get
this working but it's sort of there.  I can download emails using pop
access.  However, for some reason, after retrieving the emails, gnus
hangs until I hit C-g and then the group buffer is updated.

So, my question is: how do I find out what is happening after the emails
have downloaded?  The davmail log doesn't show anything and neither does
*Messages*.  Are there gnus variables I can set to help me figure this
out?  It does seem that davmail has does everything it should and so has
gnus up to a point.

Thank you,
eric

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian bullseye/sid

Re: davmail hangs *after* retrieving pop email from Exchange server

[Dan Christensen]

On May 27, 2021, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:

> TL;DR: how do I figure out what gnus is doing when it hangs after
> downloading emails via davmail from an Exchange server?

This doesn't answer your question, but I use fetchmail to connect to
davmail and download the messages, and then I point Gnus at the already
downloaded messages.  Works well.  My only complaint is that davmail
doesn't seem to support IDLE (or whatever the equivalent is with
Exchange), so I have to make fetchmail poll frequently if I want to
see messages quickly.

Dan

Re: davmail hangs *after* retrieving pop email from Exchange server

[Eric S Fraga]

On Thursday, 27 May 2021 at 09:14, Dan Christensen wrote:
> This doesn't answer your question, but I use fetchmail to connect to
> davmail and download the messages, and then I point Gnus at the already
> downloaded messages.  

Yes, that's an option indeed.  Hadn't thought of doing that.  Could make
things much snappier.

Thank you,
eric

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian bullseye/sid

Re: davmail hangs *after* retrieving pop email from Exchange server

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Hello,
>
> TL;DR: how do I figure out what gnus is doing when it hangs after
> downloading emails via davmail from an Exchange server?

I think the usual thing to do would be M-x toggle-debug-on-quit, then
get to the hang, then C-g. You should see a traceback telling you what
Emacs was doing when you quit it.

Re: davmail hangs *after* retrieving pop email from Exchange server

[Eric S Fraga]

On Thursday, 27 May 2021 at 08:40, Eric Abrahamsen wrote:
> I think the usual thing to do would be M-x toggle-debug-on-quit, then
> get to the hang, then C-g. You should see a traceback telling you what
> Emacs was doing when you quit it.

Thank you for this suggestion.  However, I've tried this already (debug
on error and debug on quit are my first ports of call...) and I do not
get a backtrace when I hit C-g but simply continues with the rest of the
email & news retrieval.  This does suggest a network hang of some sort
but I don't understand why/how.  These types of problems are the most
frustrating in gnus (maybe Emacs generally?).

Thanks again,
eric

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian bullseye/sid

Re: davmail hangs *after* retrieving pop email from Exchange server

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Thursday, 27 May 2021 at 08:40, Eric Abrahamsen wrote:
>> I think the usual thing to do would be M-x toggle-debug-on-quit, then
>> get to the hang, then C-g. You should see a traceback telling you what
>> Emacs was doing when you quit it.
>
> Thank you for this suggestion.  However, I've tried this already (debug
> on error and debug on quit are my first ports of call...) 

I should have known that :)

> and I do not get a backtrace when I hit C-g but simply continues with
> the rest of the email & news retrieval. This does suggest a network
> hang of some sort but I don't understand why/how. These types of
> problems are the most frustrating in gnus (maybe Emacs generally?).

They are frustrating! And my guess is that a `condition-case' somewhere
is handling the `quit' -- all of a sudden this sort of thing is
responsible for like 90% of my Emacs debugging. It's even possible that
it's the same location as Hirotaka Honda has run in to: mail splitting
inside `nnmail-get-new-mail-1'. Do you see anything in *Messages*? How
about with `gnus-verbose-backends' set to 10? Or try stepping thrhough
`nnmail-get-new-mail-1' and see if anything "bad" happens...

Re: davmail hangs *after* retrieving pop email from Exchange server

[Eric S Fraga]

Hi Eric,

So, further followup: getting new email now returns relatively quickly,
and quicker each time over the day.

I think the delays may have been due to davmail or gnus (?) doing
something about the very large number of previously read emails in my
Exchange server (hundreds of thousands, in all likelihood).  Maybe
something has incrementally been updated and no longer needs further
updates?  This is me grasping at straws but basically it seems to be
working fine now.

Thank you,
eric

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian bullseye/sid

Re: davmail hangs *after* retrieving pop email from Exchange server

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Hi Eric,
>
> So, further followup: getting new email now returns relatively quickly,
> and quicker each time over the day.
>
> I think the delays may have been due to davmail or gnus (?) doing
> something about the very large number of previously read emails in my
> Exchange server (hundreds of thousands, in all likelihood).  Maybe
> something has incrementally been updated and no longer needs further
> updates?  This is me grasping at straws but basically it seems to be
> working fine now.

Your guess is as good as mine! Building nov databases? Who knows. Glad
to hear it's working okay.

Re: davmail hangs *after* retrieving pop email from Exchange server

[Eric S Fraga]

Follow up on this.

I opened up a window with htop running to see if Emacs was waiting on
something running, e.g. in particular davmail and maybe firefox (due to
the MFA bits), but this was not the case.

Also, eventually, gnus continues.  This might suggest it times out on
some network request?  Nothing is written out anywhere so I cannot tell.

I would like to debug this further so suggestions very welcome!

Thank you,
eric

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian bullseye/sid