Gnus fetch freezes emacs

Gnus fetch freezes emacs

[Prashant Tak]

Gnus has been freezing sporadically when `gnus-group-get-new-news` is run.
And it keeps on going for hours, I have to manually intercept and signal
`keyboard-quit` and then perform the fetch operation again. This happens
in a very unpredictable manner, so it's hard to replicate. I did manage
to get a profiler report when that happened.

        7367  99% - command-execute
        7367  99%  - call-interactively
        7070  95%   - funcall-interactively
        7070  95%    - gnus-group-get-new-news
        7063  95%     - gnus-get-unread-articles
        7063  95%      - gnus-read-active-for-groups
        7063  95%       - gnus-finish-retrieve-group-infos
        7063  95%        - nntp-finish-retrieve-group-infos
        7063  95%         - nntp-with-open-group-function
        6851  92%          - #<compiled -0x1643a6a03cf748e>
        6416  86%           - nntp-accept-response
        6068  82%            - nntp-accept-process-output
        5873  79%             - nnheader-accept-process-output
          19   0%              + accept-process-output
          34   0%               nntp-find-connection-buffer
           7   0%       gnus-parent-read-child-newsrc
         297   4%   - byte-code
         297   4%    + read-extended-command
          15   0% + ...

The main culprit seems to be `nnheader-accept-process-output` but I
don't know how to proceed further. Appreciate any help/input into the matter.

-- 
Prashant Tak

Re: Gnus fetch freezes emacs

[Stephen Berman]

On Fri, 30 Jun 2023 20:03:11 +0530 Prashant Tak <prashantrameshtak@gmail.com> wrote:

> Gnus has been freezing sporadically when `gnus-group-get-new-news` is run.
> And it keeps on going for hours, I have to manually intercept and signal
> `keyboard-quit` and then perform the fetch operation again. This happens
> in a very unpredictable manner, so it's hard to replicate. I did manage
> to get a profiler report when that happened.
>
>         7367  99% - command-execute
>         7367  99%  - call-interactively
>         7070  95%   - funcall-interactively
>         7070  95%    - gnus-group-get-new-news
>         7063  95%     - gnus-get-unread-articles
>         7063  95%      - gnus-read-active-for-groups
>         7063  95%       - gnus-finish-retrieve-group-infos
>         7063  95%        - nntp-finish-retrieve-group-infos
>         7063  95%         - nntp-with-open-group-function
>         6851  92%          - #<compiled -0x1643a6a03cf748e>
>         6416  86%           - nntp-accept-response
>         6068  82%            - nntp-accept-process-output
>         5873  79%             - nnheader-accept-process-output
>           19   0%              + accept-process-output
>           34   0%               nntp-find-connection-buffer
>            7   0%       gnus-parent-read-child-newsrc
>          297   4%   - byte-code
>          297   4%    + read-extended-command
>           15   0% + ...
>
> The main culprit seems to be `nnheader-accept-process-output` but I
> don't know how to proceed further. Appreciate any help/input into the matter.

This sounds like the issue I've been having with gnus-group-get-new-news
and similar Gnus commands for more than a year and a half, see
bug#52735.  As reported there, I did some debugging but couldn't
pinpoint the problem, nor have I tried profiling yet.  But as a
workaround, I've been using the following replacement for
gnus-group-get-new-news:

(defun srb-gnus-group-get-new-news (&optional arg one-level)
  (interactive "P")
  (with-timeout (1 (kill-buffer (nntp-find-connection-buffer nntp-server-buffer))
		   (gnus-group-get-new-news))
    (gnus-group-get-new-news arg one-level)))

(define-key gnus-group-mode-map "g" 'srb-gnus-group-get-new-news)

This usually suffices but not always.  When Gnus (and hence Emacs) hangs
even when using this workaround, I've resorted to manually killing the
server buffer " *server news.gmane.io nntp *nntpd**" and then typing `g'
pretty reliably works again.

This is a very annoying issue, and if what you're experiencing is the
same, I commiserate with you, but your report also gives me hope that
it's not just some quirk of my setup or network connection.  Now we just
need some Gnus expert to chime in and guide us to try and track down the
cause of this issue and get it fixed.

Steve Berman

Re: Gnus fetch freezes emacs

[Eric S Fraga]

On Friday, 30 Jun 2023 at 21:33, Stephen Berman wrote:
> This is a very annoying issue, and if what you're experiencing is the
> same, I commiserate with you, but your report also gives me hope that
> it's not just some quirk of my setup or network connection.  

If it's further consolation, I seem to have the same problem with hangs
on a non-reproducible basis.  I've not had the time to dig into it but
it definitely sounds similar to what you and the OP are seeing.

-- 
Eric S Fraga via gnus (Emacs 30.0.50 2023-06-29) on Debian 12.0

Re: Gnus fetch freezes emacs

[Stephen Berman]

On Sat, 01 Jul 2023 11:41:17 +0100 Eric S Fraga <e.fraga@ucl.ac.uk> wrote:

> On Friday, 30 Jun 2023 at 21:33, Stephen Berman wrote:
>> This is a very annoying issue, and if what you're experiencing is the
>> same, I commiserate with you, but your report also gives me hope that
>> it's not just some quirk of my setup or network connection.
>
> If it's further consolation, I seem to have the same problem with hangs
> on a non-reproducible basis.  I've not had the time to dig into it but
> it definitely sounds similar to what you and the OP are seeing.

The more the merrier!  Or at least, maybe the higher the probability
that a Gnus guru gets hit by it and has the incentive to fix it.

Steve Berman

Re: Gnus fetch freezes emacs

[Prashant Tak]

Stephen Berman <stephen.berman@gmx.net> writes:

> On Fri, 30 Jun 2023 20:03:11 +0530 Prashant Tak <prashantrameshtak@gmail.com>
> wrote:
>
>> Gnus has been freezing sporadically when `gnus-group-get-new-news` is run.
>> And it keeps on going for hours, I have to manually intercept and signal
>> `keyboard-quit` and then perform the fetch operation again. This happens
>> in a very unpredictable manner, so it's hard to replicate. I did manage
>> to get a profiler report when that happened.
>>
>>         6416  86%           - nntp-accept-response
>>         6068  82%            - nntp-accept-process-output
>>         5873  79%             - nnheader-accept-process-output
>>           19   0%              + accept-process-output
>>
>> The main culprit seems to be `nnheader-accept-process-output` but I
>> don't know how to proceed further. Appreciate any help/input into the matter.
>
> This sounds like the issue I've been having with gnus-group-get-new-news
> and similar Gnus commands for more than a year and a half, see
> bug#52735.  As reported there, I did some debugging but couldn't
> pinpoint the problem, nor have I tried profiling yet.  But as a
> workaround, I've been using the following replacement for
> gnus-group-get-new-news:
>
> (defun srb-gnus-group-get-new-news (&optional arg one-level)
>   (interactive "P")
>   (with-timeout (1 (kill-buffer (nntp-find-connection-buffer nntp-server-buffer))
> 		   (gnus-group-get-new-news))
>     (gnus-group-get-new-news arg one-level)))
>
> (define-key gnus-group-mode-map "g" 'srb-gnus-group-get-new-news)
>
> This usually suffices but not always.  When Gnus (and hence Emacs) hangs
> even when using this workaround, I've resorted to manually killing the
> server buffer " *server news.gmane.io nntp *nntpd**" and then typing `g'
> pretty reliably works again.
>
> This is a very annoying issue, and if what you're experiencing is the
> same, I commiserate with you, but your report also gives me hope that
> it's not just some quirk of my setup or network connection.  Now we just
> need some Gnus expert to chime in and guide us to try and track down the
> cause of this issue and get it fixed.

I took a look at your bug report and indeed, the behaviour described is
identical to what I've been experiencing, hopefully someone does chime
in with an idea on how to improve the situation. Thanks for sharing your
solution for the meantime though.

-- 
Prashant Tak

Re: Gnus fetch freezes emacs

[Eric Abrahamsen]

Prashant Tak <prashantrameshtak@gmail.com> writes:

> Stephen Berman <stephen.berman@gmx.net> writes:
>
>> On Fri, 30 Jun 2023 20:03:11 +0530 Prashant Tak <prashantrameshtak@gmail.com>
>> wrote:
>>
>>> Gnus has been freezing sporadically when `gnus-group-get-new-news` is run.
>>> And it keeps on going for hours, I have to manually intercept and signal
>>> `keyboard-quit` and then perform the fetch operation again. This happens
>>> in a very unpredictable manner, so it's hard to replicate. I did manage
>>> to get a profiler report when that happened.
>>>
>>>         6416  86%           - nntp-accept-response
>>>         6068  82%            - nntp-accept-process-output
>>>         5873  79%             - nnheader-accept-process-output
>>>           19   0%              + accept-process-output
>>>
>>> The main culprit seems to be `nnheader-accept-process-output` but I
>>> don't know how to proceed further. Appreciate any help/input into the matter.
>>
>> This sounds like the issue I've been having with gnus-group-get-new-news
>> and similar Gnus commands for more than a year and a half, see
>> bug#52735.  As reported there, I did some debugging but couldn't
>> pinpoint the problem, nor have I tried profiling yet.  But as a
>> workaround, I've been using the following replacement for
>> gnus-group-get-new-news:
>>
>> (defun srb-gnus-group-get-new-news (&optional arg one-level)
>>   (interactive "P")
>>   (with-timeout (1 (kill-buffer (nntp-find-connection-buffer nntp-server-buffer))
>> 		   (gnus-group-get-new-news))
>>     (gnus-group-get-new-news arg one-level)))
>>
>> (define-key gnus-group-mode-map "g" 'srb-gnus-group-get-new-news)
>>
>> This usually suffices but not always.  When Gnus (and hence Emacs) hangs
>> even when using this workaround, I've resorted to manually killing the
>> server buffer " *server news.gmane.io nntp *nntpd**" and then typing `g'
>> pretty reliably works again.
>>
>> This is a very annoying issue, and if what you're experiencing is the
>> same, I commiserate with you, but your report also gives me hope that
>> it's not just some quirk of my setup or network connection.  Now we just
>> need some Gnus expert to chime in and guide us to try and track down the
>> cause of this issue and get it fixed.
>
> I took a look at your bug report and indeed, the behaviour described is
> identical to what I've been experiencing, hopefully someone does chime
> in with an idea on how to improve the situation. Thanks for sharing your
> solution for the meantime though.

If everyone's hitting this with NNTP servers, you can set
`nntp-connection-timeout' to a number of seconds. It is nil by default,
which I guess would result in permanent hangs.

Re: Gnus fetch freezes emacs

[Eric S Fraga]

On Sunday,  2 Jul 2023 at 16:59, Eric Abrahamsen wrote:
> If everyone's hitting this with NNTP servers, you can set
> `nntp-connection-timeout' to a number of seconds. It is nil by default,
> which I guess would result in permanent hangs.

Thank you for the suggestion.  I have set this to 15 seconds and we'll
see if this makes any difference!  I'll report back in due course
hopefully.

-- 
Eric S Fraga via gnus (Emacs 30.0.50 2023-06-29) on Debian 12.0

Re: Gnus fetch freezes emacs

[Eric S Fraga]

On Sunday,  2 Jul 2023 at 16:59, Eric Abrahamsen wrote:
> If everyone's hitting this with NNTP servers, you can set
> `nntp-connection-timeout' to a number of seconds. It is nil by default,
> which I guess would result in permanent hangs.

So this works, in the sense that it stops me waiting forever... However,
it seems (early days yet) that when it fails to open the connection to
an NNTP server, it stops retrieving news and I have to hit 'g' again to
get the counts etc. updated for other servers.  But much better than
waiting forever.

Thank you,
eric
-- 
Eric S Fraga via gnus (Emacs 30.0.50 2023-06-29) on Debian 12.0

Re: Gnus fetch freezes emacs

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Sunday,  2 Jul 2023 at 16:59, Eric Abrahamsen wrote:
>> If everyone's hitting this with NNTP servers, you can set
>> `nntp-connection-timeout' to a number of seconds. It is nil by default,
>> which I guess would result in permanent hangs.
>
> So this works, in the sense that it stops me waiting forever... However,
> it seems (early days yet) that when it fails to open the connection to
> an NNTP server, it stops retrieving news and I have to hit 'g' again to
> get the counts etc. updated for other servers.  But much better than
> waiting forever.

Yeah, I'd put in a dumb fix for this that turned out to be buggy, so we
just recently reverted it. I have a more thorough fix in progress
somewhere here, that would report a server connection failure without
interrupting the rest of the servers, but it's not done yet. I've had
very little time for coding recently, but will get to it At Some Point.

Glad it's at least better than it was. I wonder if we should have some
generous timeout set by default...

Re: Gnus fetch freezes emacs

[Stephen Berman]

On Mon, 03 Jul 2023 09:36:26 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> On Sunday,  2 Jul 2023 at 16:59, Eric Abrahamsen wrote:
>>> If everyone's hitting this with NNTP servers, you can set
>>> `nntp-connection-timeout' to a number of seconds. It is nil by default,
>>> which I guess would result in permanent hangs.

Is this variable supposed to be set in the value of gnus-select-method?
For example, like this:

(setq gnus-select-method '(nntp "news.gmane.io"
				(nntp-connection-timeout 3)))

>> So this works, in the sense that it stops me waiting forever... However,
>> it seems (early days yet) that when it fails to open the connection to
>> an NNTP server, it stops retrieving news and I have to hit 'g' again to
>> get the counts etc. updated for other servers. [...]

That sounds basically like what the function I'm using in place of
gnus-group-get-new-news (see my first post in this thread) does.  Could
such a function take effect if added to one of the server hook variables
nntp-server-opened-hook, nntp-server-action-alist or
nntp-open-connection-function?  From the descriptions in the manual it
isn't clear to me.  Or is there some better Gnus hook variable for this
purpose?  If not could one be added?

> Yeah, I'd put in a dumb fix for this that turned out to be buggy, so we
> just recently reverted it. I have a more thorough fix in progress
> somewhere here, that would report a server connection failure without
> interrupting the rest of the servers, but it's not done yet. I've had
> very little time for coding recently, but will get to it At Some Point.
>
> Glad it's at least better than it was. I wonder if we should have some
> generous timeout set by default...

It might make sense to continue this discussion in bug#52735.

Steve Berman

Re: Gnus fetch freezes emacs

[Eric Abrahamsen]

Stephen Berman <stephen.berman@gmx.net> writes:

> On Mon, 03 Jul 2023 09:36:26 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>
>>> On Sunday,  2 Jul 2023 at 16:59, Eric Abrahamsen wrote:
>>>> If everyone's hitting this with NNTP servers, you can set
>>>> `nntp-connection-timeout' to a number of seconds. It is nil by default,
>>>> which I guess would result in permanent hangs.
>
> Is this variable supposed to be set in the value of gnus-select-method?
> For example, like this:
>
> (setq gnus-select-method '(nntp "news.gmane.io"
> 				(nntp-connection-timeout 3)))

It's a defvoo, so it can either be set globally, or as a server
parameter.

>>> So this works, in the sense that it stops me waiting forever... However,
>>> it seems (early days yet) that when it fails to open the connection to
>>> an NNTP server, it stops retrieving news and I have to hit 'g' again to
>>> get the counts etc. updated for other servers. [...]
>
> That sounds basically like what the function I'm using in place of
> gnus-group-get-new-news (see my first post in this thread) does.  Could
> such a function take effect if added to one of the server hook variables
> nntp-server-opened-hook, nntp-server-action-alist or
> nntp-open-connection-function?  From the descriptions in the manual it
> isn't clear to me.  Or is there some better Gnus hook variable for this
> purpose?  If not could one be added?

I'm not sure what function you mean. Eric F is just describing the
unfortunate behavior of nntp-connection-timeout, which interrupts the
entire fetching process when it hits the timeout.

>> Yeah, I'd put in a dumb fix for this that turned out to be buggy, so we
>> just recently reverted it. I have a more thorough fix in progress
>> somewhere here, that would report a server connection failure without
>> interrupting the rest of the servers, but it's not done yet. I've had
>> very little time for coding recently, but will get to it At Some Point.
>>
>> Glad it's at least better than it was. I wonder if we should have some
>> generous timeout set by default...
>
> It might make sense to continue this discussion in bug#52735.

This doesn't seem like the same issue -- this problem is pretty well
understood.

Eric

Re: Gnus fetch freezes emacs

[Stephen Berman]

On Tue, 04 Jul 2023 10:02:34 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> Stephen Berman <stephen.berman@gmx.net> writes:
>
>> On Mon, 03 Jul 2023 09:36:26 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>>
>>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>>
>>>> On Sunday,  2 Jul 2023 at 16:59, Eric Abrahamsen wrote:
>>>>> If everyone's hitting this with NNTP servers, you can set
>>>>> `nntp-connection-timeout' to a number of seconds. It is nil by default,
>>>>> which I guess would result in permanent hangs.
>>
>> Is this variable supposed to be set in the value of gnus-select-method?
>> For example, like this:
>>
>> (setq gnus-select-method '(nntp "news.gmane.io"
>> 				(nntp-connection-timeout 3)))
>
> It's a defvoo, so it can either be set globally, or as a server
> parameter.

Ok, thanks.

>>>> So this works, in the sense that it stops me waiting forever... However,
>>>> it seems (early days yet) that when it fails to open the connection to
>>>> an NNTP server, it stops retrieving news and I have to hit 'g' again to
>>>> get the counts etc. updated for other servers. [...]
>>
>> That sounds basically like what the function I'm using in place of
>> gnus-group-get-new-news (see my first post in this thread) does.  Could
>> such a function take effect if added to one of the server hook variables
>> nntp-server-opened-hook, nntp-server-action-alist or
>> nntp-open-connection-function?  From the descriptions in the manual it
>> isn't clear to me.  Or is there some better Gnus hook variable for this
>> purpose?  If not could one be added?
>
> I'm not sure what function you mean.

This:

 (defun srb-gnus-group-get-new-news (&optional arg one-level)
   (interactive "P")
   (with-timeout (1 (kill-buffer (nntp-find-connection-buffer nntp-server-buffer))
 		   (gnus-group-get-new-news))
     (gnus-group-get-new-news arg one-level)))

 (define-key gnus-group-mode-map "g" 'srb-gnus-group-get-new-news)

>                                      Eric F is just describing the
> unfortunate behavior of nntp-connection-timeout, which interrupts the
> entire fetching process when it hits the timeout.

Is that different than what the above function does with the kill-buffer
sexp?  (Not a rhetorical question, I know next to nothing about news
servers and their connectivity issues.)

>>> Yeah, I'd put in a dumb fix for this that turned out to be buggy, so we
>>> just recently reverted it. I have a more thorough fix in progress
>>> somewhere here, that would report a server connection failure without
>>> interrupting the rest of the servers, but it's not done yet. I've had
>>> very little time for coding recently, but will get to it At Some Point.
>>>
>>> Glad it's at least better than it was. I wonder if we should have some
>>> generous timeout set by default...
>>
>> It might make sense to continue this discussion in bug#52735.
>
> This doesn't seem like the same issue -- this problem is pretty well
> understood.

Hm, I had understood from both Prashant Tak and Eric Fraga that the
problem they have is essentially the same as I do and what I reported in
that bug.  But that problem doesn't seem to be understood.  If by the
understood problem you mean the effect of nntp-connection-timeout,
doesn't that just mean using it isn't a real fix for the hang the three
of us (at least) are experiencing?  That's why I thought other
approaches need to be considered and bug#52735 seems like the
appropriate venue for that.  But I'm fine with continuing the discussion
here instead.

Steve Berman

Re: Gnus fetch freezes emacs

[Eric Abrahamsen]

Stephen Berman <stephen.berman@gmx.net> writes:

> On Tue, 04 Jul 2023 10:02:34 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> Stephen Berman <stephen.berman@gmx.net> writes:

[...]

> This:
>
>  (defun srb-gnus-group-get-new-news (&optional arg one-level)
>    (interactive "P")
>    (with-timeout (1 (kill-buffer (nntp-find-connection-buffer nntp-server-buffer))
>  		   (gnus-group-get-new-news))
>      (gnus-group-get-new-news arg one-level)))
>
>  (define-key gnus-group-mode-map "g" 'srb-gnus-group-get-new-news)
>
>>                                      Eric F is just describing the
>> unfortunate behavior of nntp-connection-timeout, which interrupts the
>> entire fetching process when it hits the timeout.
>
> Is that different than what the above function does with the kill-buffer
> sexp?  (Not a rhetorical question, I know next to nothing about news
> servers and their connectivity issues.)

The `nntp-connection-timeout' variable has different behavior in that
NNTP servers are allowed one "retry" if the connection fails. The code
around that is very confusing to me (which is why my earlier fix was
buggy).

>>>> Yeah, I'd put in a dumb fix for this that turned out to be buggy, so we
>>>> just recently reverted it. I have a more thorough fix in progress
>>>> somewhere here, that would report a server connection failure without
>>>> interrupting the rest of the servers, but it's not done yet. I've had
>>>> very little time for coding recently, but will get to it At Some Point.
>>>>
>>>> Glad it's at least better than it was. I wonder if we should have some
>>>> generous timeout set by default...
>>>
>>> It might make sense to continue this discussion in bug#52735.
>>
>> This doesn't seem like the same issue -- this problem is pretty well
>> understood.
>
> Hm, I had understood from both Prashant Tak and Eric Fraga that the
> problem they have is essentially the same as I do and what I reported in
> that bug.  But that problem doesn't seem to be understood.  If by the
> understood problem you mean the effect of nntp-connection-timeout,
> doesn't that just mean using it isn't a real fix for the hang the three
> of us (at least) are experiencing?  That's why I thought other
> approaches need to be considered and bug#52735 seems like the
> appropriate venue for that.  But I'm fine with continuing the discussion
> here instead.

Oh I see what you mean. In your bug report I'd gotten the idea that
something was going wrong with accepting process output, and had a
missed-the-forest-for-the-trees moment around it simply being a dead
process.

Using `nntp-connection-timeout' is the proper fix for this problem, it's
just got a bit of unfortunate behavior that needs to be remedied. I'd be
inclined to start a whole new bug report for a fix for that, because
it's really a new issue, with its own larger-reaching design decisions.
I suppose we could merge #52735 with that, though.

Eric

Re: Gnus fetch freezes emacs

[Stephen Berman]

On Tue, 04 Jul 2023 20:50:05 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> Stephen Berman <stephen.berman@gmx.net> writes:
>
>> On Tue, 04 Jul 2023 10:02:34 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>>
>>> Stephen Berman <stephen.berman@gmx.net> writes:
>
> [...]
>
>> This:
>>
>>  (defun srb-gnus-group-get-new-news (&optional arg one-level)
>>    (interactive "P")
>>    (with-timeout (1 (kill-buffer (nntp-find-connection-buffer nntp-server-buffer))
>>  		   (gnus-group-get-new-news))
>>      (gnus-group-get-new-news arg one-level)))
>>
>>  (define-key gnus-group-mode-map "g" 'srb-gnus-group-get-new-news)
>>
>>>                                      Eric F is just describing the
>>> unfortunate behavior of nntp-connection-timeout, which interrupts the
>>> entire fetching process when it hits the timeout.
>>
>> Is that different than what the above function does with the kill-buffer
>> sexp?  (Not a rhetorical question, I know next to nothing about news
>> servers and their connectivity issues.)
>
> The `nntp-connection-timeout' variable has different behavior in that
> NNTP servers are allowed one "retry" if the connection fails. The code
> around that is very confusing to me (which is why my earlier fix was
> buggy).

I don't follow you, but no need to elaborate further here.

>>>>> Yeah, I'd put in a dumb fix for this that turned out to be buggy, so we
>>>>> just recently reverted it. I have a more thorough fix in progress
>>>>> somewhere here, that would report a server connection failure without
>>>>> interrupting the rest of the servers, but it's not done yet. I've had
>>>>> very little time for coding recently, but will get to it At Some Point.
>>>>>
>>>>> Glad it's at least better than it was. I wonder if we should have some
>>>>> generous timeout set by default...
>>>>
>>>> It might make sense to continue this discussion in bug#52735.
>>>
>>> This doesn't seem like the same issue -- this problem is pretty well
>>> understood.
>>
>> Hm, I had understood from both Prashant Tak and Eric Fraga that the
>> problem they have is essentially the same as I do and what I reported in
>> that bug.  But that problem doesn't seem to be understood.  If by the
>> understood problem you mean the effect of nntp-connection-timeout,
>> doesn't that just mean using it isn't a real fix for the hang the three
>> of us (at least) are experiencing?  That's why I thought other
>> approaches need to be considered and bug#52735 seems like the
>> appropriate venue for that.  But I'm fine with continuing the discussion
>> here instead.
>
> Oh I see what you mean. In your bug report I'd gotten the idea that
> something was going wrong with accepting process output, and had a
> missed-the-forest-for-the-trees moment around it simply being a dead
> process.
>
> Using `nntp-connection-timeout' is the proper fix for this problem, it's
> just got a bit of unfortunate behavior that needs to be remedied. I'd be
> inclined to start a whole new bug report for a fix for that, because
> it's really a new issue, with its own larger-reaching design decisions.
> I suppose we could merge #52735 with that, though.

Feel free to open a new bug for fixing nntp-connection-timeout.  I don't
know if I can help, other than trying out suggestions and providing
feedback.  In the meantime I'll keep using my workaround replacement
function.

But I wonder, could this issue have been triggered by some change in
news.gmane.io around early to mid December 2021?  Because that's when
the problem start for me, and prior to that I don't recall ever having
this problem (perhaps sporadically but not with such persistance).

Steve Berman

Re: Gnus fetch freezes emacs

[Eric Abrahamsen]

On 07/05/23 10:04 AM, Stephen Berman wrote:
> On Tue, 04 Jul 2023 20:50:05 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> Stephen Berman <stephen.berman@gmx.net> writes:
>>
>>> On Tue, 04 Jul 2023 10:02:34 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>>>
>>>> Stephen Berman <stephen.berman@gmx.net> writes:
>>
>> [...]
>>
>>> This:
>>>
>>>  (defun srb-gnus-group-get-new-news (&optional arg one-level)
>>>    (interactive "P")
>>>    (with-timeout (1 (kill-buffer (nntp-find-connection-buffer nntp-server-buffer))
>>>  		   (gnus-group-get-new-news))
>>>      (gnus-group-get-new-news arg one-level)))
>>>
>>>  (define-key gnus-group-mode-map "g" 'srb-gnus-group-get-new-news)
>>>
>>>>                                      Eric F is just describing the
>>>> unfortunate behavior of nntp-connection-timeout, which interrupts the
>>>> entire fetching process when it hits the timeout.
>>>
>>> Is that different than what the above function does with the kill-buffer
>>> sexp?  (Not a rhetorical question, I know next to nothing about news
>>> servers and their connectivity issues.)
>>
>> The `nntp-connection-timeout' variable has different behavior in that
>> NNTP servers are allowed one "retry" if the connection fails. The code
>> around that is very confusing to me (which is why my earlier fix was
>> buggy).
>
> I don't follow you, but no need to elaborate further here.
>
>>>>>> Yeah, I'd put in a dumb fix for this that turned out to be buggy, so we
>>>>>> just recently reverted it. I have a more thorough fix in progress
>>>>>> somewhere here, that would report a server connection failure without
>>>>>> interrupting the rest of the servers, but it's not done yet. I've had
>>>>>> very little time for coding recently, but will get to it At Some Point.
>>>>>>
>>>>>> Glad it's at least better than it was. I wonder if we should have some
>>>>>> generous timeout set by default...
>>>>>
>>>>> It might make sense to continue this discussion in bug#52735.
>>>>
>>>> This doesn't seem like the same issue -- this problem is pretty well
>>>> understood.
>>>
>>> Hm, I had understood from both Prashant Tak and Eric Fraga that the
>>> problem they have is essentially the same as I do and what I reported in
>>> that bug.  But that problem doesn't seem to be understood.  If by the
>>> understood problem you mean the effect of nntp-connection-timeout,
>>> doesn't that just mean using it isn't a real fix for the hang the three
>>> of us (at least) are experiencing?  That's why I thought other
>>> approaches need to be considered and bug#52735 seems like the
>>> appropriate venue for that.  But I'm fine with continuing the discussion
>>> here instead.
>>
>> Oh I see what you mean. In your bug report I'd gotten the idea that
>> something was going wrong with accepting process output, and had a
>> missed-the-forest-for-the-trees moment around it simply being a dead
>> process.
>>
>> Using `nntp-connection-timeout' is the proper fix for this problem, it's
>> just got a bit of unfortunate behavior that needs to be remedied. I'd be
>> inclined to start a whole new bug report for a fix for that, because
>> it's really a new issue, with its own larger-reaching design decisions.
>> I suppose we could merge #52735 with that, though.
>
> Feel free to open a new bug for fixing nntp-connection-timeout.  I don't
> know if I can help, other than trying out suggestions and providing
> feedback.  In the meantime I'll keep using my workaround replacement
> function.
>
> But I wonder, could this issue have been triggered by some change in
> news.gmane.io around early to mid December 2021?  Because that's when
> the problem start for me, and prior to that I don't recall ever having
> this problem (perhaps sporadically but not with such persistance).

The nntp-connection-timeout variable has been present and nil since
1999. I put my original buggy fix in at the end of October 2021, so that
seems suspicious, but that should only have had an effect if you had set
nntp-connection-timeout to something other than the default of nil.

In general it's pretty hard to say, though...

Re: Gnus fetch freezes emacs

[Stephen Berman]

On Wed, 05 Jul 2023 11:55:36 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> On 07/05/23 10:04 AM, Stephen Berman wrote:
>> On Tue, 04 Jul 2023 20:50:05 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
[...]
>>> Using `nntp-connection-timeout' is the proper fix for this problem, it's
>>> just got a bit of unfortunate behavior that needs to be remedied. I'd be
>>> inclined to start a whole new bug report for a fix for that, because
>>> it's really a new issue, with its own larger-reaching design decisions.
>>> I suppose we could merge #52735 with that, though.
>>
>> Feel free to open a new bug for fixing nntp-connection-timeout.  I don't
>> know if I can help, other than trying out suggestions and providing
>> feedback.  In the meantime I'll keep using my workaround replacement
>> function.
>>
>> But I wonder, could this issue have been triggered by some change in
>> news.gmane.io around early to mid December 2021?  Because that's when
>> the problem start for me, and prior to that I don't recall ever having
>> this problem (perhaps sporadically but not with such persistance).
>
> The nntp-connection-timeout variable has been present and nil since
> 1999. I put my original buggy fix in at the end of October 2021, so that
> seems suspicious, but that should only have had an effect if you had set
> nntp-connection-timeout to something other than the default of nil.

I've never touched nntp-connection-timeout, I only became aware of it
when you mentioned it in this thread.  Also, October 2021 is too early:
the early to mid December date stands out to me because in early
December of that year there was a change in my internet service, and I
started experiencing the Gnus problem soon thereafter.  At first I
thought it might be due to my change in internet service, but the
problem really was and remains only with fetching news with Gnus (and
not with fetching email with Gnus, which I do from different mail
servers).  I also only use news.gmane.io for fetching news, so that's
why I thought maybe something changed there.

> In general it's pretty hard to say, though...

With that I whole-heartedly agree.

Steve Berman

Re: Gnus fetch freezes emacs

[Eric Abrahamsen]

Stephen Berman <stephen.berman@gmx.net> writes:

> On Wed, 05 Jul 2023 11:55:36 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> On 07/05/23 10:04 AM, Stephen Berman wrote:
>>> On Tue, 04 Jul 2023 20:50:05 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
> [...]
>>>> Using `nntp-connection-timeout' is the proper fix for this problem, it's
>>>> just got a bit of unfortunate behavior that needs to be remedied. I'd be
>>>> inclined to start a whole new bug report for a fix for that, because
>>>> it's really a new issue, with its own larger-reaching design decisions.
>>>> I suppose we could merge #52735 with that, though.
>>>
>>> Feel free to open a new bug for fixing nntp-connection-timeout.  I don't
>>> know if I can help, other than trying out suggestions and providing
>>> feedback.  In the meantime I'll keep using my workaround replacement
>>> function.
>>>
>>> But I wonder, could this issue have been triggered by some change in
>>> news.gmane.io around early to mid December 2021?  Because that's when
>>> the problem start for me, and prior to that I don't recall ever having
>>> this problem (perhaps sporadically but not with such persistance).
>>
>> The nntp-connection-timeout variable has been present and nil since
>> 1999. I put my original buggy fix in at the end of October 2021, so that
>> seems suspicious, but that should only have had an effect if you had set
>> nntp-connection-timeout to something other than the default of nil.
>
> I've never touched nntp-connection-timeout, I only became aware of it
> when you mentioned it in this thread.  Also, October 2021 is too early:
> the early to mid December date stands out to me because in early
> December of that year there was a change in my internet service, and I
> started experiencing the Gnus problem soon thereafter.  At first I
> thought it might be due to my change in internet service, but the
> problem really was and remains only with fetching news with Gnus (and
> not with fetching email with Gnus, which I do from different mail
> servers).  I also only use news.gmane.io for fetching news, so that's
> why I thought maybe something changed there.

It's entirely possible! The buggy fix was reverted in early May, so if
you stopped seeing the bad behavior then, that would be pretty conclusive.

Re: Gnus fetch freezes emacs

[Stephen Berman]

On Wed, 05 Jul 2023 13:34:37 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> Stephen Berman <stephen.berman@gmx.net> writes:
>
>> On Wed, 05 Jul 2023 11:55:36 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
[...]
>>> The nntp-connection-timeout variable has been present and nil since
>>> 1999. I put my original buggy fix in at the end of October 2021, so that
>>> seems suspicious, but that should only have had an effect if you had set
>>> nntp-connection-timeout to something other than the default of nil.
>>
>> I've never touched nntp-connection-timeout, I only became aware of it
>> when you mentioned it in this thread.  Also, October 2021 is too early:
>> the early to mid December date stands out to me because in early
>> December of that year there was a change in my internet service, and I
>> started experiencing the Gnus problem soon thereafter.  At first I
>> thought it might be due to my change in internet service, but the
>> problem really was and remains only with fetching news with Gnus (and
>> not with fetching email with Gnus, which I do from different mail
>> servers).  I also only use news.gmane.io for fetching news, so that's
>> why I thought maybe something changed there.
>
> It's entirely possible! The buggy fix was reverted in early May, so if
> you stopped seeing the bad behavior then, that would be pretty conclusive.

Unfortunately, I can conclusively state that I continue to get the hangs
to this day (and the sightings reported by Prashant Tak and Eric Fraga
were also just within the last week or so).

Steve Berman

Re: Gnus fetch freezes emacs

[Bob Newell]

> So this works, in the sense that it stops me waiting forever... However,
> it seems (early days yet) that when it fails to open the connection to
> an NNTP server, it stops retrieving news and I have to hit 'g' again to
> get the counts etc. updated for other servers.  But much better than
> waiting forever.

Interestingly (to me at least!) is that I rarely encounter a
hang while fetching, although at times it can be terribly
slow--- I fetch via IMAP and just from gmail, and I blame gmail
for this for the most part.

But what I do encounter from time to time is a hang when
sending.  I'll wait, press ctrl-g, and find the email has been
sent, but control was never returned.  This is with SMTP via
msmtp.  I've never tried to track it down as it isn't frequent
enough to be more than a small nuisance.

-- 
Bob Newell
Honolulu, Hawai`i

- Via GNU/Linux/Emacs/Gnus/BBDB

Re: Gnus fetch freezes emacs

[Robert Pluim]

>>>>> On Mon, 03 Jul 2023 09:22:38 -1000, Bob Newell <bobnewell@bobnewell.net> said:

    >> So this works, in the sense that it stops me waiting forever... However,
    >> it seems (early days yet) that when it fails to open the connection to
    >> an NNTP server, it stops retrieving news and I have to hit 'g' again to
    >> get the counts etc. updated for other servers.  But much better than
    >> waiting forever.

    Bob> Interestingly (to me at least!) is that I rarely encounter a
    Bob> hang while fetching, although at times it can be terribly
    Bob> slow--- I fetch via IMAP and just from gmail, and I blame gmail
    Bob> for this for the most part.

gmail IMAP can be excruciatingly slow. One mitigation is to set
gmailʼs imap folder size limit to something less than infinity (I use
1000). But I donʼt fetch from gmail to local, I leave everything on
googleʼs servers.

    Bob> But what I do encounter from time to time is a hang when
    Bob> sending.  I'll wait, press ctrl-g, and find the email has been
    Bob> sent, but control was never returned.  This is with SMTP via
    Bob> msmtp.  I've never tried to track it down as it isn't frequent
    Bob> enough to be more than a small nuisance.

I think thereʼs been some work in this area in emacs-29 and/or emacs
master to make the code more robust. You could try using emacsʼs
direct SMTP support, but that might not be a small change in your
setup.

Robert
-- 

Gmail slowness/occasional SMTP hang

[Bob Newell]

>
>     Bob> Interestingly (to me at least!) is that I rarely encounter a
>     Bob> hang while fetching, although at times it can be terribly
>     Bob> slow--- I fetch via IMAP and just from gmail, and I blame gmail
>     Bob> for this for the most part.
>
> gmail IMAP can be excruciatingly slow. One mitigation is to set
> gmailʼs imap folder size limit to something less than infinity (I use
> 1000). But I donʼt fetch from gmail to local, I leave everything on
> googleʼs servers.
>
>     Bob> But what I do encounter from time to time is a hang when
>     Bob> sending.  I'll wait, press ctrl-g, and find the email has been
>     Bob> sent, but control was never returned.  This is with SMTP via
>     Bob> msmtp.  I've never tried to track it down as it isn't frequent
>     Bob> enough to be more than a small nuisance.
>
> I think thereʼs been some work in this area in emacs-29 and/or emacs
> master to make the code more robust. You could try using emacsʼs
> direct SMTP support, but that might not be a small change in your
> setup.

I've changed the topic since this is drifting from the
original subject.

I don't do local fetching either, so to speed up direct IMAP
fetching in gmail, I've set group levels so that on a regular
fetch, only INBOX is checked, rather than everything.  (For
instance the All Mail folder contains tens of thousands of
emails ... I don't want gnus to have to figure that one out
more that once or twice per session!)  Checking only INBOX
helps considerably, but even then at times the gmail IMAP
servers are slow, and gnus can be slow in processing.
(Webmail is orders of magnitude faster, but I prefer to stay
in Emacs.)

On the SMTP side, I've looked into the option you mentioned,
using direct SMTP support rather than relying on msmtp.
There's also the appeal of keeping everything within Emacs,
but I went down the msmtp road long ago because I have dozens
of email accounts and msmtp manages sender identities really
easily.  Switching over is probably desirable but would be many
hours of work.  But nevertheless thank you for the ideas.

-- 
Bob Newell
Honolulu, Hawai`i

- Via GNU/Linux/Emacs/Gnus/BBDB

Re: Gnus fetch freezes emacs

[yeti]

Stephen Berman <stephen.berman@gmx.net> writes:

> This is a very annoying issue, and if what you're experiencing is the
> same, I commiserate with you, but your report also gives me hope that
> it's not just some quirk of my setup or network connection.

Same here.

-- 
Take Back Control!  --  Mesh The Planet!
I do not play Nethack, I do play GNUS!   o;-)
Solid facts do not need 1001 pictures.