nntp problems with hibernating and waking up in a new network

nntp problems with hibernating and waking up in a new network

[Steinar Bang]

Platform: Intel Pentium M, debian sarge
	  linux 2.6.14.3 (built by me, not supplied by debian),
	  GNU Emacs 21.4.1,
	  No Gnus v0.4 (today's CVS)

When I hibernate at home and wake the laptop up in the work network,
or hibernate at work and wake up the laptop at home, and then try to
read mail and news, my nnimap servers work fine.  But the nntp groups
are unable to connect to the servers.  According to ethereal, no
actual nntp traffic takes place.  This is the case with both agentized
and direct nntp servers.

Before hibernating I take Gnus to unplugged.  And I then unplug the
physical network plug to make linux lose its network connection, and
immediately find a new network connection when waking up in the new
location. 

Has anyone else seen similar behaviour?  Does anyone know a trick that
would make it possible for me to avoid having to start a new Emacs and
a new Gnus?

Thanx!


- Steinar

Re: nntp problems with hibernating and waking up in a new network

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

> Platform: Intel Pentium M, debian sarge
> 	  linux 2.6.14.3 (built by me, not supplied by debian),
> 	  GNU Emacs 21.4.1,
> 	  No Gnus v0.4 (today's CVS)
>
> When I hibernate at home and wake the laptop up in the work network,
> or hibernate at work and wake up the laptop at home, and then try to
> read mail and news, my nnimap servers work fine.  But the nntp groups
> are unable to connect to the servers.  According to ethereal, no
> actual nntp traffic takes place.  This is the case with both agentized
> and direct nntp servers.
>
> Before hibernating I take Gnus to unplugged.  And I then unplug the
> physical network plug to make linux lose its network connection, and
> immediately find a new network connection when waking up in the new
> location. 
>
> Has anyone else seen similar behaviour?  Does anyone know a trick that
> would make it possible for me to avoid having to start a new Emacs and
> a new Gnus?

Doesn't 'C' and 'O', in the *Server* buffer, on the particular nntp
server help?  It doesn't solve the problem, but should let you close
the connection and re-open it without restarting Emacs or Gnus.

Re: nntp problems with hibernating and waking up in a new network

[Jesper Harder]

Steinar Bang <sb@dod.no> writes:

> Before hibernating I take Gnus to unplugged.  And I then unplug the
> physical network plug to make linux lose its network connection, and
> immediately find a new network connection when waking up in the new
> location. 
>
> Has anyone else seen similar behaviour?

I've seen something similar on OS X. Gnus sometimes hangs after waking
my powerbook.  This doesn't involve a new network, though -- just
waking up to the same (wireless) network.

Re: nntp problems with hibernating and waking up in a new network

[Steinar Bang]

>>>>> Jesper Harder <jesper.harder@gmail.com>:

> I've seen something similar on OS X. Gnus sometimes hangs after
> waking my powerbook.  This doesn't involve a new network, though --
> just waking up to the same (wireless) network.

Waking up to the same network seems to work for me.  Since my WLAN
card isn't currently in operation I can't test if using a different
address on the same network will work.

Re: nntp problems with hibernating and waking up in a new network

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

> Doesn't 'C' and 'O', in the *Server* buffer, on the particular nntp
> server help?  It doesn't solve the problem, but should let you close
> the connection and re-open it without restarting Emacs or Gnus.

`C' works, but `O' just has the same behaviour I get when going
plugged, ie. a long time out period and then a "Can't open nntp:news,
go offline? (y or n)" in the minibuffer.

This behaviour has been in CVS since at least 21 May.  I know because
that was my previous CVS update, and it had this problem.  I did a new
CVS update to see if it was fixed, before reporting it.

Also note that it doesn't affect nnimap.  I've only observed this
behaviour on nntp.

Re: nntp problems with hibernating and waking up in a new network

[Bjørn Mork]

Steinar Bang <sb@dod.no> writes:

> When I hibernate at home and wake the laptop up in the work network,
> or hibernate at work and wake up the laptop at home, and then try to
> read mail and news, my nnimap servers work fine.  But the nntp groups
> are unable to connect to the servers.  According to ethereal, no
> actual nntp traffic takes place.  This is the case with both agentized
> and direct nntp servers.

The main problem when doing this is that emacs (or libc?) will cache
DNS resolvers for you, and these probably won't work in your new
network environment.  Therefore there is no nntp traffic, only
unanswered DNS queries.

Can't explain why the nnimap sessions work.  Maybe they are using some
external process to set up the connection, e.g a ssh tunnel?

Never found a good solution to the cached DNS resolver problem.  Note
that is isn't confined to emacs.  The same problem applies to most
applications running forever without mechanisms for reloading
/etc/resolv.conf, like most browsers.  

Personally I resorted to running BIND locally on the laptop...


Bjørn
-- 
Let me tell you something, you wimp, you hope that the Earth is flat.

Re: nntp problems with hibernating and waking up in a new network

[Steinar Bang]

>>>>> Bjørn Mork <bmork@dod.no>:

> The main problem when doing this is that emacs (or libc?) will cache
> DNS resolvers for you, and these probably won't work in your new
> network environment.  Therefore there is no nntp traffic, only
> unanswered DNS queries.

I thought it might be something like this, but I had hoped it was in
the lisp code in the nntp backend, since nnimap worked.

However...

> Can't explain why the nnimap sessions work.  Maybe they are using
> some external process to set up the connection, e.g a ssh tunnel?

That is indeed the case.  That is, not an SSH tunnel, but as NNIMAPS
using the "openssl" executable.

> Never found a good solution to the cached DNS resolver problem.
> Note that is isn't confined to emacs.  The same problem applies to
> most applications running forever without mechanisms for reloading
> /etc/resolv.conf, like most browsers.

Actually, my browsers survive.  I used to have a problem with a
similar situation in Opera, when I went on VPN using PPTP, and back
(this changed the /etc/resolv.conf file but the browser didn't keep
track). 

But with recent versions of Opera, this problem has gone away.

> Personally I resorted to running BIND locally on the laptop...

That won't work for me, because this would confuse the PPTP client, I
think.  Or perhaps I could just list all DNS clients at work and at
home, and let BIND sort them out?

But if so, what happens when I go online in a different network where
I'm passed a name server with the DHCP lease?

I'm thinking of a simple workaround, which would be to use an
intermediate connection like nntp-open-via-rlogin-and-telnet or
something.  But instead of connecting to a remote machine, just use
the current machine.

The idea here is to "misuse" the indirection to fork off a new process
that would actually read /etc/resolv.conf.

We'll see.

Re: nntp problems with hibernating and waking up in a new network

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> I'm thinking of a simple workaround, which would be to use an
> intermediate connection like nntp-open-via-rlogin-and-telnet or
> something.  But instead of connecting to a remote machine, just use
> the current machine.

> The idea here is to "misuse" the indirection to fork off a new process
> that would actually read /etc/resolv.conf.

Looks like nntp-open-telnet-stream is a better candidate than
nntp-open-via-rlogin-and-telnet and friends.  I think I'll try this.

Re: nntp problems with hibernating and waking up in a new network

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

>>>>>> Simon Josefsson <jas@extundo.com>:
>
>> Doesn't 'C' and 'O', in the *Server* buffer, on the particular nntp
>> server help?  It doesn't solve the problem, but should let you close
>> the connection and re-open it without restarting Emacs or Gnus.
>
> `C' works, but `O' just has the same behaviour I get when going
> plugged, ie. a long time out period and then a "Can't open nntp:news,
> go offline? (y or n)" in the minibuffer.
>
> This behaviour has been in CVS since at least 21 May.  I know because
> that was my previous CVS update, and it had this problem.  I did a new
> CVS update to see if it was fixed, before reporting it.
>
> Also note that it doesn't affect nnimap.  I've only observed this
> behaviour on nntp.

If you remove the " *nntp*server" buffer, perhaps it will work?  If
you (setq debug-on-error t) and C-g the 'O', where is it stuck?
Perhaps strace or gdb would help too.

Re: nntp problems with hibernating and waking up in a new network

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

[snip! Workaround for emacs DNS caching problem]
> Looks like nntp-open-telnet-stream is a better candidate than
> nntp-open-via-rlogin-and-telnet and friends.  I think I'll try this.

It sort of works until a group is entered, and an attempt to visit an
article is made.

What happens then is that a message like eg.
 Fetching articles for nntp+news.gmane.org:gmane.emacs.gnus.general...

is written to the minibuffer, and then nothing happens for a loong
time.  Then the article appears but the message in the minibuffer
persists, but the user interface is frozen.  I can get control over
emacs/gnus again, by pressing `C-g'.

What's written to *Messages* when entering this group, and then trying
to read an article, is:

Retrieving newsgroup: nntp+news.gmane.org:gmane.emacs.gnus.general...
Fetching headers for nntp+news.gmane.org:gmane.emacs.gnus.general...done
Scoring...done
Generating summary...done
Fetching articles for nntp+news.gmane.org:gmane.emacs.gnus.general...

Also, I can't post.  When I tried posting this article, I got the
message 
 441 Required "Subject" header is missing 
in the minibuffer.

Re: nntp problems with hibernating and waking up in a new network

[Adrian Aichner]

Simon Josefsson <jas@extundo.com> writes:

> Steinar Bang <sb@dod.no> writes:
>
>>>>>>> Simon Josefsson <jas@extundo.com>:
>>
>>> Doesn't 'C' and 'O', in the *Server* buffer, on the particular nntp
>>> server help?  It doesn't solve the problem, but should let you close
>>> the connection and re-open it without restarting Emacs or Gnus.
>>
>> `C' works, but `O' just has the same behaviour I get when going
>> plugged, ie. a long time out period and then a "Can't open nntp:news,
>> go offline? (y or n)" in the minibuffer.
>>
>> This behaviour has been in CVS since at least 21 May.  I know because
>> that was my previous CVS update, and it had this problem.  I did a new
>> CVS update to see if it was fixed, before reporting it.
>>
>> Also note that it doesn't affect nnimap.  I've only observed this
>> behaviour on nntp.
>
> If you remove the " *nntp*server" buffer, perhaps it will work?  If
> you (setq debug-on-error t) and C-g the 'O', where is it stuck?
> Perhaps strace or gdb would help too.

At least under XEmacs
(setq debug-on-quit t)
would be an even better choice.

Anytime you wonder where it's hung, just use
C-g (keyboard-quit)

-- 
Adrian Aichner
 mailto:adrian@xemacs.org
 http://www.xemacs.org/

Re: nntp problems with hibernating and waking up in a new network

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

> If you remove the " *nntp*server" buffer, perhaps it will work? 

No.  If I come out of hibernate and toggle plugged on Gnus and it gets
to go to time out on all nntp servers, there are no " *nntp*..."
buffers when it has finished.

> If you (setq debug-on-error t) and C-g the 'O', where is it stuck?
> Perhaps strace or gdb would help too.

I think that Bjørn Mork's suggestion otherwise in the thread about
Emacs caching the DNS server is the problem.  So the question is what
to do about it.

Adding
 (nntp-open-connection-function nntp-open-telnet-stream)
to nntp servers, work for going plugged, and for doing `g' in the
*Group* buffer.  But fails when fetching the articles, and _really_
fails when posting (see my discussion with myself in a different place
in this thread).

The idea behind doing this is to fork off a different process for
connecting to NNTP servers, because this process will use the current
/etc/resolv.conf nameserver, and not the one that emacs has cached,
which was the one active when emacs was started.

Re: nntp problems with hibernating and waking up in a new network

[Steinar Bang]

>>>>> Adrian Aichner <adrian@xemacs.org>:

> At least under XEmacs
> (setq debug-on-quit t)
> would be an even better choice.

> Anytime you wonder where it's hung, just use
> C-g (keyboard-quit)

Got no backtrace on `C-g' in GNU Emacs.  I guess that means this
happens inside a `condition-case'...?

There are 5 lines matching "condition-case" in buffer nntp.el:
    441:          (condition-case err
    640:                          (condition-case nil
   1201:	  (condition-case ()
   2026:    (condition-case err
   2046:	(condition-case err

The one on line 1201 is inside the function nntp-open-connection,
which looks suspiciously like the one that's called when Gnus starts
talking to the NNTP server.  That one calls nntp-open-network-stream
(in the default case), which calls the emacs built in
open-network-stream, which is where I'm guessing it is hanging.

Re: nntp problems with hibernating and waking up in a new network

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> Steinar Bang <sb@dod.no>:
> [snip! Workaround for emacs DNS caching problem]
>> Looks like nntp-open-telnet-stream is a better candidate than
>> nntp-open-via-rlogin-and-telnet and friends.  I think I'll try this.

> It sort of works until a group is entered, and an attempt to visit an
> article is made.

> What happens then is that a message like eg.
>  Fetching articles for nntp+news.gmane.org:gmane.emacs.gnus.general...
> is written to the minibuffer, and then nothing happens for a loong
> time.  Then the article appears but the message in the minibuffer
> persists, but the user interface is frozen.  I can get control over
> emacs/gnus again, by pressing `C-g'.
[snip!]
> Also, I can't post.  When I tried posting this article, I got the
> message 
>  441 Required "Subject" header is missing 
> in the minibuffer.

google didn't find anything relevant, but Gnus Xapian did! :-)
	http://article.gmane.org/gmane.emacs.gnus.general/54809

This seems to be working...
 	(nntp "news.gmane.org"
	      (nntp-open-connection-function nntp-open-telnet-stream)
	      (nntp-end-of-line "\n"))
(that is, the nntp-end-of-line setting fixed the open article problem
and if this post goes out, it fixed the 441 problem...)

Re: nntp problems with hibernating and waking up in a new network

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

>>>>>> Adrian Aichner <adrian@xemacs.org>:
>
>> At least under XEmacs
>> (setq debug-on-quit t)
>> would be an even better choice.
>
>> Anytime you wonder where it's hung, just use
>> C-g (keyboard-quit)
>
> Got no backtrace on `C-g' in GNU Emacs.  I guess that means this
> happens inside a `condition-case'...?

You can (setq debug-on-signal t) in this case, but you'll have to
press 'c' for every signal that is caught, and figure out which
signals are harmless.

> There are 5 lines matching "condition-case" in buffer nntp.el:
>     441:          (condition-case err
>     640:                          (condition-case nil
>    1201:	  (condition-case ()
>    2026:    (condition-case err
>    2046:	(condition-case err
>
> The one on line 1201 is inside the function nntp-open-connection,
> which looks suspiciously like the one that's called when Gnus starts
> talking to the NNTP server.  That one calls nntp-open-network-stream
> (in the default case), which calls the emacs built in
> open-network-stream, which is where I'm guessing it is hanging.

Try adding `(message "foo")' etc, or use edebug, or remove the
condition-case to see where it blows up.

Re: nntp problems with hibernating and waking up in a new network

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

> Try adding `(message "foo")' etc, or use edebug, or remove the
> condition-case to see where it blows up.

Hm... I've found a working workaround for the problem (wrap all nntp
connections in telnet).  Do you still need/want me to debug this?

I haven't verified that a cached DNS server is the problem.  But the
delay for each NNTP server, fits gethostbyname() time out.

And if this _is_ the problem, then the problem is outside of Gnus, and
even outside of Emacs.  It is a libc problem, and as such probably
best just worked around...

Re: nntp problems with hibernating and waking up in a new network

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

>>>>>> Simon Josefsson <jas@extundo.com>:
>
>> Try adding `(message "foo")' etc, or use edebug, or remove the
>> condition-case to see where it blows up.
>
> Hm... I've found a working workaround for the problem (wrap all nntp
> connections in telnet).  Do you still need/want me to debug this?

No.

> I haven't verified that a cached DNS server is the problem.  But the
> delay for each NNTP server, fits gethostbyname() time out.
>
> And if this _is_ the problem, then the problem is outside of Gnus, and
> even outside of Emacs.  It is a libc problem, and as such probably
> best just worked around...

Yes, probably.

gmane spam reporting problems moving between networks (Was: nntp problems with hibernating and waking up in a new network)

[Steinar Bang]

The message I'm responding to, is to the head of a thread dealing with
problems I had with NNTP servers when hibernating and moving my laptop
between home and work.  I couldn't connect to the server, because Gnus
couldn't resolve their names.

Now I'm seeing the same problem when exiting a gmane group and trying
to report spam in that group: Gnus can't find spam.gmane.org, which is
perfectly pingable from a shell.  

The NNTP problem was most probably caused by libc caching the DNS
server to use for a process, so that gethostbyname() would issue calls
to a DNS server that was no longer available.  I got around this
problem by wrapping the NNTP connections in telnet sessions (new
telnet processes would get the current DNS server).

Is it possible to do something similar with the gmane spam reporting?
Either wrap the HTTP requests done from Emacs in a telnet, or use curl
or wget, or something?

Re: gmane spam reporting problems moving between networks

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

> The message I'm responding to, is to the head of a thread dealing with
> problems I had with NNTP servers when hibernating and moving my laptop
> between home and work.  I couldn't connect to the server, because Gnus
> couldn't resolve their names.
>
> Now I'm seeing the same problem when exiting a gmane group and trying
> to report spam in that group: Gnus can't find spam.gmane.org, which is
> perfectly pingable from a shell.  
>
> The NNTP problem was most probably caused by libc caching the DNS
> server to use for a process, so that gethostbyname() would issue calls
> to a DNS server that was no longer available.  I got around this
> problem by wrapping the NNTP connections in telnet sessions (new
> telnet processes would get the current DNS server).
>
> Is it possible to do something similar with the gmane spam reporting?
> Either wrap the HTTP requests done from Emacs in a telnet, or use curl
> or wget, or something?

Wouldn't running a DNS server on localhost work?  Then you'd have the
same DNS server address regardless of IP of your local machine.

Re: gmane spam reporting problems moving between networks (Was: nntp problems with hibernating and waking up in a new network)

[Steinar Bang]

For those who may be interested: The problems indicated by the
subjects of this message, have gone away with Ubuntu 6.06 Dapper Drake
(they were present in Ubuntu 5.10 Breezy Badger).

To summarize the two old threads:
 I had a problem connecting to nntp servers and and to nnimap servers
 that weren't secure, after unplugging Gnus, hibernating the laptop,
 and waking it up again in a different network and plugging in Gnus.

It was determined that this probably was caused by GNU Emacs/libc
caching the DNS server present when Emacs was started.  That's why
secure imap worked, because that backend talked to the server through
a separate process that was started everytime Gnus connected to the
server. 

This gave me a workaround for nntp servers, ie. I could do stuff like
this in the server settings:
  	(nntp "news.gmane.org"
 	      (nntp-open-connection-function nntp-open-telnet-stream)
 	      (nntp-end-of-line "\n"))

Later I noticed a related problem in the spam reporting for gmane (the
subject of this thread).  It only worked in the network the Emacs
process was originally started in (or at least with the DNS server
having the same address as in that network).

During the summer I suddenly discovered that the problem no longer
seemed to be present for Gmane spam reporting, and for unsecure
nnimap. 

So now I've removed the nntp workarounds as well, and it works
perfectly. 

I saw this problem on a previous laptop with debian sarge, and I saw
it on the current laptop with Ubuntu 5.10 Breezy Badger.

Sarge has the following versions of libc and GNU Emacs:
 libc6:		2.3.2.ds1-22sarge3
 emacs21: 	21.4a-1

Breezy has the following versions of libc and GNU Emacs:
 libc6:		2.3.5-1ubuntu12
 emacs21: 	21.4a-1ubuntu1

Dapper has the following versions of libc and GNU Emacs:
 libc6:		2.3.6-0ubuntu20
 emacs21: 	21.4a-3ubuntu2

The Emacs version seems to be the same all around, so presumably the
big difference is in GNU libc...?  But whether it has been fixed in
the upstream libc or in Ubuntu patches, I have no idea.