POP3 fragile?

POP3 fragile?

[Florian Weimer]

Well, I've just lost ~200 messages. *sigh*

It seems that Emacs 21.3 (or Gnus 5.10.3) can't cope with network
outages during a POP3 download.  Furthermore, from time to time during
POP3 download, Emacs runs into an infinite loop, eating 100% CPU.

Re: POP3 fragile?

[Marcelo Toledo]

Could you please elaborate more?

Florian Weimer <fw@deneb.enyo.de> writes:

> Well, I've just lost ~200 messages. *sigh*
>
> It seems that Emacs 21.3 (or Gnus 5.10.3) can't cope with network
> outages during a POP3 download.  Furthermore, from time to time during
> POP3 download, Emacs runs into an infinite loop, eating 100% CPU.

-- 
Marcelo Toledo
mailto:rw@locked.org
http://rw.locked.org
Mobile: 55 71 9141-7181

Re: POP3 fragile?

[Xavier Maillard]

[-- Attachment #1: Type: text/plain, Size: 889 bytes --]

On 14 jui 2003, Florian Weimer verbalised:

>  Well, I've just lost ~200 messages. *sigh*

Oups ;)
  
>  It seems that Emacs 21.3 (or Gnus 5.10.3) can't cope with network
>  outages during a POP3 download.  Furthermore, from time to time
>  during POP3 download, Emacs runs into an infinite loop, eating 100%
>  CPU.

Not enough informations to provide help on that ! can you give us more
informations ?

AFAIK, I didn't get that since I use Gnus. I frequently hit g and get
about 1000 messages to download and no problems here. 

Concerning the 100% CPU problem, I get that when Gnus is splitting a
huge number of messages. But it only happens at the splitting's
beginning (about 2 or 3 seconds long) and then all is going right.

zeDek
-- 
http://www.gnusfr.org -- French Gnus user site

Anti-war disclaimer:
	"Bombing for peace is like fucking for virginity"

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]

Re: POP3 fragile?

[Florian Weimer]

Xavier Maillard <zedek@gnu-rox.org> writes:

> Not enough informations to provide help on that ! can you give us more
> informations ?
>
> AFAIK, I didn't get that since I use Gnus. I frequently hit g and get
> about 1000 messages to download and no problems here. 

Do you press C-g during download?

In my case, the TCP connection hung (because IP connectivity went
away) and I had to press C-g.

> Concerning the 100% CPU problem, I get that when Gnus is splitting a
> huge number of messages.

In my case, it's spinning in a tight loop around the select() call,
and no data is processed.

Re: POP3 fragile?

[Mike Woolley]

Florian Weimer <fw@deneb.enyo.de> writes:

> Do you press C-g during download?
>
> In my case, the TCP connection hung (because IP connectivity went
> away) and I had to press C-g.

This has happened to me a few times whilst using my dialup at home. I
haven't lost mail (yet) though. Usually what happens is that I get the
interrupted mails again, as they weren't deleted from the server the
first time...

> In my case, it's spinning in a tight loop around the select() call,
> and no data is processed.

This also happens to me quite regularly on my permanent connection in
the office, where I have Gnus get new mail every 5 minutes. A couple
of times a week I find it's hung and I can't C-g out of it. I have to
kill the process with Task Manager (I'm using Windows 2K) to get out
of it. Gnus 5.8 and Emacs 20.x also exhibited this behaviour, although
it seems to happen more frequently since I switched to Emacs 21.

Unfortunately, I haven't got very far in working out why it's
happening though...

Cheers,
Mike

Re: POP3 fragile?

[Stainless Steel Rat]

* Florian Weimer <fw@deneb.enyo.de>  on Tue, 15 Jul 2003
| In my case, the TCP connection hung (because IP connectivity went
| away) and I had to press C-g.

The POP3 specifications state that the server must not process DELE marks
if the connection closes (or drops) without the client issuing a QUIT
command.  Your "lost" mail should still be on the server.

-- 
Rat <ratinox@peorth.gweep.net>    \ When not in use, Happy Fun Ball should be
Minion of Nathan - Nathan says Hi! \ returned to its special container and
PGP Key: at a key server near you!  \ kept under refrigeration.
       That and five bucks will get you a small coffee at Starbucks.

Re: POP3 fragile?

[Florian Weimer]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> The POP3 specifications state that the server must not process DELE
> marks if the connection closes (or drops) without the client issuing
> a QUIT command.  Your "lost" mail should still be on the server.

Hmm.  Perhaps the connection wasn't really stuck and the QUIT that C-g
triggered was actually transmitted to the server?

The mail in question was *not* on the server.  I'd peeked at the
mailbox in advance, and I know some of the messages which had been
discarded.

Maybe the SSH tunnel is causing the problems: something is stalled, I
press C-g, and the server assumes that the messages up to this point
have been sucessfully transmitted.  Actually, this *could* be the
problem if individual messages don't have to be acknowledged at the
POP layer.

Re: POP3 fragile?

[Stainless Steel Rat]

* Florian Weimer <fw@deneb.enyo.de>  on Tue, 15 Jul 2003
| Hmm.  Perhaps the connection wasn't really stuck and the QUIT that C-g
| triggered was actually transmitted to the server?

That seems not possible.  C-g halts running Emacs-Lisp forms, right?  So,
hitting C-g should stop any form that might try to send the "QUIT<RET>"
string to the server, right?

[snip]

| Maybe the SSH tunnel is causing the problems: something is stalled, I
| press C-g, and the server assumes that the messages up to this point
| have been sucessfully transmitted.  Actually, this *could* be the
| problem if individual messages don't have to be acknowledged at the
| POP layer.

POP is not supposed to work that way.  The client can send DELE commands to
the server, and the server will keep track of them, but the DELE marks
are not supposed to be processed until the client sends the QUIT command
AND the connection closes gracefully.  Anything else, the POP server is
supposed to assume that something failed and clear the DELE marks so that
no mail is lost.

If your POP server does not work that way then it is way, way, way out of
spec, and you should be addressing this to your systems administrators.

-- 
Rat <ratinox@peorth.gweep.net>    \ Warning: pregnant women, the elderly, and
Minion of Nathan - Nathan says Hi! \ children under 10 should avoid prolonged
PGP Key: at a key server near you!  \ exposure to Happy Fun Ball.
       That and five bucks will get you a small coffee at Starbucks.

Re: POP3 fragile?

[Florian Weimer]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> * Florian Weimer <fw@deneb.enyo.de>  on Tue, 15 Jul 2003
> | Hmm.  Perhaps the connection wasn't really stuck and the QUIT that C-g
> | triggered was actually transmitted to the server?
>
> That seems not possible.  C-g halts running Emacs-Lisp forms, right?  So,
> hitting C-g should stop any form that might try to send the "QUIT<RET>"
> string to the server, right?

Does unwind-protect catch C-g?

> POP is not supposed to work that way.  The client can send DELE commands to
> the server, and the server will keep track of them, but the DELE marks
> are not supposed to be processed until the client sends the QUIT command
> AND the connection closes gracefully.

I imagined that the client might send a QUIT while many, many messages
are still buffered at the transport layer.  However, this is
impossible; the protocol ensures implicit resynchronizes before each
messages.

Strange.

Re: POP3 fragile?

[Stainless Steel Rat]

* Florian Weimer <fw@deneb.enyo.de>  on Tue, 15 Jul 2003
| Does unwind-protect catch C-g?

Catch as in "mask the interrupt"?  I believe not.
Even if it does, the worst that can happen with stock pop3.el is that
you'll have a bunch of stuff in the crashbox and copies of those messages
on the server.

| I imagined that the client might send a QUIT while many, many messages
| are still buffered at the transport layer.  However, this is
| impossible; the protocol ensures implicit resynchronizes before each
| messages.

Exactly.  The POP protocols were designed to be both simple and robust
enough for the dumbest of MS-DOS mail clients.  They are quite tolerant of
client and link failures of various sorts.  As I said, the catastrophic
loss you experienced should not be possible unless the server is doing
something very badly broken.

Recent versions of qpopper can be configured to automatically issue DELE
commands after each successful RETR, to foil (ab)use of "leave mail on
server".  But even so, messages are not purged unless the client explicitly
issues a QUIT command and closes the session gracefully.  It may be that
you have a hacked POP server that expunges messages immediately after a
successful RETR.

-- 
Rat <ratinox@peorth.gweep.net>    \ Warning: pregnant women, the elderly, and
Minion of Nathan - Nathan says Hi! \ children under 10 should avoid prolonged
PGP Key: at a key server near you!  \ exposure to Happy Fun Ball.
       That and five bucks will get you a small coffee at Starbucks.

Re: POP3 fragile?

[Florian Weimer]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> * Florian Weimer <fw@deneb.enyo.de>  on Tue, 15 Jul 2003
> | Does unwind-protect catch C-g?
>
> Catch as in "mask the interrupt"?  I believe not.
> Even if it does, the worst that can happen with stock pop3.el is that
> you'll have a bunch of stuff in the crashbox and copies of those messages
> on the server.

Are you sure?  I read 'pop3-movemail' differently, IMHO "QUIT" is
always sent to the server (look at the placement of the call to
'pop3-quit').

> Recent versions of qpopper can be configured to automatically issue DELE
> commands after each successful RETR, to foil (ab)use of "leave mail on
> server".  But even so, messages are not purged unless the client explicitly
> issues a QUIT command and closes the session gracefully.  It may be that
> you have a hacked POP server that expunges messages immediately after a
> successful RETR.

It's an qpopper 4.0.5, unpatched, unconfigured, running on FreeBSD.

Re: POP3 fragile?

[Stainless Steel Rat]

* Florian Weimer <fw@deneb.enyo.de>  on Tue, 15 Jul 2003
| Are you sure?  I read 'pop3-movemail' differently, IMHO "QUIT" is
| always sent to the server (look at the placement of the call to
| 'pop3-quit').

Positive.  I wrote it, after all :).

I checked; unwind-protect does not mask C-g.  If you press C-g while inside
pop3-movemail before pop3-quit then pop3-quit will not be evaluated and
QUIT will not be sent.  pop3-quit will also not be reached if the server
reports an error during the retrieval process (returns a -ERR code).
Either of these conditions will force a halt.  There will almost certainly
be a message sitting in the retr process buffer or the crashbox staging
buffer, but if so then the message will not be marked for deletion and
should remain on the server.  All previous messages will be in the
crashbox.

All of this applies to stock pop3.el.  I cannot say about the FSF or XEmacs
versions; I don't understand most of the changes in those two forks.

-- 
Rat <ratinox@peorth.gweep.net>    \ Ingredients of Happy Fun Ball include an
Minion of Nathan - Nathan says Hi! \ unknown glowing substance which fell to
PGP Key: at a key server near you!  \ Earth, presumably from outer space.
       That and five bucks will get you a small coffee at Starbucks.

Re: POP3 fragile?

[Marcelo Toledo]

You probably didnt lost your messages at that time, it was probably
stored in your crashbox (~/.crashbox), but im sure you already retrieved
mail again right? 

this problem is hapenning all the time?

Florian Weimer <fw@deneb.enyo.de> writes:

> In my case, the TCP connection hung (because IP connectivity went
> away) and I had to press C-g.
>
> In my case, it's spinning in a tight loop around the select() call,
> and no data is processed.

-- 
Marcelo Toledo
mailto:rw@locked.org
http://www.marcelotoledo.org
Mobile: 55 71 9141-7181

Re: POP3 fragile?

[Kai Großjohann]

Florian Weimer <fw@deneb.enyo.de> writes:

> Well, I've just lost ~200 messages. *sigh*
>
> It seems that Emacs 21.3 (or Gnus 5.10.3) can't cope with network
> outages during a POP3 download.  Furthermore, from time to time during
> POP3 download, Emacs runs into an infinite loop, eating 100% CPU.

Do you still have them in ~/.emacs-mail-crashbox (or similar), or in
~/Mail/Incoming4711?

Bad situation.  I sympathize.
-- 
~/.signature

Re: POP3 fragile?

[Florian Weimer]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Do you still have them in ~/.emacs-mail-crashbox (or similar),

No, I don't, Gnus seems to clean up and process mail if it sees these
files during startup.

> or in ~/Mail/Incoming4711?

I'm not running a development version right now.

Hmm.  I never lost mail when I used all those unstable alpha versions.
Life is not fair.

Re: POP3 fragile?

[Florian Weimer]

Okay, I think I understand the problem now.

Both Gnus and qpopper do not rollback a POP3 transaction which
completes abnormally.  Obviously, there is plenty oppertunity for mail
loss, and if I fix the server side, dupes.

Re: POP3 fragile?

[Matthias Andree]

Florian Weimer <fw@deneb.enyo.de> writes:

> Both Gnus and qpopper do not rollback a POP3 transaction which
> completes abnormally.

So ditch qpopper.

Your POP3 grinder mustn't commit DELE before QUIT in the first place, so
there's no need to unroll anything (if the POP3 server stores the DELE
on disk, outside the swap area, the server is BBD, broken by design.)

qpopper hasn't a very clean security record, so...

> Obviously, there is plenty oppertunity for mail loss, and if I fix the
> server side, dupes.

...there may be opportunities to have some fun as well. But I am
preaching to the choir, so I'll quit for now.

-- 
Matthias Andree