missing POP3 mail

missing POP3 mail

[Andrew Hilborne]

I'm running stock pgnus-0.97. Yesterday I lost some POP3 mail, and had to run
nnml-generate-nov-databases-1 in order to recover some other messages which
hadn't appeared in the appropriate groups.

These are my mail sources:

    (setq gnus-secondary-select-methods
            '((nnml "private" (nnmail-cache-accepted-message-ids t))
              (nndraft "private")))
    (setq mail-sources
          '((file :path "/var/mail/me")
            (pop :server "pop.isp.net"
                 :user "myid"
                 :password "somepass")))

The POP3 box is served by qpopper version 2.52. Sometimes, when the POP3 box is 
locked, qpopper will respond with a message (don't know number which and can't
replicate from here) saying to try again later.

Yesterday during the normal POP3 collection run this message was reproduced by
the fetchmail process in the minibuffer, and I was asked if I wished to
continue or not (yes/no.) I don't know what I typed, but I then lost POP3 mail, 
and some other mail which had not come through POP3 wasn't registered properly
in the appropriate NOV database. I use fancy-split. The missing POP3 mail
wasn't in the Incoming.* files. In fact I have lost it permanently :-(

Is this expected behaviour? Is this my fault for using beta software? What can
I do to prevent it happening again? All advice is appreciated...

Secondary question: how can I attach my POP3 server to a single group/set of
groups and only check it sometimes?

TIA.

--
Andrew Hilborne

Re: missing POP3 mail

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Andrew Hilborne <andrew.hilborne@gxn.net>  on Tue, 14 Dec 1999
| The POP3 box is served by qpopper version 2.52. Sometimes, when the POP3
| box is locked, qpopper will respond with a message (don't know number
| which and can't replicate from here) saying to try again later.

The number is not particularly relevant except within qpopper, but this is
otherwise normal behaviour for a locked drop.

| Yesterday during the normal POP3 collection run this message was
| reproduced by the fetchmail process in the minibuffer, and I was asked if
| I wished to continue or not (yes/no.) I don't know what I typed, but I
| then lost POP3 mail, and some other mail which had not come through POP3
| wasn't registered properly in the appropriate NOV database. I use
| fancy-split. The missing POP3 mail wasn't in the Incoming.* files. In
| fact I have lost it permanently :-(

What do you mean by "lost"?  If pop3 cannot open the drop because it is
locked, it cannot do anything at all to the drop.  If you continue, it
tells Gnus to skip that method, so again there is nothing being done on
your end as far as the POP server is concerned.  Are you *sure* the mail is
not still sitting on the server somewhere?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.0g (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE4VnhPgl+vIlSVSNkRAtyyAKDetmGwD1Qfot1lAoMQtMr2FcQtmwCeICwY
Z5N6uLRDpgWoLtv+ighN18U=
=bwxh
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Warning: pregnant women, the elderly, and
Minion of Nathan - Nathan says Hi! \ children under 10 should avoid prolonged
PGP Key: at a key server near you!  \ exposure to Happy Fun Ball.

Re: missing POP3 mail

[Andrew Hilborne]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> | Yesterday during the normal POP3 collection run this message was
> | reproduced by the fetchmail process in the minibuffer, and I was asked if
> | I wished to continue or not (yes/no.) I don't know what I typed, but I
> | then lost POP3 mail, and some other mail which had not come through POP3
> | wasn't registered properly in the appropriate NOV database. I use
> | fancy-split. The missing POP3 mail wasn't in the Incoming.* files. In
> | fact I have lost it permanently :-(
> 
> What do you mean by "lost"?  If pop3 cannot open the drop because it is
> locked, it cannot do anything at all to the drop.  If you continue, it
> tells Gnus to skip that method, so again there is nothing being done on
> your end as far as the POP server is concerned.  Are you *sure* the mail is
> not still sitting on the server somewhere?

It is lost: the POP3 box is empty:

    $ telnet pop3.isp.net 110
    Trying 1.2.3.4...
    Connected to pop3.isp.net.
    Escape character is '^]'.
    +OK QPOP (version 2.52) at pop3.isp.net starting.  
    user me
    +OK Password required for me.
    pass somepass
    +OK me has 0 messages (0 octets).
    quit
    +OK Pop server at pops3.isp.net signing off.
    Connection closed by foreign host.
    $

I have the list of message-ids for mail which was delivered to the box in the
relevant time interval, from the box administrators. Most of these have not
been delivered to me.

--
Andrew Hilborne

Re: missing POP3 mail

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Andrew Hilborne <andrew.hilborne@gxn.net>  on Tue, 14 Dec 1999
| I have the list of message-ids for mail which was delivered to the box in the
| relevant time interval, from the box administrators. Most of these have not
| been delivered to me.

Have they looked around for stray files in the mail spool directory?  This
does not sound like anything pop3.el could do unless the POP server also
failed somehow.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.0g (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE4Vqlcgl+vIlSVSNkRAmJaAJ9mncSddYQ7GwbjuRPvRN9NXImbogCffYPP
UbHywF0QJVi1uVVOAeCpVpY=
=4hYB
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Caution: Happy Fun Ball may suddenly
Minion of Nathan - Nathan says Hi! \ accelerate to dangerous speeds.
PGP Key: at a key server near you!  \ 

Re: missing POP3 mail

[Andrew Hilborne]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> * Andrew Hilborne <andrew.hilborne@gxn.net>  on Tue, 14 Dec 1999
> | I have the list of message-ids for mail which was delivered to the box in the
> | relevant time interval, from the box administrators. Most of these have not
> | been delivered to me.
> 
> Have they looked around for stray files in the mail spool directory?  This
> does not sound like anything pop3.el could do unless the POP server also
> failed somehow.

Quite possibly. But I worked with this particular _instance_ of qpopper for
years, used by thousands of dial-up and other users, and we suffered no such
problems. This is in contrast to pop3d which we used before that, which was
horrible.

--
Andrew Hilborne

Re: missing POP3 mail

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Andrew Hilborne <andrew.hilborne@gxn.net>  on Wed, 15 Dec 1999
| Quite possibly. But I worked with this particular _instance_ of qpopper for
| years, used by thousands of dial-up and other users, and we suffered no such
| problems. This is in contrast to pop3d which we used before that, which was
| horrible.

Well, like I said, if qpopper has the drop locked as you described, there
is nothing pop3.el or any POP client can do to it.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.0g (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE4V7acgl+vIlSVSNkRAkL4AKC2Hr8kfxFBkUkeblGhQZ/WyI1eLQCfbOGP
B4ZTejfBOVdqLUl0F5hCbJk=
=D7Be
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ If Happy Fun Ball begins to smoke, get
Minion of Nathan - Nathan says Hi! \ away immediately. Seek shelter and cover
PGP Key: at a key server near you!  \ head.

Re: missing POP3 mail

[Andrew Hilborne]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> * Andrew Hilborne <andrew.hilborne@gxn.net>  on Wed, 15 Dec 1999
> | Quite possibly. But I worked with this particular _instance_ of qpopper for
> | years, used by thousands of dial-up and other users, and we suffered no such
> | problems. This is in contrast to pop3d which we used before that, which was
> | horrible.
> 
> Well, like I said, if qpopper has the drop locked as you described, there
> is nothing pop3.el or any POP client can do to it.

This is not the issue. The issue is my suspicion that pop3.el has failed in its 
duties (the overriding one being *never* to lose mail) in the face of this
behaviour on the part of qpopper, and possibly unexpected behaviour on my own
part.

Anyway, I can't prove any of this at this time, and clearly noone else knows
anything definitive about such a problem. So I'm going to shut up, until I have 
time to try to reproduce it.

--
Andrew Hilborne

Re: missing POP3 mail

[Kai Großjohann]

Might it be the infamous terminating empty line thing again?  Gnus
calls movemail (or the corresponding function) which should produce a
file or a buffer.  Gnus then inspects that to see which format it has
got.  If Gnus thinks it is mbox format, then it thinks that every
message must start with `From ' at the beginning of a line AND END
WITH AN EMPTY LINE.

It would be conceivable that Gnus sees an mbox file which does not
have the terminating empty line.  In this case, Gnus would treat
several messages as one message (until the next empty line).

How could the missing empty line happen?  Well, the POP3 protocol
specifies that each message be terminated by `.' on a line by itself;
a terminating empty line is not required.  So if you concat two
messages as output by POP3, the empty line might not be there.

I think I used to have a long discussion with Rat about this, but I
don't know what became of it.

Can you show us what does your POP3 server output when you fetch a
message?  And what does the Incoming file look like which is generated
by Gnus?  (Set mail-source-delete-incoming to nil to see.)  Try with
messages with and without terminating empty lines (before the `.' in
the POP output).

kai
-- 
A preposition is not a good thing to end a sentence with.

Re: missing POP3 mail

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann)  on Tue, 21 Dec 1999
| I think I used to have a long discussion with Rat about this, but I
| don't know what became of it.

pop3-munge-message-separator, which does its best to generate a
syntactically valid `From ' line, with required blank line, and a valid
Content-Length header if no obvious message separator can be found.  By
"obvious" I mean mbox, Babyl, or MMDF.  I think the Content-Length header
is what specifically came from that discussion.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.1 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE4X+5Jgl+vIlSVSNkRAgmHAKCxIc7LuiCA9hvjWA3chP1rf8FYDQCeNFAq
k55dDW5u7NORaEKyV3Aj0yw=
=gVee
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ When not in use, Happy Fun Ball should be
Minion of Nathan - Nathan says Hi! \ returned to its special container and
PGP Key: at a key server near you!  \ kept under refrigeration.

Re: missing POP3 mail

[Kai Großjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> pop3-munge-message-separator, which does its best to generate a
> syntactically valid `From ' line, with required blank line, and a valid
> Content-Length header if no obvious message separator can be found.

Ah.  And if the POP3 server emits a From_ line but not the trailing
empty line, you're screwed.  But the POP3 server shouldn't emit the
From_ line anyway.

Right.  So I wonder what it is that goes wrong for Andrew...

kai
-- 
A preposition is not a good thing to end a sentence with.

Re: missing POP3 mail

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann)  on Tue, 21 Dec 1999
| Ah.  And if the POP3 server emits a From_ line but not the trailing
| empty line, you're screwed.  But the POP3 server shouldn't emit the
| From_ line anyway.

Assuming that the spool file is mbox to begin with, then the POP server
should not strip the trailing space at the end of the message body, either.

| Right.  So I wonder what it is that goes wrong for Andrew...

I still do not believe pop3.el was directly responsible for what happened.
If the maildrop is locked, the POP server will not allow a new POP session
to be opened.  I would look into why the drop was locked when Gnus tried to
open it.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.1 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE4X/t3gl+vIlSVSNkRAtAxAJ4gEckWfsTkbAWmLv+VcW49ChK+3QCfWzoY
TsQGB0gfe0X50YIkMFVcDeE=
=dMb2
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Caution: Happy Fun Ball may suddenly
Minion of Nathan - Nathan says Hi! \ accelerate to dangerous speeds.
PGP Key: at a key server near you!  \