External delivery: Pros and cons

[Sudish Joseph <joseph@cis.ohio-state.edu>]

Pros:
- Much faster startup.

Cons:
- It is possible to lose mail if you're not careful in setting things up.
- Not easy to set up, you'll have to juggle the startup files for
  both your filtering agent and gnus.

Losing mail in this context comes in two flavors.
a) You actually lose the message.
b) The message itself isn't lost, but the NOV/active file entry gets
   clobbered, so you might not know you received the message.  (Not a
   problem if you use nnmh instead of nnml.)

Avoiding lossage for the two cases:
a) Actual lossage of the entire message is possible if you use the
   copy and move commands from the summary ("B m" "B c").  In this
   case, gnus generates the new article number on the basis of
   potentially old information, so it could overwrite a newly
   delivered message.

Solution: 
Don't move/copy into groups you deliver to.  Sounds stupid, yes, but
it's not a big pain in practice.  It's possible to make this safe
using locking, but looks like it'd be a pain to implement.

b) The active file getting clobbered is not a big deal, it'll get
   corrected again by the next mail that arrives for that group.

   The NOV file can get clobbered in two ways.  It's possible for an
   entry to get deleted due to gnus overwriting a newer version.  It's
   also possible to get your nov entries entered in non-ascending
   order.

Solution:
For the active file problem, modify gnus-group-get-new-news-this-group
(M-g in the group buffer) to do an nnmh-style lookup for the nnml
group.  This would make it simple to correct the active file entry.
This is also more consistent with how nntp groups are handled (uses
the active file for mass lookup/does a GROUP for a -this-group
lookup).  A better solution is to use locking (see below).

For the NOV entry being deleted problem, use locking.  Essentially,
provide hooks for the user to run whatever locking method they want
(procmail users might run `lockfile' off such hooks).  Four hooks
would suffice: one pre- and post- pair for active file update, another
pair for NOV update.

The non-ascending NOV file should really be the filters
problem...except that doing this isn't efficient for some filters.
It's not a problem in mailagent, since only one mailagent is active
at any given time; doing a sort from procmail for _every_ message you
get is prohibitively expensive--sleeping for earlier articles to be
updated leads to the possibility of long queues of processes hanging
on one possibly-dead process) 

It'd be simple enough to get GNUS to read such malformed NOV files
properly, but Lars fears that this might cause a slow down.  I don't
think it'd be that bad.  If this is unacceptable, we could have a
variable that would let the user select a version of
gnus-get-newsgroup-headers-xover that allows unsorted nov files, while
leaving the default the way it is.

If none of the above are unacceptable, use an external script to
sanity check.correct your overviews and active file.  Or you could use
nnml-generate-nov-databases for this, but it clobbered something or
the other the last time I used it. :-)

None of the above are very difficult to implement, the only real issue
is whether any such code will be put into gnus itself.  I'd rather not
have _another_ large chunk of code I have to patch into every release
I fetch.  Besides, sooner or later, people're going to hack their
version of GNUS 5.x to do stuff like this...(I will, for one :)

-Sudish