Re: External delivery: Pros and cons

Re: External delivery: Pros and cons

[Unknown]

X-Geek-3: GE/CS d+(--) s:++>: a C++++$ ULUHO++++$ P+++$ L+++ E+++ 
   W+++$ N+++ K-? !w--- O-? !M-- !V-- PS+ PE- Y+ PGP++ t@ 5++ !X R++ 
   b+++ DI+++ D- G e+++ h+ r++ y+
X-Face:  #q.#]5@vq!Jz+E0t_/;Y^gTjR\T^"B'fbeuVGiyKrvbfKJl!^e|e:iu(kJ6c|QYB57LP*|t&YlP~HF/=h:GA6o6W@I#deQL-%#.6]!z:6Cj0kd#4]>*D,|0djf'CVlXkI,>aV4\}?d_KEqsN{Nnt778"OsbQ["56/!nisvyB/uA5Q.{)gm6?q.j71ww.>b9b]-sG8zNt%KkIa>xWg&1VcjZk[hBQ>]j~`WqXl,y1a!(>6`UM{~'X[Y_,Bv+}=L\SS*mA8=s;!=O`ja|@PEzb&i0}Qp,`Z\:6:OmRi*
X-Organization: University of Massachusetts, Amherst, MA 01003
X-Time: Fri Nov 17 04:40:49 1995
X-Mail-System: Vm 5.95 (beta) for GNU Emacs 19.14 XEmacs Lucid (beta5)
References: <199511151538.HAA00082@edmonds.home.cs.ubc.ca>
From: Manoj Srivastava <srivasta@pilgrim.umass.edu>
Date: 17 Nov 1995 04:40:48 -0500
In-Reply-To: Brian Edmonds's message of Wed, 15 Nov 1995 07:38:25 -0800
Message-ID: <gvxbuqbh93z.fsf@plymouth.pilgrim.umass.edu>
Organization: Project Pilgrim, University of Massachusetts at Amherst
Lines: 42
X-Mailer: September Gnus v0.14

Hi,

>>"Brian" == Brian Edmonds <edmonds@cs.ubc.ca> writes:

 Date> 15 Nov 1995 03:01:39 -0500
 >>
 Cons>

 Brian> One con you forgot that used to bother me a lot: when you
 Brian> filter incoming messages, you end up with one (potentially
 Brian> large: perl) process per message.  If you're on a system like
 Brian> mine where I retrieve mail via UUCP once per hour, all those
 Brian> processes starting up at once can put a serious strain on the
 Brian> system.  I have blown out my virtual space once in the past
 Brian> when I was only running with 20M.

	If you use mailagent as your external splitting agent, then
 the filtering is two phased, each mail message invokes a small filter
 written in C, that spools the files locally, and calls the
 (potentially large) perl mailagent just once for a batch of files.  I
 think it would solve your problem with batch downloads (works great
 for me on a linux box).

 Brian> I find splitting the mail inside emacs is pretty quick, unless
 Brian> you've got hundreds of messages, and far kinder on my hard
 Brian> drive and system performance.  As it is, just punting
 Brian> everything through sendmail at once produces a noticable lag.

	Well, I need mailagent for other things, and did not feel like
 maintaining two sets of filtering rules. 

 Brian> Brian.  http://www.cs.ubc.ca/spider/edmonds

	manoj

-- Burnt Sienna. That's the best thing that ever happened to
 Crayolas. Ken Weaver

Manoj Srivastava         Project Pilgrim, Department of Computer Science 
Phone: (413) 545-3918             A143B Lederle Graduate Research Center
Fax: (413) 545-1249       University of Massachusetts, Amherst, MA 01003     
email:srivasta@pilgrim.umass.edu http://www.pilgrim.umass.edu/~srivasta/

Re: External delivery: Pros and cons

[Scott Blachowicz]

Brian Edmonds <edmonds@cs.ubc.ca> wrote:

> One con you forgot that used to bother me a lot: when you filter
> incoming messages, you end up with one (potentially large: perl) process
> per message.
The "potentially large" part depends on what happens when your MTA does the
delivery.  Mailagent, for instance, seems to have a 2 step method - the MTA
runs a "filter" program that just puts the message in a queue directory.  The
potentially large perl process runs as a single process to clear that queue.

Of course, I haven't tried many alternatives, so I can't really comment on
relative merits...just on my current playthings. :-)

Scott Blachowicz    Ph: 206/283-8802x240    StatSci, a div of MathSoft, Inc.
                                            1700 Westlake Ave N #500
scott@statsci.com                           Seattle, WA USA   98109
Scott.Blachowicz@seaslug.org

Re: External delivery: Pros and cons

[Brian Edmonds]

> Date: 15 Nov 1995 03:01:39 -0500
> 
> Cons:

One con you forgot that used to bother me a lot: when you filter
incoming messages, you end up with one (potentially large: perl) process
per message.  If you're on a system like mine where I retrieve mail via
UUCP once per hour, all those processes starting up at once can put a
serious strain on the system.  I have blown out my virtual space once in
the past when I was only running with 20M.

I find splitting the mail inside emacs is pretty quick, unless you've
got hundreds of messages, and far kinder on my hard drive and system
performance.  As it is, just punting everything through sendmail at once
produces a noticable lag.

Brian.                               http://www.cs.ubc.ca/spider/edmonds

External delivery: Pros and cons

[Sudish Joseph]

Pros:
- Much faster startup.

Cons:
- It is possible to lose mail if you're not careful in setting things up.
- Not easy to set up, you'll have to juggle the startup files for
  both your filtering agent and gnus.

Losing mail in this context comes in two flavors.
a) You actually lose the message.
b) The message itself isn't lost, but the NOV/active file entry gets
   clobbered, so you might not know you received the message.  (Not a
   problem if you use nnmh instead of nnml.)

Avoiding lossage for the two cases:
a) Actual lossage of the entire message is possible if you use the
   copy and move commands from the summary ("B m" "B c").  In this
   case, gnus generates the new article number on the basis of
   potentially old information, so it could overwrite a newly
   delivered message.

Solution: 
Don't move/copy into groups you deliver to.  Sounds stupid, yes, but
it's not a big pain in practice.  It's possible to make this safe
using locking, but looks like it'd be a pain to implement.

b) The active file getting clobbered is not a big deal, it'll get
   corrected again by the next mail that arrives for that group.

   The NOV file can get clobbered in two ways.  It's possible for an
   entry to get deleted due to gnus overwriting a newer version.  It's
   also possible to get your nov entries entered in non-ascending
   order.

Solution:
For the active file problem, modify gnus-group-get-new-news-this-group
(M-g in the group buffer) to do an nnmh-style lookup for the nnml
group.  This would make it simple to correct the active file entry.
This is also more consistent with how nntp groups are handled (uses
the active file for mass lookup/does a GROUP for a -this-group
lookup).  A better solution is to use locking (see below).

For the NOV entry being deleted problem, use locking.  Essentially,
provide hooks for the user to run whatever locking method they want
(procmail users might run `lockfile' off such hooks).  Four hooks
would suffice: one pre- and post- pair for active file update, another
pair for NOV update.

The non-ascending NOV file should really be the filters
problem...except that doing this isn't efficient for some filters.
It's not a problem in mailagent, since only one mailagent is active
at any given time; doing a sort from procmail for _every_ message you
get is prohibitively expensive--sleeping for earlier articles to be
updated leads to the possibility of long queues of processes hanging
on one possibly-dead process) 

It'd be simple enough to get GNUS to read such malformed NOV files
properly, but Lars fears that this might cause a slow down.  I don't
think it'd be that bad.  If this is unacceptable, we could have a
variable that would let the user select a version of
gnus-get-newsgroup-headers-xover that allows unsorted nov files, while
leaving the default the way it is.

If none of the above are unacceptable, use an external script to
sanity check.correct your overviews and active file.  Or you could use
nnml-generate-nov-databases for this, but it clobbered something or
the other the last time I used it. :-)

None of the above are very difficult to implement, the only real issue
is whether any such code will be put into gnus itself.  I'd rather not
have _another_ large chunk of code I have to patch into every release
I fetch.  Besides, sooner or later, people're going to hack their
version of GNUS 5.x to do stuff like this...(I will, for one :)

-Sudish