Gnus5, procmail, and reading mail lists.

Gnus5, procmail, and reading mail lists.

[Mark Borges]

I'd like to take a stab at using Gnus5 to read mailing list mail as a
newsgroup. I've read through the info pages, but before I take the
plunge (and possibly lose some of this valuable list mail) I thought
I'd ask this list if the below is correct, and/or if there are any
pitfalls in what I'm planning. I checked the FAQ, and didn't see
specific answers (maybe these Q's aren't F enough).

I use procmail.

Procmail splits my incoming mail box into the mailing list boxes in my
home directory, i.e., it takes /var/mail/me and extracts mail from the
two lists foo and bar into:

	$HOME/mail/spool/foo
	$HOME/mail/spool/bar

Currently, I use VM and movemail to read these, but the scoring
capability of Gnus5 is begging me to try it instead.

So, how do Gnus / procmail exist, esp. with regard to lockfiles?

My reading of the info pages suggests I do the following:

	(setq nnmail-use-procmail t); use it
	(setq nnmail-spool-file "procmail") ; or should that be a variable?
	(setq nnmail-split-methods nil)) ; avoid unpleasant side effects
	(setq nnmail-procmail-directory (expand-file-name ~/mail/spool))
	(setq nnmail-procmail-suffix nil) ; just use the group name? I
 	                                    want the group to be foo
 	                                    and the spool file to be
 	                                    foo, as above.

Also, there was some discussion of why this:

------------------------------------------------------------------------------
If you use `procmail' to split things directory into an nnmh directory
(which you shouldn't do), you should set `nnmail-keep-last-article' to
non-`nil' to prevent Gnus from ever expiring the final article in a
mail newsgroup. This is quite, quite important.
------------------------------------------------------------------------------

was important, but I lost it. Anyway, I'm planning on using nnml as the
backend to get the NOV databases (unless procmail can do it more
efficiently). Is it just as important for nnml as nnmh?

Finally, should I restrict myself to the Gnus5 expiry process, or can
I just run a simple cron job each weekend to remove old mail? (I
suspect that might confuse Gnus5 after a while, though.)

Thanks for any tips (or pointers to procmail recipes).

  -mb-

Re: Gnus5, procmail, and reading mail lists.

[Joe Hildebrand]

> "mb" == Mark Borges <mdb@cdc.noaa.gov> writes:

   mb> I use procmail.
   mb> Procmail splits my incoming mail box into the mailing list
   mb> boxes in my home directory, i.e., it takes /var/mail/me and
   mb> extracts mail from the two lists foo and bar into:

   mb> 	$HOME/mail/spool/foo 
   mb> 	$HOME/mail/spool/bar

you will have less difficulty if you use:

$HOME/mail/spool/foo.spool
$HOME/mail/spool/bar.spool

   mb> Currently, I use VM and movemail to read these, but the scoring
   mb> capability of Gnus5 is begging me to try it instead.

   mb> So, how do Gnus / procmail exist, esp. with regard to
   mb> lockfiles?

fine.  gnus uses movemail, which does locking.  I have this at the top
of my .procmailrc, though:

# use lockfiles that Emacs movemail honours... otherwise you may lose mail
LOCKEXT=+

   mb> My reading of the info pages suggests I do the following:

   mb> 	(setq nnmail-use-procmail t); use it 
   mb> 	(setq nnmail-spool-file "procmail") ; or should that be a
   mb>     variable? 

try (setq nnmail-spool-file 'procmail)

   mb> 	(setq nnmail-split-methods nil)) ; avoid unpleasant side effects
   mb>  (setq nnmail-procmail-directory (expand-file-name ~/mail/spool)) 
   mb>  (setq nnmail-procmail-suffix nil) ; just use the
   mb>     group name? I want the group to be foo and the spool file
   mb>     to be foo, as above.

don't do this last one.  there used to be a bug with that; I don't
know if it has been fixed, yet.  just use ".spool", the default.

   mb> Also, there was some discussion of why this:

   mb> ------------------------------------------------------------------------------
   mb> If you use `procmail' to split things directory into an nnmh
   mb> directory (which you shouldn't do), you should set
   mb> `nnmail-keep-last-article' to non-`nil' to prevent Gnus from
   mb> ever expiring the final article in a mail newsgroup. This is
   mb> quite, quite
   mb> important. ------------------------------------------------------------------------------
   mb> 

   mb> was important, but I lost it. Anyway, I'm planning on using
   mb> nnml as the backend to get the NOV databases (unless procmail
   mb> can do it more efficiently). Is it just as important for nnml
   mb> as nnmh?

this was the old way.  don't worry about it.  Gnus takes care of
everything, the way you are doing it.

   mb> Finally, should I restrict myself to the Gnus5 expiry process,
   mb> or can I just run a simple cron job each weekend to remove old
   mb> mail? (I suspect that might confuse Gnus5 after a while,
   mb> though.)

use expiry.  unless your script is going to modify your .overview
file, and you are *sure* that you are never going to be using Gnus at
the time of day that your cron job fires.

-- 
Joe Hildebrand                  Fuentez Systems Concepts
hildjj@fuentez.com              11781 Lee-Jackson Hwy, Suite 700
Lead Software Engineer          Fairfax, VA 22033
	"Breakfast recapitulates phylogeny" - Spider Robinson

Re: Gnus5, procmail, and reading mail lists.

[Mark Borges]

Thanks for the tips, Joe and Steven...

>>>>> "j" == Joe Hildebrand <hildjj@fuentez.com> writes:
    j> you will have less difficulty if you use:

    j> $HOME/mail/spool/foo.spool $HOME/mail/spool/bar.spool

OK. I gave up and used the extension. I also gave up and used ~/Mail
instead of ~/mail. Gnus was stubborn about using the former for some
reason. Perhaps I was setting it too late, I don't know. I was setting
it in my `gnus-init-file', but it was being over-ridden. Is there a
preferred place for setting nnmail-directory (I assume that's the
correct variable to set)?

    j> fine.  gnus uses movemail, which does locking.  I have this at
    j> the top of my .procmailrc, though:
    j> # use lockfiles that Emacs movemail honours... otherwise you
    j> may lose mail LOCKEXT=+

OK. It sounds like it should be fine, then. I haven't had a problem
with VM w/o the LOCKEXT above, but what the heck, I'll add it to my
.procmailrc, too. 

    j> try (setq nnmail-spool-file 'procmail)

Yes, I suspected that but the doc-string wasn't clear to me.

    j> this was the old way.  don't worry about it.  Gnus takes care
    j> of everything, the way you are doing it.
OK.

Here are the nnmail related variables I explicitly set, in case anyone
cares:

------------------------------------------------------------------------------
 ;; mail reading stuff
(setq
 nnmail-use-procmail t
 nnmail-directory (expand-file-name "~/Mail")	; this is the default anyway
 nnmail-spool-file 'procmail
 nnmail-split-methods nil
 nnmail-procmail-directory (expand-file-name "~/Mail/spool")
 ;nnmail-procmail-suffix nil ; just use the group name? I
 nnml-get-new-mail t
 ;nnmail-keep-last-article t ; supposed to be important
)
------------------------------------------------------------------------------

procmail delivers like into

	~/Mail/spool/lists.foo.spool

using recipes like:

	### ---------------------------------------
	## foo
	# 
	:0 :
	* ^TOfoo-digest
	lists.foo.spool

although it wasn't clear whether I should deliver into
~/Mail/spool/lists/foo.spool, (which turns out to be wrong)
instead, since the group resides in ~/Mail/lists/foo.

Anyway, then all one needs to do is follow the info page:
------------------------------------------------------------------------------
   Go to the group buffer and type `G m'.  When prompted, answer `foo'
for the name and `nnmh' for the method.  Repeat twice for the two other
groups, `bar' and `mail.baz'.  Be sure to include all your mail groups.

   That's it.  You are now set to read your mail.  An active file for
this method will be created automatically.
------------------------------------------------------------------------------

i.e., I typed `G m' and answered `lists.foo' for the name and `nnml'
for the method. Amazingly enough, it works!

Thanks again.
  -mb-

Re: Gnus5, procmail, and reading mail lists.

[Mark Borges]

Does nnml understand the evil SYSV Content-Length header?

I just had what should have appeared as one message split into
two. I'm not sure it was Gnus' fault, as I didn't check the procmail
.spool file before Gnus deleted it. Also, the second message, which
was apparently included in the first, was very old (dated from 1985 --
hmm, on second thought, I remember 1985 pretty well) -- maybe the
headers have changed since then?

Anyway, I cat'd the two back together, and uuencoded them into a file
`nnml.bug' (in an attempt to protect the headers from being
corrupted). Notice that the first contains an X-content-length header
equal to 1344, which is larger than the first article. 

I'll try to remember to check the mailing list archive for this group
(xemacs-beta) to see how the article was originally sent. Or perhaps
there are people on this list who are also on xemacs-beta who saw the
same thing?

begin 644 nnml.bug
M6"U&<F]M+4QI;F4Z('AE;6%C<RUB971A+7)E<75E<W1 8W,N=6EU8RYE9'4@
M(%1U92!.;W8@,30@,C Z-#<Z,C4@,3DY-0I2971U<FXM4&%T:#H@/'AE;6%C
M<RUB971A+7)E<75E<W1 8W,N=6EU8RYE9'4^"E)E8V5I=F5D.B!F<F]M('-P
M<G5C92YC<RYU:75C+F5D=2!B>2!C9&,N;F]A82YG;W8@*#4N,"]334DM4U92
M-"D*"6ED($%!,3 Y-C4[(%1U92P@,30@3F]V(#$Y.34@,C Z-#<Z,C @+3 W
M,# *4F5C96EV960Z(&)Y('-P<G5C92YC<RYU:75C+F5D=2!I9"!!03(U,CDS
M"B @*#4N-C=A+TE$02TQ+C4@9F]R('AE;6%C<RUB971A+7!E;W!L92D[(%1U
M92P@,30@3F]V(#$Y.34@,C$Z,C4Z,C @+3 V,# *4F5C96EV960Z(&9R;VT@
M82YC<RYU:75C+F5D=2!B>2!S<')U8V4N8W,N=6EU8RYE9'4@=VET:"!33510
M(&ED($%!,C4R.#D*(" H-2XV-V$O241!+3$N-2!F;W(@/'AE;6%C<RUB971A
M0'-P<G5C92YC<RYU:75C+F5D=3XI.R!4=64L(#$T($YO=B Q.3DU(#(Q.C(U
M.C$X("TP-C P"E)E8V5I=F5D.B!F<F]M('=I;F<N=FEP+F)E<W0N8V]M(&)Y
M(&$N8W,N=6EU8RYE9'4@=VET:"!33510(&ED($%!,3(S,3(*(" H-2XV-V$O
M241!+3$N-2!F;W(@/'AE;6%C<RUB971A0&-S+G5I=6,N961U/BD[(%1U92P@
M,30@3F]V(#$Y.34@,C$Z,C4Z,3D@+3 V,# *4F5C96EV960Z(&)Y('=I;F<N
M=FEP+F)E<W0N8V]M(&ED($%!,#$T-C$*(" H-2XV-V(O241!+3$N-2!F;W(@
M>&5M86-S+6)E=&% 8W,N=6EU8RYE9'4I.R!4=64L(#$T($YO=B Q.3DU(#$Y
M.C(T.C0R($=-5 I$871E.B!4=64L(#$T($YO=B Q.3DU(#$Y.C(T.C0R($=-
M5 I&<F]M.B!"96X@5VEN9R \=VEN9T!N971C;VTN8V]M/@I-97-S86=E+4ED
M.B \,3DY-3$Q,30Q.3(T+D%!,#$T-C% =VEN9RYV:7 N8F5S="YC;VT^"E1O
M.B!X96UA8W,M8F5T84!C<RYU:75C+F5D=0I3=6)J96-T.B!4:&4@;6]R92!T
M:&EN9W,@8VAA;F=E("XN+@I8+6-O;G1E;G0M;&5N9W1H.B Q,S0T"DQI;F5S
M.B T"EAR968Z(&-H87)N97D@;&ES=',N>&5M86-S.CD*"G1H92!M;W)E('1H
M97D@<W1A>2!T:&4@<V%M92X@($QO;VMI;F<@=&AR;W5G:"!T:&4@=F5R>2!E
M87)L:65S= IA<F-H:79E<R!O9B!N970N96UA8W,@*'=A>2!B86-K('EO;BP@
M8F5F;W)E('1H92!'<F5A="!5<V5N970@4F5N86UI;F<*86YD('=H96X@86QL
M(&AO<W1N86UE<R!W97)E(&]F('1H92!F;W)M($9/3T)!4BY!4E!!*2P@86)O
M=70@=&AE"G1E;G1H(&%R=&EC;&4@<F5A9',Z"@I8+49R;VTM3&EN93H@<F]C
M:&5S=&5R(6IS<T!M:70M961D:64@5&AU($UA>2 S," P,3HU-3HP-" Q.3@U
M"E)E8V5I=F5D.B!F<F]M($U)5"U%1$1)12!B>2!-250M4%)%4" H-"XQ,B\T
M+C@I('=I=&@@0TA!3U,@"B @(" @(" @:60@04$Q,#(U-CL@5&AU+" S,"!-
M87D@.#4@,#$Z-#8Z,S4@961T"E)E8V5I=F5D.B!B>2!M:70M961D:64N05)0
M02 H-"XQ,B\T+C@I("!I9"!!03(T,38Q.R!4:'4L(#,P($UA>2 X-2 P,3HT
M-#HS-"!E9'0*4F5L87DM5F5R<VEO;CH@=F5R<VEO;B!"(#(N,3 N,2 V+S(T
M+S@S.R!S:71E(&UI="UE9&1I92Y554-0"E!O<W1I;F<M5F5R<VEO;CH@=F5R
M<VEO;B!"(#(N,3 N,B Y+S$X+S@T.R!S:71E(')O8VAE<W1E<BY554-0"E!A
M=&@Z(&UI="UE9&1I92%G96YR860A<&%N9&$A=&%L8V]T="%H87)V87)D(7-E
M:7-M;R%R;V-H97-T97(A:G-S"D9R;VTZ(')O8VAE<W1E<B%J<W- ;6ET+65D
M9&EE("A*;VX@4RX@4W1U;7!F*0I.97=S9W)O=7!S.B!N970N96UA8W,*4W5B
M:F5C=#H@1TY5($5M86-S"DUE<W-A9V4M260Z(#PY.3$S0')O8VAE<W1E<BY5
M54-0/@I$871E.B!4=64L(#(X+4UA>2TX-2 R,#HQ-CHR-"!%1%0*07)T:6-L
M92U)+D0N.B!R;V-H97-T92XY.3$S"E!O<W1E9#H@5'5E($UA>2 R." R,#HQ
M-CHR-" Q.3@U"D1I<W1R:6)U=&EO;CH@;F5T"D]R9V%N:7IA=&EO;CH@52X@
M;V8@4F]C:&5S=&5R+"!#4R!$97!T+@I!<'!A<F5N=&QY+51O.B!E;6%C<RUN
M971N97=S+61I<W1R:6)U=&EO;D!M:70M<')E< I3=&%T=7,Z($\*3&EN97,Z
M(#D*6')E9CH@8VAA<FYE>2!L:7-T<RYX96UA8W,Z,3 *"E!E979E*',I.@I/
M;F4@=&AI;F<@22!F;W5N9"!B;W1H97)S;VUE('=I=&@@1TY5($5M86-S(&ES
M('1H92 B:&%R9'=I<FEN9R(*;V8@0V]N=')O;"UH('1O(&$@<&%R=&EC=6QA
M<B!F=6YC=&EO;B!O;B!T:&4@9VQO8F%L(&UA<"X*22!U<V4@8F%C:W-P86-E
M(&%S(&UY(&5R87-E(&-H87)A8W1E<B!A;F0@=V]U;&0@<')E9F5R('1O(&ME
M97 *:70@=&AA="!W87DN("!)(&AA=F4@<F5D969I;F5D(&ET('1O(&)E($53
M0RT_(&)U="!T:&4@9&]C=6UE;G1A=&EO;@II<R!R96%L;'D@<V5T('5P;VX@
M0V]N=')O;"UH+B @22!T:&EN:R!T:&4@8FEN9&EN9W,@<VAO=6QD(&)E(&UO
5<F4*9FQE>&EB;&4N"@I;971C70H*
 
end

Re: Gnus5, procmail, and reading mail lists.

[Steven L. Baur]

I don't think this is a Gnus bug.  There's no way of being sure
without having the original spool file.  If an e-mail message got out
containing a body line consisting of a valid mailbox From_ line, the
behavior you describe will occur.

I noticed that if I insert a raw mail file sendmail.el quotes any
leading From_ ... lines as >From_ ...  before sending the message.

Note that on storage from spool Gnus changes From_ to X-From-Line:.

-- 
steve@miranova.com baur

Re: Gnus5, procmail, and reading mail lists.

[Steven L. Baur]

>>>>> "mb" == Mark Borges <mdb@cdc.noaa.gov> writes:

    mb> I'd like to take a stab at using Gnus5 to read mailing list
    mb> mail as a newsgroup.
...
    mb> I checked the FAQ, and didn't see specific answers
    mb> (maybe these Q's aren't F enough).

Er, something like that.  I have *a lot* of good material set aside,
the problem is digesting and compacting it.

    mb> Procmail splits my incoming mail box into the mailing list
    mb> boxes in my home directory, i.e., it takes /var/mail/me and
    mb> extracts mail from the two lists foo and bar into:

    mb> 	$HOME/mail/spool/foo $HOME/mail/spool/bar

This is fine.

    mb> So, how do Gnus / procmail exist, esp. with regard to
    mb> lockfiles?

It snarfs the spool files with the Emacs movemail program.  Movemail
is lockfile compatible with procmail.

    mb> My reading of the info pages suggests I do the following:

	(setq nnmail-use-procmail t); use it
	(setq nnmail-spool-file "procmail") ; or should that be a variable?

Yes.  Use:
(setq nnmail-spool-file 'procmail)

	(setq nnmail-split-methods nil)) ; avoid unpleasant side effects
	(setq nnmail-procmail-directory (expand-file-name ~/mail/spool))
	(setq nnmail-procmail-suffix nil) ; just use the group name? I
 	                                    want the group to be foo
 	                                    and the spool file to be
 	                                    foo, as above.

If you are going to use Gnus expiration, it is important to also set:
(setq nnmail-keep-last-article t)

And, if you want Gnus to go out and grab the spool files you will also
need:

(setq nnml-get-new-mail t)


    mb> Also, there was some discussion of why this:

...

    mb> was important, but I lost it.

See the hypermail archive:
<URL: http://www.miranova.com/gnus-list/Oct-1995/0256.html>

    mb> Anyway, I'm planning on using
    mb> nnml as the backend to get the NOV databases (unless procmail
    mb> can do it more efficiently). Is it just as important for nnml
    mb> as nnmh?

The basic difference between nnml and nnmh is that nnml uses NOV
files, and nnmh does not.

If you are going to generate NOV databases out of procmail, you will
have to deal with the same locking issues you face with the spool
files, with the added nicety that while the NOV database is being
updated Gnus must be locked out too.  I don't think anybody is doing
it this way.

    mb> Finally, should I restrict myself to the Gnus5 expiry process,
    mb> or can I just run a simple cron job each weekend to remove old
    mb> mail? (I suspect that might confuse Gnus5 after a while,
    mb> though.)

I'd recommend sticking with expiration from Gnus.  I'd be very
interested, though, in hearing reports of using News software for
maintenance.  Is it possible to make a News-lite distribution from the
canonical sources explicitly for this kind of use?

    mb> Thanks for any tips (or pointers to procmail recipes).

Procmail is black magic.  It just works.
Here are the recipes I'm currently using for this mailing list:

I had previously been using the pattern:
* ^TOding(-request)?
This doesn't work 100% mainly due to messages being Cc:'ed to you and the
mailing list.


# Ding Gnus (Usenet Newsreader for Emacs)
:0 c:
* Return-Path:.*ding.*ifi.uio.no
/usr/local/Web/gnus-list/gnus-mail-archive


:0 w:
* Return-Path:.*ding.*ifi.uio.no
dgnus-list.spool

Regards,
-- 
steve@miranova.com baur

Re: Gnus5, procmail, and reading mail lists.

[Joe Hildebrand]

> "Steven" == Steven L Baur <steve@miranova.com> writes:

   Steven> LOCKEXT=+

   Steven> I thought movemail *did* use .lock for an extension.  At
   Steven> least that's what the XEmacs sources seem to imply.

I got that from somewhere.... anyone want to 'fess up?  I haven't lost
any mail (that I know of :) using either that or the default.  Maybe
I'll go have a look at the 19.29 movemail source.

   Steven> If you are going to use Gnus expiration, it is important to
   Steven> also set: (setq nnmail-keep-last-article t)

I thought this was only if procmail was delivering directly to the
nnm[lh] directory.

   Steven> If you are going to generate NOV databases out of procmail,
   Steven> you will have to deal with the same locking issues you face
   Steven> with the spool files, with the added nicety that while the
   Steven> NOV database is being updated Gnus must be locked out too.
   Steven> I don't think anybody is doing it this way.

Anymore.  Some people may still be using add-active-nov.

   mb> Finally, should I restrict myself to the Gnus5 expiry process,
   mb> or can I just run a simple cron job each weekend to remove old
   mb> mail? (I suspect that might confuse Gnus5 after a while,
   mb> though.)

   Steven> I had previously been using the pattern: ^TOding(-request)?
   Steven> This doesn't work 100% mainly due to messages being Cc:'ed
   Steven> to you and the mailing list.

procmailrc(5) says:
     If the regular expression contains `^TO' it will be  substi-
     tuted by `(^((Original-)?(Resent-)?(To|Cc|Bcc)|(X-Envelope
     |Apparently(-Resent)?)-To):(.*[^a-zA-Z])?)', which should
     catch all destination specifications.

which should catch CCs.

-- 
Joe Hildebrand                  Fuentez Systems Concepts
hildjj@fuentez.com              11781 Lee-Jackson Hwy, Suite 700
Lead Software Engineer          Fairfax, VA 22033
	"Breakfast recapitulates phylogeny" - Spider Robinson

Re: Gnus5, procmail, and reading mail lists.

[Edward J. Sabol]

Excerpts from [ding]: (13-Nov-95) Re: Gnus5, procmail, and reading mail lists.
by Joe Hildebrand

>> "Steven" == Steven L Baur <steve@miranova.com> writes:
Steven> LOCKEXT=+
Steven> I thought movemail *did* use .lock for an extension. At least that's
Steven> what the XEmacs sources seem to imply.

>> "Joe" == Joe Hildebrand <hildjj@fuentez.com> writes:
Joe>  I got that from somewhere.... anyone want to 'fess up? I haven't lost
Joe> any mail (that I know of :) using either that or the default. Maybe I'll
Joe> go have a look at the 19.29 movemail source.

I don't know about 19.29, but 19.28 most definitely uses "foo.lock" semaphore
files for locking, and *not* "foo+". I find it hard to believe that they
would change that for 19.29. Also, in 19.28, movemail would only use
"foo.lock" semaphore files if and only MAIL_USE_LOCKF and MAIL_USE_FLOCK are
both undefined. Otherwise, it uses lockf() or flock() respectively. I advise
everyone who has "LOCKEXT=+" in his/her .procmailrc file to check the source
code for movemail and verify this. If you have procmail set to use "foo+"
semaphore files and movemail only recognizes "foo.lock" semaphore files, you
can and probably will lose e-mail at some point, unless you have
MAIL_USE_LOCKF or MAIL_USE_FLOCK defined, in which case it's a moot point
what the name of the locking semaphore file is. (Please note that you should
*not* have MAIL_USE_LOCKF or MAIL_USE_FLOCK defined if your mail spool
directory is NFS-mounted. If that's the case, then you should always use
semaphore lock files.)

Later,
Ed

Re: Gnus5, procmail, and reading mail lists.

[Steven L. Baur]

>>>>> "Joe" == Joe Hildebrand <hildjj@fuentez.com> writes:
    mb> So, how do Gnus / procmail exist, esp. with regard to
    mb> lockfiles?

    Joe> fine.  gnus uses movemail, which does locking.  I have this
    Joe> at the top of my .procmailrc, though:

# use lockfiles that Emacs movemail honours... otherwise you may lose mail
LOCKEXT=+

I thought movemail *did* use .lock for an extension.  At least that's
what the XEmacs sources seem to imply.

-- 
steve@miranova.com baur

Re: Gnus5, procmail, and reading mail lists.

[Steven L. Baur]

>>>>> "Joe" == Joe Hildebrand <hildjj@fuentez.com> writes:

>>>>> "Steven" == Steven L Baur <steve@miranova.com> writes:
    Steven> If you are going to use Gnus expiration, it is important
    Steven> to also set: (setq nnmail-keep-last-article t)

    Joe> I thought this was only if procmail was delivering directly
    Joe> to the nnm[lh] directory.

You're correct.

    Steven> If you are going to generate NOV databases out of
    Steven> procmail, you will have to deal with the same locking
    Steven> issues you face with the spool files, with the added
    Steven> nicety that while the NOV database is being updated Gnus
    Steven> must be locked out too.  I don't think anybody is doing it
    Steven> this way.

    Joe> Anymore.  Some people may still be using add-active-nov.

When we had a similar discussion last month, something about this was
mentioned, but it went over my head.  I just reread the thread, and
saw again the comments about a procmail NOV hack, which I assume
add-active-nov is referring to.  I don't see a reference to it in
either the Gnus, XEmacs lisp source, nor procmail 3.10 source (or
dejanews for that matter).

    Steven> I had previously been using the pattern:
    Steven> ^TOding(-request)?  This doesn't work 100% mainly due to
    Steven> messages being Cc:'ed to you and the mailing list.

    Joe> procmailrc(5) says:
...
    Joe> which should catch CCs.

Exactly.  You end up with two messages with identical message-ids
going into the same nnml folder, one of which is now in a Gnus black
hole, and cannot be referenced reasonably, or two identical messages
going into the hyperarchive.

-- 
steve@miranova.com baur

Re: Gnus5, procmail, and reading mail lists.

[Per Abrahamsen]

>>>>> "SLB" == Steven L Baur <steve@miranova.com> writes:

SLB> Exactly.  You end up with two messages with identical message-ids
SLB> going into the same nnml folder, one of which is now in a Gnus black
SLB> hole, and cannot be referenced reasonably, or two identical messages
SLB> going into the hyperarchive.

If you use procmail, I suggest using the following code to sort out
duplicates (stolen from someone on the procmail list):

#####################
# Prevent dubplicates

:0
{
 :0 Whc: .msgid.lock     # w produces (non-fatal) error msg
 | formail -D 8192 .msgid.cache

 :0 a
 mail/duplicate/.
}

Re: Gnus5, procmail, and reading mail lists.

[Steven L. Baur]

>>>>> "Per" == Per Abrahamsen <abraham@dina.kvl.dk> writes:

>>>>> "SLB" == Steven L Baur <steve@miranova.com> writes:

    Per> If you use procmail, I suggest using the following code to
    Per> sort out duplicates (stolen from someone on the procmail
    Per> list):
...

You might want to adjust the 8192 upwards if processing a lot of mail ...

That's cleverer than the brute force method in the procmail docs.  There
are two problems with that approach that bother me.  And they're both
related.

If I ran that recipe before splitting mail for this list into its
place, I would definitely have all replies to message I write archived
ahead of the actual message.  This may not really be a problem, but it
is a nit.

Secondly, I am a bit of a neat freak with respect to saving mail.  I
want all the original (incoming) headers saved, and for mailing list
mail, I want only messages with headers coming from the list owner.

(I feel strongly about this now, because I allowed some non-Gnus
mailing list stuff to get mixed in with the ``real'' stuff, and spent
a lot of time, correcting it.  Never again.)

This recipe is first come first served, and the e-mail courtesy copy
(with the wrong, for me, headers) will get placed in my general
mailing list folder if it arrives first.

I see two other possible (stronger) solutions.
One can get more creative with the procmail recipe.  First separate
out all mailing list mail with proper mailing list headers.  Second,
take all the mail that matches a TO: <all-mailing-lists> and separate
*that* out into a duplicates folder.  I can see how to do that with
some ugly duplicated procmail recipes.

The second possibility, is to put Gnus to work with secondary
splitting.  Sort out all mailing list stuff by official headers (as
above), and let the rest go into a general folder.  Allow Gnus to do
splitting on the general folder only, and have it catch the duplicate
CC: to a mailing list stuff.

I don't know which of these two methods would be the easiest to
maintain.

-- 
steve@miranova.com baur

Re: Gnus5, procmail, and reading mail lists.

[Joe Hildebrand]

> "Steven" == Steven L Baur <steve@miranova.com> writes:

   Steven> When we had a similar discussion last month, something
   Steven> about this was mentioned, but it went over my head.  I just
   Steven> reread the thread, and saw again the comments about a
   Steven> procmail NOV hack, which I assume add-active-nov is
   Steven> referring to.  I don't see a reference to it in either the
   Steven> Gnus, XEmacs lisp source, nor procmail 3.10 source (or
   Steven> dejanews for that matter).

Yes, it was a hack.  No, even I don't use it anymore.  The only place
it was probably documented was on my web server.

If anyone is still using add-active-nov, could you please contact me?
I'm interested in getting everyone cut over to the Gnus way.

For those people using procmail in the approved way, I have a Tcl/Tk
script that does fairly nice biffing.  Let me know if you want to try
it.

-- 
Joe Hildebrand                  Fuentez Systems Concepts
hildjj@fuentez.com              11781 Lee-Jackson Hwy, Suite 700
Lead Software Engineer          Fairfax, VA 22033
	"Breakfast recapitulates phylogeny" - Spider Robinson

Re: Gnus5, procmail, and reading mail lists.

[Scott Blachowicz]

Joe Hildebrand <hildjj@fuentez.com> wrote:

> If anyone is still using add-active-nov, could you please contact me?
> I'm interested in getting everyone cut over to the Gnus way.

Well...I've been thinking of _starting_ to use it.  I use mailagent to
deliver my messages directly into MH folders instead of into .spool files
because I want to be able to check for messages in that folder without
having to fire up ding get the messages filed away.  Hmmm...I suppose I
could write a script to do the filing, but then I'd need something like
your 'add-active-nov' to get my NOV files updated, right?

> For those people using procmail in the approved way, I have a Tcl/Tk
> script that does fairly nice biffing.  Let me know if you want to try
> it.

Is it referenced in your web page?  That is, would I have seen it in the
same place that I grabbed add-active-nov from?  If so, I've probably got
it laying around here somewhere.  Otherwise, I'd appreciate seeing it.

Thanx,
Scott Blachowicz    Ph: 206/283-8802x240    StatSci, a div of MathSoft, Inc.
                                            1700 Westlake Ave N #500
scott@statsci.com                           Seattle, WA USA   98109
Scott.Blachowicz@seaslug.org

Re: Gnus5, procmail, and reading mail lists.

[Joe Hildebrand]

> "Scott" == Scott Blachowicz <scott@statsci.com> writes:

   Scott> Well...I've been thinking of _starting_ to use it.  I use
   Scott> mailagent to deliver my messages directly into MH folders
   Scott> instead of into .spool files because I want to be able to
   Scott> check for messages in that folder without having to fire up
   Scott> ding get the messages filed away.  Hmmm...I suppose I could
   Scott> write a script to do the filing, but then I'd need something
   Scott> like your 'add-active-nov' to get my NOV files updated,
   Scott> right?

Just have mailagent deliver to .spool files.  You can then use
[jangle].  I have updated:

<URL:http://www.fuentez.com/public-info/ding/ding.html>

to include [jangle], as well as instructions on how to use procmail
"the right way".

-- 
Joe Hildebrand                  Fuentez Systems Concepts
hildjj@fuentez.com              11781 Lee-Jackson Hwy, Suite 700
Lead Software Engineer          Fairfax, VA 22033
	"Breakfast recapitulates phylogeny" - Spider Robinson

Re: Gnus5, procmail, and reading mail lists.

[Sudish Joseph]

Scott Blachowicz writes:
> Well...I've been thinking of _starting_ to use it.  I use mailagent to
> deliver my messages directly into MH folders instead of into .spool files
> because I want to be able to check for messages in that folder without
> having to fire up ding get the messages filed away.  Hmmm...I suppose I
> could write a script to do the filing, but then I'd need something like
> your 'add-active-nov' to get my NOV files updated, right?

If you're using mailagent as your filter, you might consider extending
mailagent itself to generate your .overview entry.  I played around
with mailagent recently, and this seems like a simple thing to do --
most of the hard work is already done by mailagent.  It provides all
the header info you require in a hash, and it has a well-defined
interface for writing your own commands.

This way you'll save the fork of another perl script per mail, along
with a fork of formail.

-Sudish

Re: Gnus5, procmail, and reading mail lists.

[Scott Blachowicz]

Sudish Joseph <joseph@cis.ohio-state.edu> wrote:

> If you're using mailagent as your filter, you might consider extending
> mailagent itself to generate your .overview entry.  I played around
> with mailagent recently, and this seems like a simple thing to do --
> most of the hard work is already done by mailagent.

Yeah...I've already got stuff like this:

    Return-path: /ding-request@ifi\.uio\.no/i {
        ASSIGN list gnu/emacs/ding;
        SAVE +%#list;
        REJECT -t ListUpdate;
    };

    # Update some external files & such
    <ListUpdate> {
        PERL /homes/scott/var/mailagent/ListUpdate %#list;
    };

which all runs in the same perl process.  My plan was to update my
ListUpdate perl script to update NOV files too (right now it just makes
sure that the list is in my ~/Mail/.folders file), but I haven't gotten
around to setting things up to read the mailing lists from ding yet.

BTW, the perl process that does this doesn't happen at sendmail-delivery
time.  With mailagent, my .forward file runs a "filter" program that
basically just drops the message into a queue directory that gets
processed by a different program (i.e. "mailagent") that gets kicked off
by "filter" every so often.  That way the MTA is waiting for "me" to
decide what to do with each & every message.

Scott Blachowicz    Ph: 206/283-8802x240    StatSci, a div of MathSoft, Inc.
                                            1700 Westlake Ave N #500
scott@statsci.com                           Seattle, WA USA   98109
Scott.Blachowicz@seaslug.org

Re: Gnus5, procmail, and reading mail lists.

[Sudish Joseph]

Joe Hildebrand writes:
>> "Steven" == Steven L Baur <steve@miranova.com> writes:
Steven> I just reread the thread, and saw again the comments about a
Steven> procmail NOV hack, which I assume add-active-nov is referring
Steven> to.
> Yes, it was a hack.  No, even I don't use it anymore.  The only place
> it was probably documented was on my web server.

I think it can also be accessed off Lars' page somewhere, too.

> If anyone is still using add-active-nov, could you please contact me?
> I'm interested in getting everyone cut over to the Gnus way.

This seems too strong to me.  Sure, supporting completely safe
external mail delivery from within ding is extremely hard to do.  But
"completely safe" here relates to getting things to work correctly if
you're delivering directly to groups that you might do a "B m" or 
"B c" into.  If you're willing to forgo that luxury, it's possible to
achieve safe external delivery.

I switched over to using .spool files 2-3 weeks back, primarily to see
if startup split time wouldn't be a significant factor.  I now find it
a pain to have to wait through all the "splitting" messages before I
can read mail.  YMMV, of course.

As for biffing externally delivered mail, I have a Tk-from-Perl5
script that will biff such messages properly.  (It's modeled on
XMultibiff and offers seperate header and message previews.)

I've been thinking about this stuff and discussed some approaches with
Lars (unsuccessfully :<).  I think I'll post a complete list of
pros/cons/methods on this topic separately.

-Sudish

Re: Gnus5, procmail, and reading mail lists.

[Per Abrahamsen]

The speed of Gnus mail splitting with nnml/nnmh is almost solely
depending on what file system you use.  I have used Gnus mail
splitting with three different file systems:

NFS: Incredible slow.  You can read each split message in the
minibuffer several times, take a sip of your tea, and speculate about
the possible content before the next message arrives.

Linux ext2fs: Incredible fast.  You can see some flimmer in the
message area, but haven't a chance to see what they are about.  I get
the impression that the speed of mail splitting on Linux is limited by
how fast it can print those messages on the screen.

BSD-FFS: Somewhere in between.  You can read the messages if you stay
alert.

The reason is probably that NFS is write through, and that BSD-FFS
directory content is write through, while Linux ext2fs buffers
everything. 

Conclusions:

1. LINUX R00LZ!

   (but set nnmail-delete-incoming to nil, just in case...)

2. There is really nothing Lars can do to speed up mail splitting. 

3. Using procmail to split mail	instead of Gnus doesn't speed up
   anything, unless you deliver directly into the folders instead 
   spool files.

4. If you don't run Linux, consider using nnfolder instead of nnml.

5. Consider reading mail on the NFS server instead of a client.

6. Consider buying PrestoServe(TM) for your NFS server.

Re: Gnus5, procmail, and reading mail lists.

[Kai Grossjohann]

>>>>> "Per" == Per Abrahamsen <abraham@dina.kvl.dk> writes:

  Per> NFS: Incredible slow.  You can read each split message in the
  Per> minibuffer several times, take a sip of your tea, and speculate
  Per> about the possible content before the next message arrives.

Dear Per,

it seems that moving the nnmail-spool-directory and the nnml-directory
to a local disk (or CacheFS?) would make a lot of sense to speed up
splitting.  Does it make sense to move the spool directory only?  Does
it make sense to move the nnml directory only?

Also, is the speed of entering an nnml group affected by the same
effect?

tia,
        \kai{}
--
~/.signature

Re: Gnus5, procmail, and reading mail lists.

[Per Abrahamsen]

>>>>> "KG" == Kai Grossjohann <grossjoh@dusty.informatik.uni-dortmund.de> writes:

KG> it seems that moving the nnmail-spool-directory and the nnml-directory
KG> to a local disk (or CacheFS?) would make a lot of sense to speed up
KG> splitting.  

I couldn't find `nnmail-spool-directory'.

I don't know the time properties of CacheFS, so I can't comment on
that.  

KG> Does it make sense to move the spool directory only?  

Assuming you mean `nnmail-spool-file', no.  Reading from NFS is quite
fast unless the server or net is overloaded, it is writing that is
slow, especially creating a lot of small files (which is what nnml
does).

KG> Does it make sense to move the nnml directory only?

Yes.

KG> Also, is the speed of entering an nnml group affected by the same
KG> effect?

No.  Gnus entirely to blame for that delay, except in pathological
cases. 

Re: Gnus5, procmail, and reading mail lists.

[Steven L. Baur]

>>>>> "Kai" == Kai Grossjohann <grossjoh@dusty.informatik.uni-dortmund.de> writes:

>>>>> "Per" == Per Abrahamsen <abraham@dina.kvl.dk> writes:
    Per> NFS: Incredible slow.  You can read each split message in the
    Per> minibuffer several times, take a sip of your tea, and
    Per> speculate about the possible content before the next message
    Per> arrives.

    Kai> Also, is the speed of entering an nnml group affected by the
    Kai> same effect?

The speed of entering an nnml group is system dependent as
demonstrated by my nnheader-insert-head patch.  XEmacs allocates a new
64k buffer for each invocation of insert-file-contents which can
be an extraordinarily slow and painful process.

-- 
steve@miranova.com baur

Re: Gnus5, procmail, and reading mail lists.

[Jason L Tibbitts III]

>>>>> "Joe" == Joe Hildebrand <hildjj@fuentez.com> writes:

Joe> If anyone is still using add-active-nov, could you please contact me?

I still use it.  I haven't had the time lately to make any kind of switch
to a better method and for my application I don't see a better method.

I use it for keeping the ding-list archives, and I want to keep things
pretty much instantly updated, so I can't wait until I get into GNUS for
the splits to happen.  Here's my Procmail recipe:

:0: ding.lock
*^TOding@ifi.uio.no
{
      :0 c
      ding-list-recent/.

      :0 c
      ding-list/.

      :0 hw
        | ( /home/tibbs/bin/add-active-nov $LASTFOLDER ; \
          /usr/local/bin/tail -500 ding-list/.overview > ding-list-recent/.overview ; \
          rm -f blurgh `ls -1t ding-list-recent/* | sed -e 1,500d` )
}

As you see, I really need Procmail to keep the recent list, and I need the
.overview file to be up to date while I'm splitting.  Of course, if anyone
has any better ways of doing this, I'm all ears.  I know this works without
problems.

Joe> I'm interested in getting everyone cut over to the Gnus way.

I wonder if I can keep doing the archives my while doing everything else
the proper way without any conflicts.

 - J<

Re: Gnus5, procmail, and reading mail lists.

[Joe Hildebrand]

> "Jason" == Jason L Tibbitts <tibbs@sina.hpc.uh.edu> writes:


   Jason> As you see, I really need Procmail to keep the recent list,
   Jason> and I need the .overview file to be up to date while I'm
   Jason> splitting.  Of course, if anyone has any better ways of
   Jason> doing this, I'm all ears.  I know this works without
   Jason> problems.

   Joe> I'm interested in getting everyone cut over to the Gnus way.

I guess I'm worried that people are going to lose mail, and blame me,
since my name is on the script.  Even though it's not my fault Gnus
can't do locking :)

   Jason> I wonder if I can keep doing the archives my while doing
   Jason> everything else the proper way without any conflicts.

No problem.  Keep your archive delivery script.  Add one more branch
at the top of the {} block:


:0: ding.lock
*^TOding@ifi.uio.no
{
      :0c:
      ding.spool

      # turn add-active-nov on, from here to the end of this block.
      # since the block starts a sub-shell, this shouldn't propagate
      # out of the block.
      TRAP=add-active-nov

      ...
}

Remove the TRAP from the beginning of your .procmailrc, 
then see <URL:http://www.fuentez.com/public-info/ding/> for
instructions on how to set up ding to slurp your .spool files.

-- 
Joe Hildebrand                  Fuentez Systems Concepts
hildjj@fuentez.com              11781 Lee-Jackson Hwy, Suite 700
Lead Software Engineer          Fairfax, VA 22033
	"Breakfast recapitulates phylogeny" - Spider Robinson

Re: Gnus5, procmail, and reading mail lists.

[Steven L. Baur]

>>>>> "mb" == Mark Borges <mdb@cdc.noaa.gov> writes:

>>>>> "j" == Joe Hildebrand <hildjj@fuentez.com> writes:
    j> you will have less difficulty if you use:

    j> $HOME/mail/spool/foo.spool $HOME/mail/spool/bar.spool

    mb> OK. I gave up and used the extension. I also gave up and used
    mb> ~/Mail instead of ~/mail. Gnus was stubborn about using the
    mb> former for some reason. Perhaps I was setting it too late, I
    mb> don't know. I was setting it in my `gnus-init-file', but it
    mb> was being over-ridden. Is there a preferred place for setting
    mb> nnmail-directory (I assume that's the correct variable to
    mb> set)?

You are setting nnmail-directory.  You need to set nnml-directory.
There should be no problem setting this in gnus-init-file.

I'd recommend adding trailing slashes.  eg.
(setq  nnml-directory "~/mail/")
(setq  nnmail-procmail-directory "~/Mail/spool/")

Use of expand-file-name does not appear to be required.

Regards,
-- 
steve@miranova.com baur