saving to nnml groups

saving to nnml groups

[Darren/Torin/Who Ever...]

-----BEGIN PGP SIGNED MESSAGE-----

I'm going to nnml groups (from nnmh) relatively soon.

I've got two questions:
1) How do I save articles so that they appear in the .overview files?
   If I just use gnus-summary-save-in-folder, it just uses rcvstore
   which won't help.  I want to use gnus+nnml to read folders I save to.
2) I use mailagent to save *all* my mail to folders.  I never use
   nnmail-split and friends.  (I know I'm not supposed to, but I'm a big
   boy. (and can even program.)) Is expiration the only time that Gnus
   will update the .overview (NOV) file?  Is there a lock I can have it
   check for in case mailagent is there right then?

Darren
- -- 
<torin@daft.com> <http://www.daft.com/~torin> <torin@debian.org> <torin@io.com>
Darren Stalder/2608 Second Ave, @282/Seattle, WA 98121-1212/USA/+1-800-921-4996
@ Do you have your clothes on? I probably don't. Take yours off. Feel better. @
@ Sysadmin, webweaver, postmaster for hire.  C/Perl/CGI programmer and tutor. @

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3
Charset: noconv
Comment: Processed by Mailcrypt 3.4, an Emacs/PGP interface

iQCVAwUBM4VHJY4wrq++1Ls5AQFeiwP/bRbzfG7u6tIv/G7HTSvDTmeutzQVIAzt
c4ClmtnWPGFWipTFgYb1/PpHULd9Xu1H5TWdJgGTNwZabEH6KZICTB7hFWstSm+A
yyHf8X1SvCDYktRUtsUK9xFX3fw7KiHouHhsCOCVTJ4mdTpMfzaIAUY2vUXTwG7w
MjZG8DbrDEw=
=sFvu
-----END PGP SIGNATURE-----

Re: saving to nnml groups

[Kai Grossjohann]

>>>>> Darren/Torin/Who Ever writes:

  Darren> 1) How do I save articles so that they appear in the
  Darren>    .overview files?  If I just use
  Darren>    gnus-summary-save-in-folder, it just uses rcvstore which
  Darren>    won't help.  I want to use gnus+nnml to read folders I
  Darren>    save to.

What I'd do is to replace every occurrence of "nnmh" in .newsrc.eld
with "nnml" and use M-x nnml-generate-nov-databases RET to create the
initial overview files.  After that, "B c" and "B m" will do a good
job of updating the overview files.

  Darren> 2) I use mailagent to save *all* my mail to folders.  I
  Darren>    never use nnmail-split and friends.  (I know I'm not
  Darren>    supposed to, but I'm a big boy. (and can even program.))
  Darren>    Is expiration the only time that Gnus will update the
  Darren>    .overview (NOV) file?  Is there a lock I can have it
  Darren>    check for in case mailagent is there right then?

You don't have to use nnmail-split to do it "right".  Check out the
nnmail-use-procmail, nnmail-procmail-directory and
nnmail-procmail-suffix variables.  You tell your mailagent to write
the articles to files in nnmail-procmail-directory, and
nnmail-use-procmail tells Gnus to automatically read articles for the
nnml:foo.bar group from the file foo.bar.SUFFIX in
nnmail-procmail-directory (SUFFIX being the value of
nnmail-procmail-suffix, as you've probably guessed).

Been there, done that.  Worked like a charm.  But now I use
nnmail-split and don't think it's slow at all (for 50 to 100 mails in
the morning).

kai
-- 
Life is hard and then you die.

Re: saving to nnml groups

[Sudish Joseph]

Kai Grossjohann writes:
> Been there, done that.  Worked like a charm.  But now I use
> nnmail-split and don't think it's slow at all (for 50 to 100 mails in
> the morning).

IMHO, it depends on the speed of the filesystem you're writing to.  At
school, splitting over NFS was unbearably slow, so I did use direct
asynch delivery to nnml folders (this was before gnus-demon was
around, though).  

These days, I'm lucky enough to be using a local ext2 filesystem with
Linux.  This is much, much, much better, but it can still be slow if
you've got 100's of messages.  However, setting up up a gnus-daemon
handler to auto-split mail every 10 mins or so ensures that I never
have a huge number of messages to be split at any given time.
Definitely the way to go if you can leave your Emacs up for days.


To go back to the original question, expiration isn't the only time
when files in your nnml folders get manipulated by Gnus.  Splitting
also clobbers the overview file and articles -- in general, it is a
very bad idea to combine asynch delivery and Gnus splitting.  Use the
latter, asynch delivery is unsupported and needs you to use the
programming skills you mentioned. :>

Even if you don't use Gnus splitting at all, you still have to be
careful with the `B' commands.  Moving an article into a group with `B 
m' can actually lose mail -- Gnus doesn't re-read the overview file or 
the active file before such commands, so an asynch-delivered article
can get clobbered -- which is a lot worse than nov file corruption.

If you're willing to forgo `B m's into asynch groups and Gnus
splitting, you can get away with asynch delivery -- with some work.
You'll need to write a script or two to walk your nnml tree and
compare the output of readdir(3) with the overview file.  Two kinds of 
problems to look for: missing nov entries (easy enough to regenerate), 
and out of order nov entries (these won't be a problem with mailagent, 
since it serializes all delivery, but is definitely a problem with
procmail).  (Lars refused to slow down overview file parsing by
handling the latter correctly in Gnus -- probably just as well, since
then people may assume that asynch delivery is supported. :/)

I used to have such a script, but don't seem to have retained it after
leaving school.  There's a pbiff program I wrote for Perl/Tk (whatever
beta was current in mid 95, so the Tk part is guaranteed to be broken
today) that already walks the nnml tree -- it used to be stored
somewhere on Lars's site.  It's the same thing I modified for checking
the tree, might offer you a start if you plan to walk down this
lonely, unsupported, deprecated, warned-against road. :)

Last, and certainly the least, there's the active file.  It'll get
corrupted (in the sense of missing new entries) too.  This is self
correcting, though, since the next message for that group will fix the 
problem.  It's easy to look for this problem from an external script.


This issue was discussed in great detail in late '95, you might wish
to consult the archives.

My opinion is that splitting to separate spool files is entirely
useless for anything other than the purpose of biffing.  It doesn't
give you any speedups (a minor slowdown actually, depending on your
filesystem), and just complicates things.  If you aren't into biffing, 
avoid the nnml-.*-procmail stuff entirely.  (It does give you a little 
more parallelism with procmail, but that doesn't translate to any
speedup when you hit `g').  

Someone's going to say that procmail/mailagent can do fancier stuff
than Gnus' in this regard.  Sure, but how many actually do such
things?  Even if you do, IMHO, it's better to simply add a header to
the message in the external filter and split on that header in Gnus.

As for biffing, IMHO, a neato-keen way to do it with one spool file is
to modify from(1) to read the split-fancy variable and emulate Gnus'
split to tell you where messages would go or to just show you messages
that go to a particular folder, etc.  The from(1) I use is written in
perl and I would've done this if parsing balanced expressions wasn't
such an annoyance with perl.  Emacs' read functions are so cool for
this kind of thing.  Hmm, maybe have Gnus write out a version that's
easy for from(1) to parse?

-Sudish

Re: saving to nnml groups

[Kai Grossjohann]

>>>>> Sudish Joseph writes:

  Sudish> To go back to the original question, expiration isn't the
  Sudish> only time when files in your nnml folders get manipulated by
  Sudish> Gnus.  Splitting also clobbers the overview file and
  Sudish> articles -- in general, it is a very bad idea to combine
  Sudish> asynch delivery and Gnus splitting.  Use the latter, asynch
  Sudish> delivery is unsupported and needs you to use the programming
  Sudish> skills you mentioned. :>

I presume using nnmh will be better because it doesn't have .overview
files, but I don't grok the implications with regard to the active
file.  Anyone?

(Yes, doing the splitting on a local disk is definitely a win.)

kai
-- 
Life is hard and then you die.

Re: saving to nnml groups

[Sudish Joseph]

Kai Grossjohann writes:
> I presume using nnmh will be better because it doesn't have .overview
> files, but I don't grok the implications with regard to the active
> file.  Anyone?

The active file gets clobbered routinely if you do asynch delivery.
Not A Big Deal, IMO -- the very next message delivered to that group
will rectify the problem.

When I used asynch delivery, my assumptions were (a) Gnus splitting
over NFS was unusable at OSU and (b) that the only important thing was
to ensure that you never clobber an actual article.  All the rest --
.overview/active files -- are very easy to check and (re)generate.

I used to run my nnml-fixer script after quitting Gnus or when my
pbiff showed me articles that Gnus didn't (it'd show you the
From/Subject headers in any group and even an article, if you selected
one).  Since pbiff used the atime/mtime kluge (*) to detect new
articles, it was impossible for both Gnus and pbiff to miss an
article.

-Sudish

(*) I used a slightly modified version of Joe Hildebrand's perl
delivery script.  One of the mods was to do the equivalent of 
"utime +($now = time) - 1, $now, $article;" before exiting.  This
was sufficient to guarantee that either Gnus or pbiff would see
$article -- if pbiff didn't see it, Gnus had to have read it.

Re: saving to nnml groups

[Sudish Joseph]

Kai Grossjohann writes:
> I presume using nnmh will be better because it doesn't have .overview
> files, but I don't grok the implications with regard to the active
> file.  Anyone?

In my other long and boring followup, I forgot one point, so here's
another long and boring followup.  Two for the price of one.  Whee!

IMHO, using any backend other than nnml is a lose with Gnus.  nnml is
fast mainly because it maintains the overview info.  nnfolder has
better I/O patterns than nnml/nnmh -- reading one very large file is
better than reading lots of small files on every filesystem I've seen
-- but the cost of parsing the nnfolder headers to generate the
summary still dominates in a *big* way.  

The difference is *very* noticeable even on this PPro-200, where you'd
expect slowdowns due to filesystem access to be more noticeable than
on a slower CPU.  I use nnfolder for Gcc-based archiving of mail and
Gnus crawls when entering a 200-message folder (200 isn't even _close_
to large for some of my nnml folders where I have 30 day expire times
and often do C-u SPC to search for messages).

Using a backend with NOV support is the way to go, IMHO.  (VM is so
quick because it caches similar info in the headers of the message
itself.  VM even caches summary display info, so summary generation is 
very quick -- but has the drawback of sometimes giving you weird
indentation as you delete parents.)

-Sudish

Re: saving to nnml groups

[Darren/Torin/Who Ever...]

-----BEGIN PGP SIGNED MESSAGE-----

Sudish Joseph, in an immanent manifestation of deity, wrote:
>Even if you don't use Gnus splitting at all, you still have to be
>careful with the `B' commands.  Moving an article into a group with `B 
>m' can actually lose mail -- Gnus doesn't re-read the overview file or 
>the active file before such commands, so an asynch-delivered article
>can get clobbered -- which is a lot worse than nov file corruption.

Well, I don't use mailagent to file things into the groups that I"d B m
messages into, so that's not a problem.  Still my own saver function
might be safer.

>Last, and certainly the least, there's the active file.  It'll get
>corrupted (in the sense of missing new entries) too.  This is self
>correcting, though, since the next message for that group will fix the 
>problem.  It's easy to look for this problem from an external script.

'Course if my script that updates the .overview works with the active
file as well, I shouldn't have the problem.

Darren
- -- 
<torin@daft.com> <http://www.daft.com/~torin> <torin@debian.org> <torin@io.com>
Darren Stalder/2608 Second Ave, @282/Seattle, WA 98121-1212/USA/+1-800-921-4996
@ Do you have your clothes on? I probably don't. Take yours off. Feel better. @
@ Sysadmin, webweaver, postmaster for hire.  C/Perl/CGI programmer and tutor. @

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3
Charset: noconv
Comment: Processed by Mailcrypt 3.4, an Emacs/PGP interface

iQCVAwUBM4d7uI4wrq++1Ls5AQExfAP8CX0IWyfMDE93tU4fIe+xDwS389XuEc8c
Z0DzLu05N1Z9ybtcRgs2mBQk5DMv/JfwiVahMwX4ZvsBFZ7LG5QyYairH4pZyTtE
lNtpSPkMDmNKFapLFT7m9nf93BD47hZhSfFysCSoiZ8JMWxCMeBnWZEDfMnHnisD
52trQfNBQ7A=
=g4EB
-----END PGP SIGNATURE-----

Re: saving to nnml groups

[Darren/Torin/Who Ever...]

-----BEGIN PGP SIGNED MESSAGE-----

Kai Grossjohann, in an immanent manifestation of deity, wrote:
>What I'd do is to replace every occurrence of "nnmh" in .newsrc.eld
>with "nnml" and use M-x nnml-generate-nov-databases RET to create the
>initial overview files.  After that, "B c" and "B m" will do a good
>job of updating the overview files.

The first part here is easy and I figured that out.  I hadn't really
played with the mail-groups commands much before.

>You don't have to use nnmail-split to do it "right".  Check out the
>nnmail-use-procmail, nnmail-procmail-directory and
>nnmail-procmail-suffix variables.  You tell your mailagent to write
>the articles to files in nnmail-procmail-directory, and

Nope.  That doesn't work for me.  I need mailagent to deliver to MH
messages since I have various jobs that process them.  I don't want them
to sit in spool files.

Darren
- -- 
<torin@daft.com> <http://www.daft.com/~torin> <torin@debian.org> <torin@io.com>
Darren Stalder/2608 Second Ave, @282/Seattle, WA 98121-1212/USA/+1-800-921-4996
@ Do you have your clothes on? I probably don't. Take yours off. Feel better. @
@ Sysadmin, webweaver, postmaster for hire.  C/Perl/CGI programmer and tutor. @

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3
Charset: noconv
Comment: Processed by Mailcrypt 3.4, an Emacs/PGP interface

iQCVAwUBM4d6ho4wrq++1Ls5AQEOBAP+MhCI5NacOzXRpyA10L8400X1O/tttS3M
E3xqU3y+1Jtrq3GUhASBKXetm+VNHWeYKCCpje6g3JjF+Rr2i0pVWqbNVMArycoY
XQmYUJcQmLbqLYwYdLJLnItPr4yC8KPZni6rntCLjwxKN3NmTUhrfUzRECU7f4LR
MP0Ao5t28KU=
=i0/v
-----END PGP SIGNATURE-----

Re: saving to nnml groups

[Ching-Mo Chang]

"Darren/Torin/Who Ever..." <torin@daft.com> writes:

> 1) How do I save articles so that they appear in the .overview files?
>    If I just use gnus-summary-save-in-folder, it just uses rcvstore
>    which won't help.  I want to use gnus+nnml to read folders I save to.

I use `gnus-summary-save-in-nnml' as below. It was modified from
`gnus-summary-save-in-mail' function. It save current article to
`gnus-newsgroup-name' folder in "~/News/archive". It will update
the .overview and active files. I hope someone with more experiences
can provide a better one for this. 


(defun gnus-summary-save-in-nnml (&optional arg1 arg2)
  "Save this article using `nnml'."
  (interactive)
  (gnus-set-global-variables)
  (gnus-eval-in-buffer-window gnus-save-article-buffer
    (save-excursion
      (save-restriction
	(widen)
	(let ((nnml-directory "~/News/archive")
	      (nnml-active-file "~/News/archive/active"))
	  (nnml-request-accept-article gnus-newsgroup-name)
	  (nnml-save-nov))
	))))

(setq gnus-default-article-saver 'gnus-summary-save-in-nnml)

(defun gnus-summary-save-article-nnml (&optional arg)
  "Save the current article using `nnml'.
If N is a positive number, save the N next articles.
If N is a negative number, save the N previous articles.
If N is nil and any articles have been marked with the process mark,
save those articles instead."
  (interactive "P")
  (gnus-set-global-variables)
  (let ((gnus-default-article-saver 'gnus-summary-save-in-nnml))
    (gnus-summary-save-article arg)))


-- 
Ching-Mo Chang <chang@wsu.edu> <chang@theta.math.wsu.edu>

Re: saving to nnml groups

[Paul Graham]

i really wish gnus would grow the hooks needed to do clean external mail
processing.  i use procmail to deposit mail in my nnml directories because
it's currently inconvenient to leave gnus running all the time and the
inrush of mail in the morning (300-1000 messages) is far too slow on my
little SS20.  i've modified the external async mail filer (i can't recall
the URL) and nnheader to do some common locking but neither emacs (in terms
of kernel level support) or gnus (architecturally) seem interested in
sharing access to objects like the active file and overview files.  i think
the only common problem is access to the overview database during ``B del''
but it seems to resolve itself properly.

    torin> I've got two questions: 1) How do I save articles so that they
    torin> appear in the .overview files?  If I just use
    torin> gnus-summary-save-in-folder, it just uses rcvstore which won't
    torin> help.  I want to use gnus+nnml to read folders I save to.  2) I
    torin> use mailagent to save *all* my mail to folders.  I never use
    torin> nnmail-split and friends.  (I know I'm not supposed to, but I'm
    torin> a big boy. (and can even program.)) Is expiration the only time
    torin> that Gnus will update the .overview (NOV) file?  Is there a lock
    torin> I can have it check for in case mailagent is there right then?

-- 
 paul
    pjg@acsu.Buffalo.EDU    |public keys at:
                            |     http://urth.acsu.Buffalo.EDU/~pjg/key.html
    if the above contains opinions they are mine unless marked otherwise.

Re: saving to nnml groups

[Lars Magne Ingebrigtsen]

Paul Graham <pjg@acsu.buffalo.edu> writes:

> i really wish gnus would grow the hooks needed to do clean external mail
> processing.  i use procmail to deposit mail in my nnml directories because
> it's currently inconvenient to leave gnus running all the time and the
> inrush of mail in the morning (300-1000 messages) is far too slow on my
> little SS20.  i've modified the external async mail filer (i can't recall
> the URL) and nnheader to do some common locking but neither emacs (in terms
> of kernel level support) or gnus (architecturally) seem interested in
> sharing access to objects like the active file and overview files.  i think
> the only common problem is access to the overview database during ``B del''
> but it seems to resolve itself properly.

I don't think Gnus is ever going to be able to do these kinds of
things.  It's just too much work.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: saving to nnml groups

[Paul Graham]

is this horribly wrong?  it's the sort of thing i'm talking about but since
i don't do lisp it may be that this is secretly eating 5% of my mail (i
should change it to eat 50%).

is a more general salting of these sorts of things about the code what's
too much work or i'm i just hopelessly confused?

*** lisp/nnheader.el.orig       Mon Mar 24 11:02:53 1997
--- lisp/nnheader.el    Mon Apr  7 00:29:24 1997
***************
*** 519,526 ****
--- 519,532 ----
                             (file-name-directory ,temp-file)))
                   (make-directory (file-name-directory ,temp-file) t))
                 ;; Save the file.
+                ;; after using lockfile to get a lock
+                (call-process "lockfile" nil nil nil
+                              (concat ,temp-file ".lock"))
                 (write-region (point-min) (point-max)
                               ,temp-file nil 'nomesg)
+                ;; release the lock
+                (call-process "rm" nil nil nil "-f"
+                              (concat ,temp-file ".lock"))
                 ,temp-results)))
           ;; Kill the buffer.
           (when (buffer-name ,temp-buffer)


    larsi> Paul Graham <pjg@acsu.buffalo.edu> writes:
    >> i really wish gnus would grow the hooks needed to do clean external
    >> mail processing.  i use procmail to deposit mail in my nnml

    larsi> I don't think Gnus is ever going to be able to do these kinds of
    larsi> things.  It's just too much work.

-- 
 paul
    pjg@acsu.Buffalo.EDU    |public keys at:
                            |     http://urth.acsu.Buffalo.EDU/~pjg/key.html
    if the above contains opinions they are mine unless marked otherwise.

Re: saving to nnml groups

[Lars Magne Ingebrigtsen]

Paul Graham <pjg@acsu.buffalo.edu> writes:

> is this horribly wrong?  it's the sort of thing i'm talking about but since
> i don't do lisp it may be that this is secretly eating 5% of my mail (i
> should change it to eat 50%).

[modifications to `nnheader-temp-write' deleted.]

Well, that function isn't used much to write mail files, so making it
lock doesn't hurt, but it doesn't help either.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: saving to nnml groups

[Paul Graham]

i have been unclear.  i want to have mail work like news.  i used to use mh
but it got too slow as my mailbox grew but i still like to use *some* mh
tools (read-only) but i need an overview database so i want nnml not nnmh.
however moving mail from inbox to mailgroup is too slow when batched.  so
... i want mail to be like news.  i took the perl program for async mail
movement and changed it somewhat and added the code to nnheader to lock the
*active* file (i still get to odd note about the overview file changing).

is the right answer to have another method similar to nnml but aware that
something else (the mail equivalent of inews) is updating the overview
database and the active file?



    larsi> [modifications to `nnheader-temp-write' deleted.]
    larsi> Well, that function isn't used much to write mail files, so
    larsi> making it lock doesn't hurt, but it doesn't help either.


-- 
 paul
    pjg@acsu.Buffalo.EDU    |public keys at:
                            |     http://urth.acsu.Buffalo.EDU/~pjg/key.html
    if the above contains opinions they are mine unless marked otherwise.

Re: saving to nnml groups

[Lars Magne Ingebrigtsen]

Paul Graham <pjg@acsu.buffalo.edu> writes:

> is the right answer to have another method similar to nnml but aware that
> something else (the mail equivalent of inews) is updating the overview
> database and the active file?

Yes, probably.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: saving to nnml groups

[David Moore]

Paul Graham <pjg@acsu.buffalo.edu> writes:

> +                (call-process "lockfile" nil nil nil
> +                              (concat ,temp-file ".lock"))
>                  (write-region (point-min) (point-max)
>                                ,temp-file nil 'nomesg)
> +                ;; release the lock
> +                (call-process "rm" nil nil nil "-f"
> +                              (concat ,temp-file ".lock"))

will you be shipping lockfile as well?  I beleive it comes with mh and
procmail, but it's hardly a standard utility (unfortunately).