Something fundamental - how nov works

Something fundamental - how nov works

[Harry Putnam]

I thought I sort of understood nov and how it worked but apparently
there is a large hole in my understanding.  I had assumed that if nov
were generated with nnml-generate-nov-databases-1 on a certain group.

That the messages in that group would become visible to gnus even if I
had copied them there behind gnus back, from the command line. Long
as they were legal nnml messages.

For example:
Create nnml group with G m 

Then copy several messages from some other nnml group there, giving
them arbitrary numbers. (from the command line)

If I run nnml-g-n-d-1 on that group some of the messages become
visible but not all.  However if I enter the group and press j
<NUMBER> using numbers of messages that do not appear, they are pulled
up by there number.  The nov file contains information about them yet
they are invisible to gnus.

What is at work here?  Apparently some stuff is written to messages
coming into a group by gnus, like the `X-From-Line' thing and maybe
`Xref' and `Lines'.  Not clear if that has to be there and be accurate
or not.

More example:

I've created an nnml group, copied 3 messages there from another nnml
group (from command line)  I've numbered them 2 4 6.

After running nnml-g-n-db-1 on that group gnus sees two of them 2,4.

No matter what I do 6 isn't visible to gnus.  It is in the nov file
though.  So apparently some other condition is not being met here.

I'm writing a little shell script that writes a certain kind of file
to a gnus nnml directory.  What must I do for gnus to see these
messages?

My script gives the message headers and body that look like:

X-From-Line: reader@reader.local.lan Sat Nov 24 18:40:59 PST 2001
From: 1124184059@reader
To: reader@reader
Subject: start the coffee later
Date: Sat Nov 24 18:40:59 PST 2001

get to it 
or whatever bla bla

Re: Something fundamental - how nov works

[Karl Kleinpaste]

Harry Putnam <reader@newsguy.com> writes:
> What is at work here?

Score-down and expungement.

Enter the group with M-RET and tell us if all are visible.

Re: Something fundamental - how nov works

[Harry Putnam]

Karl Kleinpaste <karl@charcoal.com> writes:

> Harry Putnam <reader@newsguy.com> writes:
>> What is at work here?
>
> Score-down and expungement.

I don't have any score rules in place like that.

> Enter the group with M-RET and tell us if all are visible.

No.  My steps were.. Delete .marks and .overview
Run nnml-gen-n-db-1 ~/Mail2/todo

Dired view of ~/Mail2/todo
  /home/reader/Mail2/todo:
  used 44 available 2887736
  drwxr-xr-x    2 reader   reader       4096 Nov 24 20:20 .
  drwxr-xr-x   13 reader   reader       4096 Nov 24 17:25 ..
  -rw-r--r--    1 reader   reader          4 Nov 24 20:20 .marks
  -rw-------    1 reader   reader        781 Nov 24 20:19 .overview
  -rw-r--r--    1 reader   reader        190 Nov 24 18:40 1
  -rw-r--r--    1 reader   reader        186 Nov 24 18:41 2
  -rw-r--r--    1 reader   reader        173 Nov 24 19:03 3
  -rw-r--r--    1 reader   reader        186 Nov 24 19:06 4
  -rw-r--r--    1 reader   reader        186 Nov 24 19:07 5
  -rw-r--r--    1 reader   reader        184 Nov 24 19:07 6
  -rw-r--r--    1 reader   reader        188 Nov 24 19:07 7


`.overview' contains info on all seven.  Each entry has this one odd
section: 	fake+none+133

Where the terminating number starts at 127 and goes to 133
`.marks' contains `nil'

Now entering with M-<RET> shows:

 1   24-Nov [   0: 1124183958@reader   ] start the coffee now
 1   24-Nov [   0: 1124184059@reader   ] start the coffee later
 1   24-Nov [   0: 1124190303@reader   ] something 6:30
 1   24-Nov [   0: 1124190639@reader   ] some1006657599 1006657602

C-u Z R once inside doesn't show any more.
However `j <number> will pull up those that are missing


Can you tell me briefly how gnus enters a message into a group? I mean
what gnus adds to the message and what it keeps track of.  I realize
nov will get whatever I've stipulated but assume defaults.

A further detail is that `M-c' from group (remove all data) doesn't
change the picture either.

Re: Something fundamental - how nov works

[Paul Jarc]

Harry Putnam <reader@newsguy.com> wrote:
> Then copy several messages from some other nnml group there, giving
> them arbitrary numbers. (from the command line)
>
> If I run nnml-g-n-d-1 on that group some of the messages become
> visible but not all.

Dunno what's up here, but FWIW, nnmaildir would not have a problem
with this.  It *expects* to find new messages in its maildirs written
behind its back, and it never rewrites its messages, so you wouldn't
have to worry about X-From-Line and such.


paul

Re: Something fundamental - how nov works

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

> I thought I sort of understood nov and how it worked but apparently
> there is a large hole in my understanding.  I had assumed that if nov
> were generated with nnml-generate-nov-databases-1 on a certain group.
>
> That the messages in that group would become visible to gnus even if I
> had copied them there behind gnus back, from the command line. Long
> as they were legal nnml messages.
>
> For example:
> Create nnml group with G m 
>
> Then copy several messages from some other nnml group there, giving
> them arbitrary numbers. (from the command line)
>
> If I run nnml-g-n-d-1 on that group some of the messages become
> visible but not all.  However if I enter the group and press j
> <NUMBER> using numbers of messages that do not appear, they are pulled
> up by there number.  The nov file contains information about them yet
> they are invisible to gnus.
>
> What is at work here?

Did you check if the active file was updated correctly?

> Apparently some stuff is written to messages coming into a group by
> gnus, like the `X-From-Line' thing and maybe `Xref' and `Lines'.
> Not clear if that has to be there and be accurate or not.

Nnml shouldn't care.

> More example:
>
> I've created an nnml group, copied 3 messages there from another nnml
> group (from command line)  I've numbered them 2 4 6.
>
> After running nnml-g-n-db-1 on that group gnus sees two of them 2,4.
>
> No matter what I do 6 isn't visible to gnus.  It is in the nov file
> though.  So apparently some other condition is not being met here.

I assume you are entering the group with C-u RET?

Do you have any `display' group properties on the group?  Perhaps post
the output from `G E' on the group, and `G p' on the topics in which
the group is located.

Perhaps you are better of using nnmh, nnmbox, nnmaildir or similar
instead of nnml if you want to write to the server outside of Gnus
though.  Or write your script in elisp, using Gnus functions to enter
articles into the group.

Re: Something fundamental - how nov works

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:

> Did you check if the active file was updated correctly?

I hadn't posted back about it yet, but I did discover that was the
source of at least some troubles.  Apparently when
nnml-generate-nov-databases-1 is run the ~/Mail/active file is not
updated.  Is that a good thing?  Is that even a good place to keep
information on what is read?

> >
> > No matter what I do 6 isn't visible to gnus.  It is in the nov file
> > though.  So apparently some other condition is not being met here.
> 
> I assume you are entering the group with C-u RET?

Yes and also with Karls suggestion of M-<RET>
 
> Do you have any `display' group properties on the group?  Perhaps post
> the output from `G E' on the group, and `G p' on the topics in which
> the group is located.

No topic in use.

Nothing special there, no.  This output shows only 3 messages but this
is accurate at this point>
`G E'
("nnml:todo" 3 nil nil "nnml:")

> Perhaps you are better off using nnmh, nnmbox, nnmaildir or similar
> instead of nnml if you want to write to the server outside of Gnus
> though.  Or write your script in elisp, using Gnus functions to enter
> articles into the group.

Question about this last suggestion:
Do you mean that nnml, nnmbox, nnmaildir in some way are less
sensititve to sneaky input.  (input from outside gnus) can you explain
that a bit? 

I found a way that makes use of your last point there.  Using gnus
functions, without having to know enough lisp to pull it off.

By writing the data in mbox format to a spool where gnus thinks it is
procmail output, slurps it and writes it to the nnml group.

That works pretty well.

A problem I considered there is the event where my script is writing
to the procmail like spool when gnus is slurping.  I guess that is
where locking of some kind comes up.

I investigated `man lockfile' and was disgusted to find that the
discussion and example there are so general as to be useless for a
practical guide.

It appears there is some way to lock the spool file but the syntax
given there doesn't work for me.

With these details:
homeboy script = `write_mbox.sh'

spool file  = ~/spool/todo.in

`todo.in' is the target file of the homeboy script and also seen by gnus
as a file to slurp when pulling in `mail-sources'

There is some chance that both activities could occur at once

So it seems one could introduce a call to `lockfile' in write_mbox.sh
that would lock the target file  during the time the script writes to it.

But various attempts at calling lockfile in the script like this:
cat write_mbox.sh

[...]
  lockfile ~/spool/todo.in.lock
   ..[ script action part]...
  rm -f  ~/spool/todo.in.lock
[...]

Looks like what the man page suggests but by commenting out the rm -f
so the lock stays on for testing, I find I can access that file at
will from the command line, write to it with echo or whatever, so
apparently it isn't being locked at all.

If I say `lockfile ~/spool/todo.in'
lockfile just hangs ad infinitum.
But the file is still accessabel to other activitiy.

99.9 percent of the time this wouldn't come up, but there is some
chance it would.

I thought about trying to employ movemail somehow but it appears to be
usefull only for rmail type files (babyl?)

A couple of people mentioned nnmaildir as a better approach, so I'm
investigating that now.

But it seems that in any case the potential to access a file by two
activities at once would exist.  So I guess I need to figure out how
to lock it for a moment while my script writes to it.

Re: Something fundamental - how nov works

[Dan Christensen]

Harry Putnam <reader@newsguy.com> writes:

> By writing the data in mbox format to a spool where gnus thinks it is
> procmail output, slurps it and writes it to the nnml group.

That's what you should do.

> A problem I considered there is the event where my script is writing
> to the procmail like spool when gnus is slurping.  I guess that is
> where locking of some kind comes up.

Right.

> [...]
>   lockfile ~/spool/todo.in.lock
>    ..[ script action part]...
>   rm -f  ~/spool/todo.in.lock
> [...]

This is correct, except that maybe you should check whether lockfile
succeeds.  (It could be killed while waiting.)  For example:

if ! lockfile ~/spool/todo.in.lock
then
  echo "Unable to lock todo.in."
  exit 13
fi

You might also want to specify something like "-r3" on the lockfile
line so that it just tries a few times and then gives up.

> Looks like what the man page suggests but by commenting out the rm -f
> so the lock stays on for testing, I find I can access that file at
> will from the command line, write to it with echo or whatever, so
> apparently it isn't being locked at all.

All that "locked" means is that future calls to lockfile will wait
until the lock file is removed.  In one window, type "lockfile foo".
In another, type "lockfile foo".  This will wait.  In the first
window, type "rm -f foo" and you will see the second lockfile return.

The idea is that all programs that access the file xxx first
run lockfile on xxx.lock.  

-- 
Dan Christensen
jdc+news@uwo.ca

Re: Something fundamental - how nov works

[Harry Putnam]

Dan Christensen <jdc+news@uwo.ca> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>> By writing the data in mbox format to a spool where gnus thinks it is
>> procmail output, slurps it and writes it to the nnml group.
>
> That's what you should do.
>
>> A problem I considered there is the event where my script is writing
>> to the procmail like spool when gnus is slurping.  I guess that is
>> where locking of some kind comes up.
>
> Right.
>
>> [...]
>>   lockfile ~/spool/todo.in.lock
>>    ..[ script action part]...
>>   rm -f  ~/spool/todo.in.lock
>> [...]
>
> This is correct, except that maybe you should check whether lockfile
> succeeds.  (It could be killed while waiting.)  For example:

[...]

> All that "locked" means is that future calls to lockfile will wait
> until the lock file is removed.  In one window, type "lockfile foo".
> In another, type "lockfile foo".  This will wait.  In the first
> window, type "rm -f foo" and you will see the second lockfile return.

Too bad the man page lacks this clear description. Now I get the idea
a little better.

> The idea is that all programs that access the file xxx first
> run lockfile on xxx.lock.  

OK, that seems kind of lame really since it depends on all apps that
might access a file, playing nicely.  But anyway in the instant case,
does that mean that gnus must be told to run lockfile before it
begins to slurp?  Or does gnus know about that already?

Example:  I run my homeboy  script and it is writing to $FILE

Gnus' cron driven batch fetch rolls around and starts fetching right about
that time.  My homeboy script has preceeded its write to $FILE with a
call to lockfile but hasn't yet finished.

Do I need to do something more to prevent bad things happening in that
scene? OR does gnus DTR by itself.
 

Re: Something fundamental - how nov works

[Dan Christensen]

Harry Putnam <reader@newsguy.com> writes:

> Dan Christensen <jdc+news@uwo.ca> writes:
>
>> The idea is that all programs that access the file xxx first
>> run lockfile on xxx.lock.  
>
> OK, that seems kind of lame really since it depends on all apps that
> might access a file, playing nicely.  But anyway in the instant case,
> does that mean that gnus must be told to run lockfile before it
> begins to slurp?  Or does gnus know about that already?

By default, Gnus uses the movemail program, which should run lockfile.
On my Debian linux system, movemail is hidden away in

  /usr/lib/emacs/21.1/i386-debian-linux-gnu/movemail

You can test that this works by running movemail manually on a file
you have locked.  It should wait until you remove the lockfile.

% cd /tmp
% echo test > test1
% lockfile test1.lock
% /usr/lib/emacs/21.1/i386-debian-linux-gnu/movemail test1 test2
[this waits until I "rm -r test1.lock" in another window]

By doing a similar test, I found that movemail does *not* lock the
destination file, but won't run if that file already exists.

One wishlist item I've always had for movemail was the ability to
append to an existing file, and lock it while writing.  I use a script
to do this currently.

Dan