Archiving large NNML groups quickly.

Archiving large NNML groups quickly.

[Daniel Pittman]

Now that the NNML marks, et al, are set up so that NNML groups are
independent, is it possible for me to quickly move my current
linux-kernel archive to a backup by doing...

] cd ~/Mail && mv linux.kernel linux.kernel.1

...when Gnus is not running? I think this should just work and I will
get a new group linux.kernel.1 and all, but I wanted to check...

        Daniel

-- 
I'm in favor of legalizing drugs. According to my values system, if people
want to kill themselves, they have every right to do so. Most of the harm that
comes from drugs is because they are illegal.
        -- Milton Friedman

Re: Archiving large NNML groups quickly.

[Harry Putnam]

Daniel Pittman <daniel@rimspace.net> writes:

> Now that the NNML marks, et al, are set up so that NNML groups are
> independent, is it possible for me to quickly move my current
> linux-kernel archive to a backup by doing...
>
> ] cd ~/Mail && mv linux.kernel linux.kernel.1
>
> ...when Gnus is not running? I think this should just work and I will
> get a new group linux.kernel.1 and all, but I wanted to check...

After moving:
I think you will have to add it to ~/Mail/active by hand.

Once thats done you can press "^" in group buffer <RET> on nnml server
and subscribe to it.  I think the nov file will be fine without
nnml-generate-nov-databases-1 <RET> ~/Mail/kernel.1 <RET>

Re: Archiving large NNML groups quickly.

[Simon Josefsson]

On Tue, 30 Oct 2001, Daniel Pittman wrote:

> Now that the NNML marks, et al, are set up so that NNML groups are
> independent, is it possible for me to quickly move my current
> linux-kernel archive to a backup by doing...
> 
> ] cd ~/Mail && mv linux.kernel linux.kernel.1
> 
> ...when Gnus is not running? I think this should just work and I will
> get a new group linux.kernel.1 and all, but I wanted to check...

You need to unsubscribe to the old group, and subscribe to the new one
within Gnus, but other than that I think it would work.  Hm, I think nnml
notices new groups automagically if they are part of the active file,
maybe you can use "echo linux.kernel.1 0 1 0 y >> active" to do everything
from the cmd line.  IMHO the active file should be eliminated, and a
".active" file in each nnml group should be used instead.

Re: Archiving large NNML groups quickly.

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> IMHO the active file should be eliminated, and a ".active" file in
> each nnml group should be used instead.

This means that Gnus has to recursively search ~/Mail, at least if
long file names are not in effect.  This could be quite slow.

But having Gnus rely on the active file being correct is not good, of
course.  What can be done?

kai
-- 
Lisp is kinda like tpircstsoP

Re: Archiving large NNML groups quickly.

[Simon Josefsson]

On Tue, 30 Oct 2001, Kai Großjohann wrote:

> Simon Josefsson <jas@extundo.com> writes:
> 
> > IMHO the active file should be eliminated, and a ".active" file in
> > each nnml group should be used instead.
> 
> This means that Gnus has to recursively search ~/Mail, at least if
> long file names are not in effect.  This could be quite slow.

No recursive searching is needed, I think, just use the subscribed group
name in Gnus to find the filename.  Of course, instead of reading only 1
file (active) you need to read N files, but it is only necessary to do it
once and after that keep things in memory -- right now the active file is
often read from disk.  So in the end, it would probably be about the same
speed.  When listing all groups, a recursive find is needed though.

Or maybe the active file should simply be removed completely (not replaced
by .active), the active info could be kept in .newsrc.eld (.marks)  
instead.  If we change the format of .marks it would even be simple to
manipulate it from shell/perl scripts.  Hm, what do you think?

Re: Archiving large NNML groups quickly.

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> Or maybe the active file should simply be removed completely (not replaced
> by .active), the active info could be kept in .newsrc.eld (.marks)  
> instead.  If we change the format of .marks it would even be simple to
> manipulate it from shell/perl scripts.  Hm, what do you think?

If it is not necessary to recursively search too often, then it's
fine.  In particular, avoiding recursive search on startup and on `g'
would be enough, I think.

Changing the .marks format is nice but not urgent, I guess.

kai
-- 
Lisp is kinda like tpircstsoP

Re: Archiving large NNML groups quickly.

[Per Abrahamsen]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Changing the .marks format is nice but not urgent, I guess.

If we are going to change any (more) file formats, it should be before
5.10 is released (which I guess isn't urgent either).

Re: Archiving large NNML groups quickly.

[Matt Armstrong]

Simon Josefsson <jas@extundo.com> writes:

> On Tue, 30 Oct 2001, Kai Großjohann wrote:
>
>> Simon Josefsson <jas@extundo.com> writes:
>> 
>> > IMHO the active file should be eliminated, and a ".active" file in
>> > each nnml group should be used instead.
>> 
>> This means that Gnus has to recursively search ~/Mail, at least if
>> long file names are not in effect.  This could be quite slow.
>
> No recursive searching is needed, I think, just use the subscribed
> group name in Gnus to find the filename.  Of course, instead of
> reading only 1 file (active) you need to read N files, but it is
> only necessary to do it once and after that keep things in memory --
> right now the active file is often read from disk.  So in the end,
> it would probably be about the same speed.  When listing all groups,
> a recursive find is needed though.
>
> Or maybe the active file should simply be removed completely (not
> replaced by .active), the active info could be kept in .newsrc.eld
> (.marks) instead.  If we change the format of .marks it would even
> be simple to manipulate it from shell/perl scripts.  Hm, what do you
> think?

Keeping the active file out of the backend's directory tree might ease
problems generated by having nnfolder and nnml backends using the same
active file.

But my personal preference would be to see .newsrc.eld contain less,
and push more of the meta data into mostly-human-readable files like
the active and marks files.  This would require some serious effort
though.

-- 
matt

Re: Archiving large NNML groups quickly.

[Steinar Bang]

>>>>> "Matt Armstrong" <matt+dated+1007049498.cdc3aa@lickey.com>:

[snip!]
> But my personal preference would be to see .newsrc.eld contain less,

I _very_ much agree!

Re: Archiving large NNML groups quickly.

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:

> On Tue, 30 Oct 2001, Daniel Pittman wrote:
>
>> Now that the NNML marks, et al, are set up so that NNML groups are
>> independent, is it possible for me to quickly move my current
>> linux-kernel archive to a backup by doing...
>> 
>> ] cd ~/Mail && mv linux.kernel linux.kernel.1
>> 
>> ...when Gnus is not running? I think this should just work and I will
>> get a new group linux.kernel.1 and all, but I wanted to check...
>
> You need to unsubscribe to the old group, and subscribe to the new one
> within Gnus, but other than that I think it would work.  Hm, I think nnml
> notices new groups automagically if they are part of the active file,
> maybe you can use "echo linux.kernel.1 0 1 0 y >> active" to do everything
> from the cmd line.  IMHO the active file should be eliminated, and a
> ".active" file in each nnml group should be used instead.

I've experimented with this a little and this looks like at least one
good way to do it from command line ( a helper tool is included).

1)
   cp -R ~/Mail/target ~/Mail/target.1
2) 
   Get the highest and lowest message numbers in "target.1 and add this entry
   to ~/Mail/active: "target.1 high_num low_num y
   (A homeboy scritp included to extract that information)

3) 
   Two techniques available here  
   a. In a running gnus go to server buffer. Press <RET> on
   nnml (or whatever) server.  Subscribe to target.1
4)
   Back in group buffer press `g' before entering the new group
NOTE:
This method seems to maintain any marks that were in "target".

Helper script:
For usage type "add_active help" but in general:
add_active -d <no args> - display most of the variables inside the script
add_active -s GROUP - Shows what will be appended.
add_active -w GROUP - Write the data to ~/Mial/active
(Note: -w flag causes a backup copy of ~/Mail/active to be
 written to ~/Mail/active_MNTHDAYHOURMINSEC before writing)

(NOTE: Beware of lightly tested script)

cat add_active

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#!/bin/sh
# Keywords: add_active - insert an entry into Gnus ~/Mail/active file
# key Date: Oct 30 2001 2
# &&

## BEGIN DOCUMENTATION[S] ======================================
usage () {
cat >&2 <<EOM
Purpose:  Append entry to Gnus ~/Mail/active file containing
          highest and lowest messagge number like: "GROUP HI LO y"

Usage:  \`$(basename $0) -[dsw] GROUP <-s -w> require an argument
         (Note: do not include path with GROUP)
       -d - <no args allowed> display most of the variables inside
            this script.
       -s - <requires a GROUP arg> Show what the results would be
       -w - <requires a GROUP arg> Writes to active file. 
            (Note: -w flag cause a copy of ~/Mail/active to be saved
             to ~/Mail/active_$(date +"%m%d%H%M%S") before writing.
EOM
} ## end

## BEGIN SAFEGAURDS ============================================
if [ -z "$1" -o "$1" = help ];then
    usage
    exit
fi
## BEGIN VARIABLE[s] ============================================
AWK=/bin/awk
GROUP=$2
ACTIVE=~/Mail/active
BASE=/home/reader/Mail

ACT_ENTRY=$(ls $BASE/$GROUP |sort -n| $AWK '
	NR<2{ORS=" ";last=$0}END {$FNR;print;print last " y"}')

## BEGIN GETOPTS SECTION =========================================
while getopts "ds:w:" opt; do
       case "$opt" in
## Display variables
    d) 
       $AWK '/[[:alnum:]]+=/' $0
        ;;
## show results
    s) echo -e "$GROUP $ACT_ENTRY"
       echo "Type \`$(basename $0) -w GROUP' to write it'"
       ;;
## Write results
    w) 
       cp $ACTIVE "$ACTIVE"_$(date +"%m%d%H%M%S") &&      
       echo "$GROUP $ACT_ENTRY" >> $ACTIVE 
       echo -e ""$GROUP $ACT_ENTRY" written to $ACTIVE"
       ;;
  esac
done

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^