Catching up Mail Groups

Catching up Mail Groups

[Aharon (Al) Schkolnik]

Hi all.

	I really like gnus. (Shall I say that a few more times before
I start complaining ?  ;-) ).


	My biggest complaint is the way read articles are handled in
mail groups. I use :


(setq gnus-auto-expirable-newsgroups ".*:mail\..*")


which means that if I mark articles with E, they get expired. That is
a-good-thing. The problem is that I frequently forget, and mark
articles with d (that is what I'm used to doing in NNTP groups), or
simply want to catch-up the the group (with c). Then, the
articles are not expired. They stay around (invisibly) until I get
around to painstakenly cleaning them up by entering the group with C-u
<spc>, and then doing E down-arrow E down-arrow ad-nasuem. This is
a-bad-thing. So, I tried:

(setq gnus-total-expirable-newsgroups ".*:mail\..*")


This, I think, caused the desired effect of expiring the
articles. Although, quite frankly, I don't really understand the
difference between auto-expirable, and total-expirable. Anyway, the
performance hit was totally unacceptable - it took many, many seconds
each time I left a mail group (Expiring articles ........). This is
a-bad-thing. 


What I, and I believe many others, want is for articles in mail groups
which are not marked to be saved (u command), and which are read or
deleted, to be deleted after the expiration period. I guess what this
all means is that we want deletion (or whatever is done by
catching-up, the d command, etc.) to be treated as expiration. Of
course, this must be efficient. 


Thanks for your attention.


--
  The day is short, and the work is great,     |   Aharon Schkolnik
  and the laborers are lazy, and the reward    |   Aharon@Matat.Health.Gov.IL
  is great, and the Master of the house is     |
  impatient. - Ethics Of The Fathers Ch. 2     |

Re: Catching up Mail Groups

[Jason L Tibbitts III]

>>>>> "AS" == Aharon (Al) Schkolnik <aharon@healdb.matat.health.gov.il> writes:

AS> What I, and I believe many others, want is for articles in mail groups
AS> which are not marked to be saved (u command), and which are read or
AS> deleted, to be deleted after the expiration period.  I guess what this
AS> all means is that we want deletion (or whatever is done by catching-up,
AS> the d command, etc.) to be treated as expiration.

This is exactly, positively, 100% what total-expire does.  Every message
that is killed, deleted or caught-up is deleted after the delay you set
(default 28 days).  Now, if I'm not mistaken, Gnus has to stat the files
(nnml) to see if they should go away.  This could, perhaps, be made more
efficient by storing some kind of age of each article somewhere so that the
stat doesn't get done.  I don't know what Gnus has to do to expire things
in an nnfolder group; I suspect it's a pile of region deletions which
causes a bunch of buffer gap movement and VM thrashing.  But then, nnfolder
isn't very good for big folders.  Myself, I'm drooling over the prospect of
nngdbm.

BTW, on a large nnml group (~1000 articles) with 28 day expiry it takes
about three seconds to do the expiry if the disk is local to the machine.
This goes up dramatically if I'm on an NFS mounted volume, even when the
interconnect is our 1+Gbit network.  NFS sucks for this kind of thing.

AS> Of course, this must be efficient.

That's just the problem, isn't it?  It's kind of difficult to say where the
bottleneck is since you don't provide any information at all about your
environment, how many articles you're looking at or what backends you're
using.  Perhaps with a little more information we could give you some
optimization tips.

 - J<

Re: Catching up Mail Groups

[Greg Stark]

I'm surprised at all this talk about expiration being slow; I'm using nnml
stored in that that lumbering behemoth filesystem, AFS.  And I read my mail
frequently throughout the day and get a fair amount of traffic, so it runs the
expiry process much more often than necessary and frequently deletes mail.

It takes only a few extra seconds between each group which seems reasonably
efficient. It looks like maybe some specific backend or option is triggering
some inefficient behaviour. 

I use gnus-total-exirable-newsgroups set to ^nnml:mail\.
and I use nnml as my backend.  I think dates are in the overview so
nnmh might be much less efficient.  The point movement in nnmbox/nnbabyl
should be fine, better even if it sorted the article-ids numerically instead
of alphabetically (as per my previous mail). Though it probably has to parse
date headers for all the newsgroups then run through them all a second time to
actually expire stuff, hmm that's sort of innefficient.

greg

Re: Catching up Mail Groups - Efficiency - More Info.

[Aharon (Al) Schkolnik]

>>>>> "Jason" == Jason L Tibbitts <tibbs@uh.edu> writes:

>>>>> "AS" == Aharon (Al) Schkolnik <aharon@healdb.matat.health.gov.il> writes:

    AS> Of course, this must be efficient.

    Jason> That's just the problem, isn't it?  It's kind of difficult
    Jason> to say where the bottleneck is since you don't provide any
    Jason> information at all about your environment, how many
    Jason> articles you're looking at or what backends you're using.
    Jason> Perhaps with a little more information we could give you
    Jason> some optimization tips.



Here is what I believe to be the relevant part of the newsrc.eld file:


("nnbabyl:mail.gnus" 2 ((1 . 2191)) ((save 1818 1861 2151 2154)
(expire (2089 . 2096) (2098 . 2121) (2123 . 2124) (2126 . 2159) 2161
(2164 . 2181) (2183 . 2186) (2188 . 2190)) (reply 1477 1861 2191)
(tick 489 639 644 866 987 991 (1477 . 1478) 1550 1643 1655 1661 1676
(1716 . 1717) 1727 1815 1818 1834 1861 1870 1877 1891 1899 1910 1919
1960 2036 2038 2097 2122 2125 2160 (2162 . 2163) 2182 2187 2191))
(nnbabyl ""))

--
  The day is short, and the work is great,     |   Aharon Schkolnik
  and the laborers are lazy, and the reward    |   Aharon@Matat.Health.Gov.IL
  is great, and the Master of the house is     |
  impatient. - Ethics Of The Fathers Ch. 2     |

Re: Catching up Mail Groups - Efficiency - More Info.

[Lars Magne Ingebrigtsen]

aharon@healdb.matat.health.gov.il (Aharon (Al) Schkolnik) writes:

> Here is what I believe to be the relevant part of the newsrc.eld file:
> 
> ("nnbabyl:mail.gnus" 2 ((1 . 2191)) ((save 1818 1861 2151 2154)
[...]
> (tick 489 639 644 866 987 991 (1477 . 1478) 1550 1643 1655 1661 1676

This is the problem.  When a group has a wide "spread" (which is the
result when old articles are ticked), then total-expiry will be slow.
nnbabyl will here try to look for articles in the 489-2191 range each
time expiry is performed.  This will take quite a while.

-- 
  "Yes.  The journey through the human heart 
     would have to wait until some other time."

Renumbering articles in mail groups

[Aharon (Al) Schkolnik]

Hi all.


	A few days ago, I reported that total-expiry was very
slow. Lars explained that the problem is that the articles in some of
my groups are "widely spread". That is, there is a large gap between
the number of the first article in the group and the last article in
the group. This is caused by ticking articles, and leaving them in the
group for a long time, and in the meantime ticking newer article. Now,
I'm wondering if there is a clean way to have gnus renumber the
articles. I use babyl. Following are Lars and my messages. 


TIA.

Aharon


>>>>> "Aharon" == Aharon (Al) Schkolnik <aharon@healdb.matat.health.gov.il> writes:

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
    Lars> aharon@healdb.matat.health.gov.il (Aharon (Al) Schkolnik)
    Lars> writes:
    >>> Here is what I believe to be the relevant part of the
    >>> newsrc.eld file:
    >>> 
    >>> ("nnbabyl:mail.gnus" 2 ((1 . 2191)) ((save 1818 1861 2151
    >>> 2154)
    Lars> [...]
    >>> (tick 489 639 644 866 987 991 (1477 . 1478) 1550 1643 1655
    >>> 1661 1676

    Lars> This is the problem.  When a group has a wide "spread"
    Lars> (which is the result when old articles are ticked), then
    Lars> total-expiry will be slow.  nnbabyl will here try to look
    Lars> for articles in the 489-2191 range each time expiry is
    Lars> performed.  This will take quite a while.
--
  The day is short, and the work is great,     |   Aharon Schkolnik
  and the laborers are lazy, and the reward    |   Aharon@Matat.Health.Gov.IL
  is great, and the Master of the house is     |
  impatient. - Ethics Of The Fathers Ch. 2     |

Re: Renumbering articles in mail groups

[Muhammad Farrukh Khan]

Aharon (Al) Schkolnik <aharon@healdb.matat.health.gov.il> writes:

> 
> I'm wondering if there is a clean way to have gnus renumber the
> articles. I use babyl. Following are Lars and my messages. 
> 
> Aharon
> 
 
I had the same query few days ago.  I got a response about reducing gaps, but
keeping high numbers, eg. 1 99 becomes 99 100.

I use nnml.  What I chose to do was to use an MH routine, folder.
So, folder -all -recursive -pack -verbose cleanly packed all my nnml
directories.  I then applied nnml-gen-nov-databases.

Don't know the side effects as yet, but I guess it works.  Those without MH
can perhaps write a Perl one-liner to renumber all the files in the
subdirectories of a directory.

Farrukh

-- 
-----------------------------------------
Muhammad Farrukh Khan, khan@cs.purdue.edu
-----------------------------------------

Re: Renumbering articles in mail groups

[Lars Magne Ingebrigtsen]

aharon@healdb.matat.health.gov.il (Aharon (Al) Schkolnik) writes:

> Now, I'm wondering if there is a clean way to have gnus renumber the
> articles.

`M P b' to process-mark all articles, `B m GROUP RET' to move the
messages.  The trick is to just move them to the same group that
you're currently in.

-- 
  "Yes.  The journey through the human heart 
     would have to wait until some other time."

Re: Catching up Mail Groups

[Hallvard B Furuseth]

jvinson@cheux.ecs.umass.edu (Jack Vinson) writes:
> >>>>> "AS" == Aharon Schkolnik <aharon@healdb.matat.health.gov.il> writes:
> 
> AS> Now, I noticed another thing which might be related. When I
> AS> entered the group to see exactly what is in it (using <ctrl-u> <spc>)
> AS> gnus asked me if I wanted to see all 1703 articles ! Why would it
> AS> think there are 1703 articles ?
> 
> Because you have saved those 36 articles over a period of time, and the
> lowest number is, say 50 and the highest is 1753.  Since Gnus is a
> newsreader, it assumes that the article count is continuous between the
> lowest and highest article numbers.

"since it's a newsreader"?  That assumtion can be _very_ wrong in news:
For example if the earliest article is crossposted to a low-volume
newsgroup with slower expiry, or if the article has a very late expiry
date -- for example because it's a FAQ posting, or group charter, or
some other article which considers itself more important than others in
the group.



Regards,

Hallvard

Re: Catching up Mail Groups

[Jack Vinson]

>>>>> "AS" == Aharon Schkolnik <aharon@healdb.matat.health.gov.il> writes:

AS> Now, I noticed another thing which might be related. When I
AS> entered the group to see exactly what is in it (using <ctrl-u> <spc>)
AS> gnus asked me if I wanted to see all 1703 articles ! Why would it
AS> think there are 1703 articles ?

Because you have saved those 36 articles over a period of time, and the
lowest number is, say 50 and the highest is 1753.  Since Gnus is a
newsreader, it assumes that the article count is continuous between the
lowest and highest article numbers.  Obviously, when you enter with C-u SPC
it will only get your saved articles and those which have yet to be
expired.

-- 
(setq sender '((name . "Jack Vinson") (email . "jvinson@cheux.ecs.umass.edu")
               (employer . "University of Massachusetts, Jim Douglas")
               (phone . "413.545.6144")
               (url . <http://www.cis.upenn.edu/~vinson/>))

Re: Catching up Mail Groups

[Aharon Schkolnik]

Mail-from: From aharon Wed Mar 20 11:58:23 1996
Received: by healdb.matat.health.gov.il; id AA15953; Wed, 20 Mar 1996 11:57:29 +0300
References: <nag1cfkdi.fsf@healdb.matat.health.gov.il>
	<ufag2b4i77j.fsf@sina.hpc.uh.edu>
In-Reply-To: Jason L Tibbitts III's message of 20 Mar 1996 00:56:00 -0600
Message-Id: <n91gwf8g7.fsf@healdb.matat.health.gov.il>
Mime-Version: 1.0 (generated by tm-edit 7.44)
Content-Type: text/plain; charset=US-ASCII
Lines: 49
Xref: healdb.matat.health.gov.il mail.from_me:84
X-Gnus-Newsgroup: mail.from_me:84   Wed Mar 20 11:58:41 1996

>>>>> "Jason" == Jason L Tibbitts <tibbs@uh.edu> writes:

>>>>> "AS" == Aharon (Al) Schkolnik <aharon@healdb.matat.health.gov.il> writes:
    AS> What I, and I believe many others, want is for articles in
    AS> mail groups which are not marked to be saved (u command), and
    AS> which are read or deleted, to be deleted after the expiration
    AS> period.  I guess what this all means is that we want deletion
    AS> (or whatever is done by catching-up, the d command, etc.) to
    AS> be treated as expiration.

    Jason> This is exactly, positively, 100% what total-expire does.
    Jason> Every message that is killed, deleted or caught-up is
    Jason> deleted after the delay you set (default 28 days).  


	Actually, that's what I thought. I just wasn't sure. Then, the
problem which remains is the performance problem. See below.


    Jason> BTW, on a large nnml group (~1000 articles) with 28 day
    Jason> expiry it takes about three seconds to do the expiry if the
    Jason> disk is local to the machine.


    AS> Of course, this must be efficient.

    Jason> That's just the problem, isn't it?  It's kind of difficult
    Jason> to say where the bottleneck is since you don't provide any
    Jason> information at all about your environment, how many
    Jason> articles you're looking at or what backends you're using.

	Here's my example. I use babyl. I use the default expiration
period (I thought it was 7 days). I have a group with 132
articles left in it. 35 are saved (marked !). The rest are expired
(marked E). It took 100 seconds to expire the articles when I left the
group ! Now, I noticed another thing which might be related. When I
entered the group to see exactly what is in it (using <ctrl-u> <spc>)
gnus asked me if I wanted to see all 1703 articles ! Why would it
think there are 1703 articles ?


TIA.


--
  The day is short, and the work is great,     |   Aharon Schkolnik
  and the laborers are lazy, and the reward    |   Aharon@Matat.Health.Gov.IL
  is great, and the Master of the house is     |
  impatient. - Ethics Of The Fathers Ch. 2     |