Idea for turbo expiry

Idea for turbo expiry

[Jason L Tibbitts III]

I dipped into the expiry code to see why it was taking so much CPU every
time I exited an nnml group.  It looks to me like it builds a list of all
expirable articles and stats each of them in turn to see if they're old
enough.

My idea is to make the assumption that articles come in chronologically, so
that if an article has a lower number, it it older.  Thus you can run the
list of articles in numerical order and bail after you find one that is too
young to expire.  I figure (judging from how much mail I save in some of my
active folders) that this would cut my expiry time by a factor of 100 or
more.

Yes, it's easy to fool.  Just touch (or edit) a random article and watch it
hold up expiry of everything after it.  I'm willing to make that sacrifice,
at least for some groups.

 - J<

Re: Idea for turbo expiry

[David Moore]

Jason L Tibbitts III <tibbs@hpc.uh.edu> writes:

> I dipped into the expiry code to see why it was taking so much CPU every
> time I exited an nnml group.  It looks to me like it builds a list of all
> expirable articles and stats each of them in turn to see if they're old
> enough.

	Yeah, it does this if you use total-expire.

> My idea is to make the assumption that articles come in chronologically, so
> that if an article has a lower number, it it older.  Thus you can run the
> list of articles in numerical order and bail after you find one that is too
> young to expire.  I figure (judging from how much mail I save in some of my
> active folders) that this would cut my expiry time by a factor of 100 or
> more.

	I was thinking a good thing to do would be to get a list of all
of the files in the directory and intersect that list with the read
list, and only remove things in the intersection.  This wins big over
both the default and the start at low numbers approaches when the total
number of articles in the group is in the 10 thousands and you've only
got say 100 messages actually sitting there.

	But in any case the method used for total-expire can be improved
upon.

-- 
David Moore <dmoore@ucsd.edu>       | Computer Systems Lab      __o
UCSD Dept. Computer Science - 0114  | Work: (619) 534-8604    _ \<,_
La Jolla, CA 92093-0114             | Fax:  (619) 534-1445   (_)/ (_)
<URL:http://oj.egbt.org/dmoore/>    |

Re: Idea for turbo expiry

[Jason L Tibbitts III]

>>>>> "DM" == David Moore <dmoore@UCSD.EDU> writes:

DM> I was thinking a good thing to do would be to get a list of all of the
DM> files in the directory and intersect that list with the read list, and
DM> only remove things in the intersection.

Is it really true that Gnus doesn't know the difference between articles
that are read but not expired and articles that are completely gone?  I can
understand why the front end wouldn't care, but the backend has to have a
way to figure it out without going through the directory.  Couldn't it use
the overview file if available?  I think that would be quicker iterating
through the directory.

I still think it would be worthwhile to let the user choose to assume
chronological ordering so that expiry can bail early.  I have groups with
really long expiry times, like a year, and some of them hold upwards of ten
thousand articles.  These take _forever_ to process.  (I've been using Gnus
since (ding) Gnus Warp 0.6 and have processed a _lot_ of mail through it.)

 - J<

Re: Idea for turbo expiry

[Lars Magne Ingebrigtsen]

Jason L Tibbitts III <tibbs@hpc.uh.edu> writes:

> I dipped into the expiry code to see why it was taking so much CPU every
> time I exited an nnml group.  It looks to me like it builds a list of all
> expirable articles and stats each of them in turn to see if they're old
> enough.

Yup.  The added complexity is there because you can gzip (etc) the
files, so nnml has to dance and jiggle around a bit to find the names
of the files.  

> My idea is to make the assumption that articles come in chronologically, so
> that if an article has a lower number, it it older.  Thus you can run the
> list of articles in numerical order and bail after you find one that is too
> young to expire. 

Or we could just assume that all the articles are in the .overview
file and use that to find out what articles are in the group.  That
should be much quicker, and only slightly less general.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen