Not running expiry when I exit a group

Not running expiry when I exit a group

[Jason L Tibbitts III]

OK, I know I have seen this, but I went all over the docs and just couldn't
find it.  Perhaps I'm insane.

I'm tired of waiting for expiry when I exit a group.  I would prefer to run
expiry when I save my .newsrc or when I quit Gnus (which happens rarely).
I recall a variable that told Gnus when it should run expiry, but I just
can't remember.  Please, refresh my memory (or tell me that I'm making this
up).

 - J<

Re: Not running expiry when I exit a group

[Hrvoje Niksic]

Jason L Tibbitts III <tibbs@hpc.uh.edu> writes:

> I'm tired of waiting for expiry when I exit a group.

Me too.  It's terribly annoying for large groups -- and it afflicts
even the commands like `C-x C-s' which quit the group internally.
This was so acute for xemacs-beta that I had to cut the expiry-wait to
only 3 days, to make the expiry process faster.  Now I can live
normally again.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
ED WILL NOT CORRUPT YOUR PRECIOUS BODILY FLUIDS!!

Re: Not running expiry when I exit a group

[Jan Vroonhof]

Hrvoje Niksic <hniksic@srce.hr> writes:

> Me too.  It's terribly annoying for large groups -- and it afflicts
> even the commands like `C-x C-s' which quit the group internally.
> This was so acute for xemacs-beta that I had to cut the expiry-wait to
> only 3 days, to make the expiry process faster.  Now I can live
> normally again.

Same problem here... It seems to me that 'total-expiry' takes longer
as the the other ones. Why is it so slow anyway? Even under elisp a
few thousand compares should be doable in a few secconds.

Jan

Re: Not running expiry when I exit a group

[Jan Vroonhof]

Jan Vroonhof <vroonhof@math.ethz.ch> writes:

> as the the other ones. Why is it so slow anyway? Even under elisp a
> few thousand compares should be doable in a few secconds.

It is not slow because of the compares... (I implemented a dummy
nnmail-expirable-article-p that returns nil always and it is still slow)

1. It is slow because it looks
up each article by searching for it from (point-min).

I just did a

(progn
   (setq i 3000)
   (save-excursion
     (set-buffer bufje)
     (while (< 0 i)
       (setq i (+ i -1))
       (goto-char (point-min))
       (search-forward "X-Gnus-Article-Number: 3451"))))

where bufje is the nnfolder:xemacs-beta buffer and it took about as
long. (nnolder-request-expire-articles does about the same).
I think there is a lot of room for optimization here.

2. My articles in nnfolder:xemacs-beta are numbered 31xx - 3451... but
   Gnus does a
    gnus-request-expire-articles((594 595 596 597 598
   So it searches about 2500 times too many!.

There's some baaaadddd things going on here.

Re: Not running expiry when I exit a group

[Christopher Davis]

JLT> == Jason L Tibbitts III <tibbs@hpc.uh.edu>
Hrv> == Hrvoje Niksic <hniksic@srce.hr>

 JLT> I'm tired of waiting for expiry when I exit a group.

 Hrv> Me too.  It's terribly annoying for large groups -- and it afflicts
 Hrv> even the commands like `C-x C-s' which quit the group internally.

I actually use it as an excuse to get myself out from in front of the
computer for a little while; it's a good way to avoid RSI or eyestrain.
Get up, stretch, get something to drink...

Re: Not running expiry when I exit a group

[Sudish Joseph]

Christopher Davis writes:
> I actually use it as an excuse to get myself out from in front of the
> computer for a little while; it's a good way to avoid RSI or eyestrain.
> Get up, stretch, get something to drink...

Getting a drink every time you quit a group could cause other problems 
than RSI, I guess. :-)

Are you using total-expire?  I use auto-expire and have no speed
problems.  Once a month or so, I do 

M-: (setq gnus-total-expirable-newsgroups ".") RET C-c M-C-x

and get rid of stuff that might have escaped auto-expiry.  This last
is quite slow, but bearable when it's not the norm.

-Sudish

Re: Not running expiry when I exit a group

[Christopher Davis]

SJ> == Sudish Joseph <sj@eng.mindspring.net>

 ckd> I actually use it as an excuse to get myself out from in front of
 ckd> the computer for a little while; it's a good way to avoid RSI or
 ckd> eyestrain.  Get up, stretch, get something to drink...

 SJ> Getting a drink every time you quit a group could cause other
 SJ> problems than RSI, I guess. :-)

Heh.  The only groups it matters for are like xemacs-beta, where the
volume runs high--and the drinks are water, iced tea, or orange juice.

 SJ> Are you using total-expire?

Nope, auto-expire.  It's not a long wait, I just use it as a reminder to
get away from the machine for a few seconds.

Re: Not running expiry when I exit a group

[Kai Grossjohann]

>>>>> Jason L Tibbitts writes:

  Jason> I'm tired of waiting for expiry when I exit a group.

(remove-hook 'gnus-summary-prepare-exit-hook 'gnus-summary-expire-articles)

I manually run expire from the group buffer when I go to lunch.

kai
-- 
A large number of young women don't trust men with beards.
(BFBS Radio)

Re: Not running expiry when I exit a group

[Jason L Tibbitts III]

>>>>> "KG" == Kai Grossjohann <grossjohann@charly.cs.uni-dortmund.de> writes:

KG> (remove-hook 'gnus-summary-prepare-exit-hook
KG> 'gnus-summary-expire-articles)

Thank you.  I was looking in gnus-exit-group-hook, although I still
distinctly remember a list that determined when expiry would be run.

Perhaps I was dreaming it.

 - J<

Re: Not running expiry when I exit a group

[David S. Goldberg]

Here's what I have in my .gnus:

;;;; do expiration in a demon instead of at summary exit
(remove-hook 'gnus-summary-prepare-exit-hook 'gnus-summary-expire-articles)
;;; expire all groups if idle for an hour
(gnus-demon-add-handler 'gnus-group-expire-all-groups nil 60)

Since I leave Emacs up for many days at a time (well, unless there's a
new Gnus or W3 or something to install :-) expiry happens when I'm
away from the office.

--
Dave Goldberg
Post: The Mitre Corporation\MS B305\202 Burlington Rd.\Bedford, MA 01730
Phone: 617-271-3887
Email: dsg@mitre.org

Re: Not running expiry when I exit a group

[Justin Sheehy]

Jason L Tibbitts III <tibbs@hpc.uh.edu> writes:

> I'm tired of waiting for expiry when I exit a group.  I would prefer to run
> expiry when I save my .newsrc or when I quit Gnus (which happens rarely).

(remove-hook 'gnus-summary-prepare-exit-hook 'gnus-summary-expire-articles) 

at the right time will get rid of the expiry hook.  You can then add
it somewhere else, or run expiry whenever you feel like it by pressing
'C-c M-C-x'.  (lunchtime or the end of the day are ideal)

-Justin

 

Re: Not running expiry when I exit a group

[Dewey M. Sasser]

>>>>> "JLT3" == Jason L Tibbitts <tibbs@hpc.uh.edu> writes:

    JLT3> I'm tired of waiting for expiry when I exit a group.

What size groups do you guys read?  I'm on what I consider a "very
large" mailing list (50-200 messages/day), and expiry rarely takes
more than a few seconds.  Are you trying to save those few seconds, or
are you on something with a LOT of traffic (or, perhaps, what back end
are you using? I'm using nnml.)

You probably shouldn't cc: the list in replying to this question, but
if there's curiosity on the list, I can post a summary of any
responses I get.

Thanks,

-- 
Dewey M. Sasser 			voice: (617) 494-6000 
dewey@newvision.com                     PGP Key from public servers  
                                        PGP mail preferred.
---
The  American public knows what it wants, and deserves to get it good
and hard
                     --H. L. Mencken

Re: Not running expiry when I exit a group

[Steinar Bang]

>>>>> "Dewey M. Sasser" <dewey@newvision.com>:

>>>>> "JLT3" == Jason L Tibbitts <tibbs@hpc.uh.edu> writes:
JLT3> I'm tired of waiting for expiry when I exit a group.

> What size groups do you guys read?  I'm on what I consider a "very
> large" mailing list (50-200 messages/day), and expiry rarely takes
> more than a few seconds.  Are you trying to save those few seconds, or
> are you on something with a LOT of traffic (or, perhaps, what back end
> are you using? I'm using nnml.)

I have a noticable delay when I exit the nnml group handling the VFR
mailing list (100 to 150 messages a day).  I've been toying with the
idea that it has something to do with having a lot of sparsely ticked
articles in the group...?

Re: Not running expiry when I exit a group

[David Moore]

Steinar Bang <sb@metis.no> writes:

> I have a noticable delay when I exit the nnml group handling the VFR
> mailing list (100 to 150 messages a day).  I've been toying with the
> idea that it has something to do with having a lot of sparsely ticked
> articles in the group...?

If you use total-expire, Gnus loops over every article in the "active
range" checking the dates on the files to see if they should be
expired.  If you have an old ticked or cached article the active range
can be huge.

This is one of the good arguments for changing Gnus' dependence on the
notion of a single active range.  However, it requires changes to a lot
of code.


-- 
David Moore <dmoore@ucsd.edu>       | Computer Systems Lab      __o
UCSD Dept. Computer Science - 0114  | Work: (619) 534-8604    _ \<,_
La Jolla, CA 92093-0114             | Fax:  (619) 534-1445   (_)/ (_)
<URL:http://oj.egbt.org/dmoore/>    | In a cloud bones of steel.

Re: Not running expiry when I exit a group

[Steinar Bang]

>>>>> David Moore <dmoore@UCSD.EDU>:

> Steinar Bang <sb@metis.no> writes:
>> I have a noticable delay when I exit the nnml group handling the
>> VFR mailing list (100 to 150 messages a day).  I've been toying
>> with the idea that it has something to do with having a lot of
>> sparsely ticked articles in the group...?

> If you use total-expire,

total-expire is what I use, because only total-expire makes sense to
me. 

> Gnus loops over every article in the "active range" checking the
> dates on the files to see if they should be expired.  If you have an
> old ticked or cached article the active range can be huge.  

This is probably what happens to me.

> This is one of the good arguments for changing Gnus' dependence on
> the notion of a single active range.  However, it requires changes
> to a lot of code.

Anything that can make the behaviour of Gnus more efficient, without
requiring me to change my habits, I see as a good thing.


- Steinar

Re: Not running expiry when I exit a group

[Kai Grossjohann]

>>>>> David Moore writes:

  David> If you use total-expire, Gnus loops over every article in the
  David> "active range" checking the dates on the files to see if they
  David> should be expired.  If you have an old ticked or cached
  David> article the active range can be huge.

Does Gnus behave differently with auto-expire?  How is the behavior
different?

Does one need to mark articles as "read" rather than "dormant" or
"ticked" in order to benefit from the improvements from changing from
total-expire to auto-expire?

kai
-- 
A large number of young women don't trust men with beards.
(BFBS Radio)

Re: Not running expiry when I exit a group

[Justin Sheehy]

Kai Grossjohann <grossjohann@charly.cs.uni-dortmund.de> writes:

>   David> If you use total-expire, Gnus loops over every article in the
>   David> "active range" checking the dates on the files to see if they
>   David> should be expired.  If you have an old ticked or cached
>   David> article the active range can be huge.
> 
> Does Gnus behave differently with auto-expire?  How is the behavior
> different?

[disclaimer: I haven't actually checked the code.  this is an assumption]

I think that the only reason that total-expire needs to loop that way
is because the articles are not already marked as expirable.  The loop
is simply to determine which articles are expirable.

Since auto-expire explicitly marks read articles as expirable, it
would not need this behavior.  It can just go ahead to the work of
actually expiring articles.

> Does one need to mark articles as "read" rather than "dormant" or
> "ticked" in order to benefit from the improvements from changing from
> total-expire to auto-expire?

I'm not really sure what you mean here.

'dormant' and 'ticked' are two different ways to say 'unread'.

Dormant or ticked (or other unread) articles are not expired at all
with either total-expire or auto-expire.

The only way that these articles have anything to do with this problem
is that they may keep the active range very large, thus making the
loop in total-expire take a very long time.

-Justin

 

Re: Not running expiry when I exit a group

[Kai Grossjohann]

>>>>> Justin Sheehy writes:

  Kai> Does one need to mark articles as "read" rather than "dormant"
  Kai> or "ticked" in order to benefit from the improvements from
  Kai> changing from total-expire to auto-expire?

  Justin> The only way that these articles have anything to do with
  Justin> this problem is that they may keep the active range very
  Justin> large, thus making the loop in total-expire take a very long
  Justin> time.

Right.  The questions was whether that makes the loop in auto-expire
take a long time, too.  But you just explained that there is no such
loop (to determine which articles are expirable) in auto-expire,
therefore it doesn't make any difference whether I mark articles as
read or as dormant or ticked.

Hm.  Maybe I want to try auto-expire for a while after all?  Hmmm...

kai
-- 
Why doesn't the sun come out at night when the light would come in handy?
(Terry Pratchett)

Re: Not running expiry when I exit a group

[Hrvoje Niksic]

Justin Sheehy <dworkin@ccs.neu.edu> writes:

> I think that the only reason that total-expire needs to loop that way
> is because the articles are not already marked as expirable.  The loop
> is simply to determine which articles are expirable.

Yes, but this thread started with an assertion (hopefully true) that
even in Elisp a thousand or so iterations shouldn't take too long.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
Ask not for whom the <CONTROL-G> tolls.

Re: Not running expiry when I exit a group

[Jan Vroonhof]

Hrvoje Niksic <hniksic@srce.hr> writes:

> > I think that the only reason that total-expire needs to loop that way
> > is because the articles are not already marked as expirable.  The loop
> > is simply to determine which articles are expirable.
> 
> Yes, but this thread started with an assertion (hopefully true) that
> even in Elisp a thousand or so iterations shouldn't take too long.

It depends on what is iterated, which again depends on the back end.
For me (using nnfolder:xemacs-beta ) the slowness comes from the way
article seeking is implemented. As I posted before it does the
equivalent of

(setq bufje <xemacs-beta buffer>)
(progn
   (setq i 3000)
   (save-excursion
     (set-buffer bufje)
     (while (< 0 i)
       (setq i (+ i -1))
       (goto-char (point-min))
       (search-forward "X-Gnus-Article-Number: 3451"))))

3000 * searching 1.5MB is slow... There is no optimization for the
fact that the messages are consecutive.


It should just check all messages one for one (maybe stopping if it
found one that is too new) and intersect the result with the set of
non-read messages.

Jan

Re: Not running expiry when I exit a group

[David Moore]

Kai Grossjohann <grossjohann@charly.cs.uni-dortmund.de> writes:

> >>>>> David Moore writes:
> 
>   David> If you use total-expire, Gnus loops over every article in the
>   David> "active range" checking the dates on the files to see if they
>   David> should be expired.  If you have an old ticked or cached
>   David> article the active range can be huge.
> 
> Does Gnus behave differently with auto-expire?  How is the behavior
> different?

With auto-expire (or manually marking things E), Gnus just removes the
files marked E.  Even if you have a large range, only articles recently
marked E should still be in the list, so it should go faster.
total-expire tries checking every article in the range.

But then again, I don't use auto-expire anymore, as it took too much
thinking about, and doesn't work well with scoring.

> Does one need to mark articles as "read" rather than "dormant" or
> "ticked" in order to benefit from the improvements from changing from
> total-expire to auto-expire?

If you use auto-expire you should get expiry benefits w/o changing those 
things, as it just looks only at the articles marked as expirable.  But
then things like adaptive scoring breaks.

Re: Not running expiry when I exit a group

[Christopher Davis]

DM> == David Moore <dmoore@UCSD.EDU>

 DM> But then again, I don't use auto-expire anymore, as it took too much
 DM> thinking about, and doesn't work well with scoring.

Slight emendation: it works fine with scoring.  It works terribly with
adaptive scoring.  Manually scoring up messages from your boss (and
scoring down messages from "spam@spam.spam" or whatever) works fine.

Re: Not running expiry when I exit a group

[Steinar Bang]

>>>>> Paul Franklin <paul@cs.washington.edu>:

> There's also a _big_ difference in doing this with a local file
> system and a filesystem that's across the network.

True.  If I run Gnus in an emacs process started on the server where
my home directory resides, the respons is acceptable.

But I will in the future occasionally even want to access nnml folders
over ftp connections, so I would like expiry to be as efficient as
possible without juggling disks around.

Hm... how about having a Gnus daemon do expiry every night, and not do
expiry on exiting groups?  Has anyone done this?  Does anyone have a
cookbook receipe they would like to share?


- Steinar

Re: Not running expiry when I exit a group

[Jason L Tibbitts III]

>>>>> "DMS" == Dewey M Sasser <dewey@newvision.com> writes:

DMS> What size groups do you guys read?

Relatively large, 50+ messages/day and total-expire, because I of all
people should be using adaptive scoring.  Lots of old ticked articles.
nnml is the backend.  I get over 1000 pieces of email per day and couldn't
manage it without Gnus.  But then, I didn't get nearly as much mail before
I started using Gnus.  Hmmm....

 - J<

Re: Not running expiry when I exit a group

[Hrvoje Niksic]

"Dewey M. Sasser" <dewey@newvision.com> writes:

> large" mailing list (50-200 messages/day), and expiry rarely takes
> more than a few seconds.  Are you trying to save those few seconds,

Exactly that.  Waiting a few seconds every time I press `q', `C-x C-s'
or similar is very annoying to me.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
Good pings come in small packets.

Re: Not running expiry when I exit a group

[Jason L Tibbitts III]

>>>>> "Hrv" == Hrvoje Niksic <hniksic@srce.hr> writes:

Hrv> Waiting a few seconds every time I press `q', `C-x C-s' or similar is
Hrv> very annoying to me.

Exactly.  I use Gnus for all of my mail, so when I pop into my unfiltered
inbox to read a single message and pop back out, a few seconds is really
too much to waste.  Plus my inbox has a 300+ day expiry time and a couple
hundred ticked articles, some of the old.  Gnus does not handle this
situation well at all.  (Until, that it, I took the expiry run out o
gnus-prepare exit-summary-hook.  Now things are speedy.)

 - J<

Re: Not running expiry when I exit a group

[Wesley Hardaker]

>>>>> In article <ufabu7v89yn.fsf@sina.hpc.uh.edu>, Jason L Tibbitts III <tibbs@hpc.uh.edu> writes:

Jason> I'm tired of waiting for expiry when I exit a group.  I would
Jason> prefer to run expiry when I save my .newsrc or when I quit Gnus
Jason> (which happens rarely).  I recall a variable that told Gnus
Jason> when it should run expiry, but I just can't remember.  Please,
Jason> refresh my memory (or tell me that I'm making this up).

As many have pointed out:
 
 (remove-hook 'gnus-summary-prepare-exit-hook 'gnus-summary-expire-articles)

For other ideas, this is what I do:

 (setq gnus-use-demon t)
 (gnus-demon-add-handler 'gnus-group-expire-all-groups 30 30)

Which runs the expire process automagically every time I've been idle
for 30 minutes or so...  Works like a dream and I rarely ever see it
happen!

Wes

Re: Not running expiry when I exit a group

[Steinar Bang]

>>>>> Wesley Hardaker <whardake@sphysdec1.unil.ch>:

> As many have pointed out:
 
>  (remove-hook 'gnus-summary-prepare-exit-hook 'gnus-summary-expire-articles)

> For other ideas, this is what I do:

>  (setq gnus-use-demon t)
>  (gnus-demon-add-handler 'gnus-group-expire-all-groups 30 30)

> Which runs the expire process automagically every time I've been
> idle for 30 minutes or so...  Works like a dream and I rarely ever
> see it happen!

I'm trying this out now.  Is there a log somewhere of what is expired?
It's nice to know that things are actually happening.


- Steinar

Re: Not running expiry when I exit a group

[Brad Howes]

>>>> 'Steinar Bang (sb@metis.no)' asked the following:
[snip]
 >> Which runs the expire process automagically every time I've been
 >> idle for 30 minutes or so...  Works like a dream and I rarely ever
 >> see it happen!

 SB> I'm trying this out now.  Is there a log somewhere of what is expired?
 SB> It's nice to know that things are actually happening.

For Emacs, look in the `*Messages*' buffer. I see tons of `Expiring...'
and `Deleting...' messages. When I first tried this out, instead of
`gnus-group-expire-all-groups', I used this for the daemon to timestamp
when the expire actually took place:

  (defun my-expire-all ()
     (gnus-group-expire-all-groups)
     (message "%s: expired all gropus" 
              (format-time-string "%D %r" (current-time))))


Brad

-- 
Brad Howes                          Motorola E-Mail ID: XBH001
EMT Development                     SMTP E-Mail: bhowes@cssun3.corp.mot.com
Motorola Corporate - MD H1780       Voice: 602 441 1522  Fax: 602 441 5455