Long time to exit summary buffer, possible speed enhancement?

Long time to exit summary buffer, possible speed enhancement?

[Shane Holder]

When I try to exit my mail group, it takes a longer than I think it
should ~10 seconds.  There is nothing to expire, nothing that gnus
should do.  I think part of the problem is that my list of articles is
pretty big, and maybe larger than it needs to be.  Gnus trys to be
smart and use ranges of numbers when possible, but if an article
doesn't exist, it will break up the range

  (11068 . 11096) (11098 . 11104)

for example.  Article 11097 doesn't exist, and thus this range gets
broken.  Would it be possible to make gnus understand that if an
article doesnt exist that it can collapse the range?

-- 
Shane Holder                                 e-mail: holder@rsn.hp.com
Hewlett Packard                               phone:     (214)497-4182
3000 Waterview                           I like you, but not enough to
Richardson, TX 75083                     give you my root password.

Re: Long time to exit summary buffer, possible speed enhancement?

[Lars Magne Ingebrigtsen]

Shane Holder <holder@mordor.rsn.hp.com> writes:

> When I try to exit my mail group, it takes a longer than I think it
> should ~10 seconds.  There is nothing to expire, nothing that gnus
> should do.  I think part of the problem is that my list of articles is
> pretty big, and maybe larger than it needs to be.  Gnus trys to be
> smart and use ranges of numbers when possible, but if an article
> doesn't exist, it will break up the range
> 
>   (11068 . 11096) (11098 . 11104)
> 
> for example.  Article 11097 doesn't exist, and thus this range gets
> broken.  Would it be possible to make gnus understand that if an
> article doesnt exist that it can collapse the range?

Well, it can't -- then the range would be incorrect, wouldn't it?

Anyways, range handling isn't all that slow, so the ~10 second delay
probably is caused by something else.  Try `(setq debug-on-quit t)'
and then `C-g' when things "hang" (sorta).  The resulting backtrace
should tell you which function is being slow.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: Long time to exit summary buffer, possible speed enhancement?

[Ken Raeburn]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
> Shane Holder <holder@mordor.rsn.hp.com> writes:
> > When I try to exit my mail group, it takes a longer than I think it
> > should ~10 seconds.  There is nothing to expire, nothing that gnus
> > should do.  I think part of the problem is that my list of articles is
> > pretty big, and maybe larger than it needs to be.
 ...
> Anyways, range handling isn't all that slow, so the ~10 second delay
> probably is caused by something else.  Try `(setq debug-on-quit t)'
> and then `C-g' when things "hang" (sorta).  The resulting backtrace
> should tell you which function is being slow.

Check for sparse article numbers.  I've got some nnml groups where I've
ticked some old articles and read (and thus deleted) lots of others with
higher numbers.  When I check the backtrace as Lars suggests, I find
most of the time is spent in the request-expire loop, checking article
numbers where all traces of the article have long since been deleted.
For example, if you've ticked article 100, and deleted 101-199 months
ago, it'll still check 101-199 to see if they exist.  I think I brought
this up before, during or maybe after the September development cycle.

A hack workaround to this is to move all the articles (mark everything
with `#', then hit `B m') from the mail group to that same mail group,
which causes sequential renumbering and thus removes the empty ranges.
Unfortunately, it also discards any xref info if those messages were
also stored in other mail groups.  And it's only a temporary fix.

If there's no info on an article in the overview, I don't see the point
in checking for the file on disk.

Another lament for the lack of multi-threaded elisp: The expiration
ought to be done in background after exiting the group, at a lower
priority than redisplay and user input processing, and probably async
article prefetching.  Just because it takes a while to stat and maybe
delete a few hundred existing article files (even after the above
sparse-numbering problem is fixed) doesn't mean I should have to wait
before I can start reading from the next group.  Something could
probably be thrown together to use idle cycles for this...

Ken

Re: Long time to exit summary buffer, possible speed enhancement?

[Hallvard B Furuseth]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
>Shane Holder <holder@mordor.rsn.hp.com> writes:
> 
>> Would it be possible to make gnus understand that if an
>> article doesnt exist that it can collapse the range?
> 
> Well, it can't -- then the range would be incorrect, wouldn't it?

Why, no.  Articles get increasing numbers as they arrive.  They will not
not interleave.  Hmm - except if the article number wraps around.  That
can happen, right?  So *big* holes in ranges must not be collapsed since
new articles may be arriving at the beginning of the hole.  Instead,
nonexisting article numbers should be removed from the beginning of a
range following a big hole, to make room for new articles.

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
> Anyways, range handling isn't all that slow, so the ~10 second delay
> probably is caused by something else.  Try `(setq debug-on-quit t)'
> and then `C-g' when things "hang" (sorta).  The resulting backtrace
> should tell you which function is being slow.

Ken Raeburn <raeburn@cygnus.com> writes:
> Check for sparse article numbers.  I've got some nnml groups where I've
> ticked some old articles and read (and thus deleted) lots of others with
> higher numbers.  When I check the backtrace as Lars suggests, I find
> most of the time is spent in the request-expire loop, checking article
> numbers where all traces of the article have long since been deleted.
> For example, if you've ticked article 100, and deleted 101-199 months
> ago, it'll still check 101-199 to see if they exist.  I think I brought
> this up before, during or maybe after the September development cycle.
> 
> A hack workaround to this is to move all the articles (mark everything
> with `#', then hit `B m') from the mail group to that same mail group,
> which causes sequential renumbering and thus removes the empty ranges.
> Unfortunately, it also discards any xref info if those messages were
> also stored in other mail groups.  And it's only a temporary fix.

Besides, you can't do that with news articles.  An old news article will
be followed by a *huge* hole in a newsgroup with high voloume and rapid
expiry, if it is crossposted to a group with slow expiry or if it has an
Expires: header which makes it stay around for some time.


Regards,

Hallvard

Re: Long time to exit summary buffer, possible speed enhancement?

[Ken Raeburn]

Hallvard B Furuseth <h.b.furuseth@usit.uio.no> writes:

> > A hack workaround to this is to move all the articles (mark everything
> > with `#', then hit `B m') from the mail group to that same mail group,

> Besides, you can't do that with news articles.  An old news article will
> be followed by a *huge* hole in a newsgroup with high voloume and rapid
> expiry, if it is crossposted to a group with slow expiry or if it has an
> Expires: header which makes it stay around for some time.

But does leaving a big news group (as opposed to a mail group) incur
the same delays?  (The original poster reported the problem with mail
groups, not news groups.)  I doubt it, since the slowdown I'm seeing
is due to gnus-driven expiration.  If leaving a news group is slow
also, that's probably a different problem.

Re: Long time to exit summary buffer, possible speed enhancement?

[Steinar Bang]

>>>>> Hallvard B Furuseth <h.b.furuseth@usit.uio.no>:

> Ken Raeburn <raeburn@cygnus.com> writes:
>> Check for sparse article numbers.  I've got some nnml groups where I've
>> ticked some old articles and read (and thus deleted) lots of others with
>> higher numbers.  When I check the backtrace as Lars suggests, I find
>> most of the time is spent in the request-expire loop, checking article
>> numbers where all traces of the article have long since been deleted.
>> For example, if you've ticked article 100, and deleted 101-199 months
>> ago, it'll still check 101-199 to see if they exist.  I think I brought
>> this up before, during or maybe after the September development cycle.

>> A hack workaround to this is to move all the articles (mark everything
>> with `#', then hit `B m') from the mail group to that same mail group,
>> which causes sequential renumbering and thus removes the empty ranges.
>> Unfortunately, it also discards any xref info if those messages were
>> also stored in other mail groups.  And it's only a temporary fix.

> Besides, you can't do that with news articles.  An old news article will
> be followed by a *huge* hole in a newsgroup with high voloume and rapid
> expiry, if it is crossposted to a group with slow expiry or if it has an
> Expires: header which makes it stay around for some time.

I too, see this on news articles.  It happens mostly on groups with
high traffic and long expiry (ie. the no.* hierarchy.  (Thank *God*, I
unsubscribed to no.general)).

Re: Long time to exit summary buffer, possible speed enhancement?

[Steinar Bang]

>>>>> Ken Raeburn <raeburn@cygnus.com>:

> Hallvard B Furuseth <h.b.furuseth@usit.uio.no> writes:
>>> A hack workaround to this is to move all the articles (mark everything
>>> with `#', then hit `B m') from the mail group to that same mail group,

>> Besides, you can't do that with news articles.  An old news article will
>> be followed by a *huge* hole in a newsgroup with high voloume and rapid
>> expiry, if it is crossposted to a group with slow expiry or if it has an
>> Expires: header which makes it stay around for some time.

> But does leaving a big news group (as opposed to a mail group) incur
> the same delays?  (The original poster reported the problem with
> mail groups, not news groups.)  I doubt it, since the slowdown I'm
> seeing is due to gnus-driven expiration.  If leaving a news group is
> slow also, that's probably a different problem.

Oh!  Was it when leaving, you saw the delays?  Shows me not to respond
before seeing what it's all about.

In those newsgroup, with big gaps, I see slow startup time (slow for
the number of articles unmarked + number of new).

Are there any traces I can make to find out where it uses its time?

Re: Long time to exit summary buffer, possible speed enhancement?

[Shane Holder]

>>>>> "Ken" == Ken Raeburn <raeburn@cygnus.com> writes:

  Ken> Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
  >> Shane Holder <holder@mordor.rsn.hp.com> writes:

  >> > When I try to exit my mail group, it takes a longer than I
  >> > think it should ~10 seconds.  There is nothing to expire,
  >> > nothing that gnus should do.  I think part of the problem is
  >> > that my list of articles is pretty big, and maybe larger than
  >> > it needs to be.

  >> Anyways, range handling isn't all that slow, so the ~10 second
  >> delay probably is caused by something else.  Try `(setq
  >> debug-on-quit t)' and then `C-g' when things "hang" (sorta).  The
  >> resulting backtrace should tell you which function is being slow.

  Ken> A hack workaround to this is to move all the articles (mark
  Ken> everything with `#', then hit `B m') from the mail group to
  Ken> that same mail group, which causes sequential renumbering and
  Ken> thus removes the empty ranges.  Unfortunately, it also discards
  Ken> any xref info if those messages were also stored in other mail
  Ken> groups.  And it's only a temporary fix.

I just tried this suggestion, and I'm not sure that it's a good thing
to do, it was taking a 'long' time (1 in 3-5 sec).  I have ~1700
articles (expiry of 60 days) and I don't have that long to wait.  It
also seems to loose any marks that have been applied to the articles
(expiry for example), and if you use expiry it will screw that up also
because it will change the modification time on the file, which nnml
uses for expiration.

-- 
Shane Holder                                 e-mail: holder@rsn.hp.com
Hewlett Packard                               phone:     (214)497-4182
3000 Waterview                       Never underestimate the bandwidth
Richardson, TX 75083                 of a truck moving at 70 MPH.

Re: Long time to exit summary buffer, possible speed enhancement?

[Ken Raeburn]

Shane Holder <holder@mordor.rsn.hp.com> writes:

>   Ken> A hack workaround to this is to move all the articles (mark
>   Ken> everything with `#', then hit `B m') from the mail group to
>   Ken> that same mail group, which causes sequential renumbering and
>   Ken> thus removes the empty ranges.  Unfortunately, it also discards
>   Ken> any xref info if those messages were also stored in other mail
>   Ken> groups.  And it's only a temporary fix.
> 
> I just tried this suggestion, and I'm not sure that it's a good thing
> to do, it was taking a 'long' time (1 in 3-5 sec).  I have ~1700
> articles (expiry of 60 days) and I don't have that long to wait.  It

Yes, it's slow.  I'm not sure why.  But that's another thing that
could be explored using debug-on-quit as Lars suggested.  However,
once the pass is finished, exiting the group should be much faster.

> also seems to loose any marks that have been applied to the articles

I thought `B m' was supposed to preserve marks, but I could be wrong.

> (expiry for example), and if you use expiry it will screw that up also
> because it will change the modification time on the file, which nnml
> uses for expiration.

Also true.  Like I said, it's a hack.

Ken

Re: Long time to exit summary buffer, possible speed enhancement?

[Ken Raeburn]

Steinar Bang <sb@metis.no> writes:

> In those newsgroup, with big gaps, I see slow startup time (slow for
> the number of articles unmarked + number of new).
> 
> Are there any traces I can make to find out where it uses its time?

As Lars said earlier in this thread:

> Anyways, range handling isn't all that slow, so the ~10 second delay
> probably is caused by something else.  Try `(setq debug-on-quit t)'
> and then `C-g' when things "hang" (sorta).  The resulting backtrace
> should tell you which function is being slow.

I recommend doing this but hitting C-g a few times during the slow
bits.  If it's in the same code most of the time you trap into the
debugger, that's a more convincing case for that code being the time
sink.  If you do it only once, you could get a traceback in a part
that's actually very efficient if your timing happens to be poor.

Ken

Re: Long time to exit summary buffer, possible speed enhancement?

[Lars Magne Ingebrigtsen]

Hallvard B Furuseth <h.b.furuseth@usit.uio.no> writes:

> Why, no.  Articles get increasing numbers as they arrive.  They will not
> not interleave.  Hmm - except if the article number wraps around.  That
> can happen, right?  So *big* holes in ranges must not be collapsed since
> new articles may be arriving at the beginning of the hole.  Instead,
> nonexisting article numbers should be removed from the beginning of a
> range following a big hole, to make room for new articles.

Actually, I'm not sure what we're discussing here.  I though
originally it was about article marks, and those lists (or rather,
ranges) have to be kept precise.  If this number of ticked articles is
wrong, you'll get faulty information displayed.

Or were we talking about something else?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: Long time to exit summary buffer, possible speed enhancement?

[Kai Grossjohann]

>>>>> Shane Holder writes:

  Shane> When I try to exit my mail group, it takes a longer than I
  Shane> think it should ~10 seconds.

Many things have been said about the "right" way to do this.  I'd like
to suggest that you turn off expiry on exiting groups ("(remove-hook
'gnus-summary-prepare-exit-hook 'gnus-summary-expire-articles)") and
run expiry once a day, say when you go to lunch.

kai
-- 
Life is hard and then you die.

Re: Long time to exit summary buffer, possible speed enhancement?

[Colin Rafferty]

> Hallvard B Furuseth <h.b.furuseth@usit.uio.no> writes:

>> Why, no.  Articles get increasing numbers as they arrive.  They will not
>> not interleave.  Hmm - except if the article number wraps around.  That
>> can happen, right?  So *big* holes in ranges must not be collapsed since
>> new articles may be arriving at the beginning of the hole.  Instead,
>> nonexisting article numbers should be removed from the beginning of a
>> range following a big hole, to make room for new articles.

I did not see the original message (mail problems over the weekend), but
it got me to thinking.  On my 32 bit machine, the maximum article number
in XEmacs is 134217727.  1+ this number is -134217728, which would
probably exercise a bug in Java Gnus.

On the other hand, assuming (pessimisticly) that we all move to 64 bit
machine within ten years (and square MAX_INT), I would have to receive
about 37000 messages per day in order to wrap around.

Even my black-hole newsgroup doesn't get more than 10k messages in a day
(and that is a very bad day indeed).

-- 
Colin

Re: Long time to exit summary buffer, possible speed enhancement?

[Hallvard B Furuseth]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
>Hallvard B Furuseth <h.b.furuseth@usit.uio.no> writes:
> 
>> Why, no.  Articles get increasing numbers as they arrive.  They will not
>> not interleave.  Hmm - except if the article number wraps around.  That
>> can happen, right?  So *big* holes in ranges must not be collapsed since
>> new articles may be arriving at the beginning of the hole.  Instead,
>> nonexisting article numbers should be removed from the beginning of a
>> range following a big hole, to make room for new articles.
> 
> Actually, I'm not sure what we're discussing here.  I though
> originally it was about article marks, and those lists (or rather,
> ranges) have to be kept precise.  If this number of ticked articles is
> wrong, you'll get faulty information displayed.

Huh.  Next time, maybe I should read the entire thread before I jump in.
The message I responded to did not seem to say so, but who knows...


Regards,

Hallvard

Re: Long time to exit summary buffer, possible speed enhancement?

[Hallvard B Furuseth]

>Colin Rafferty <craffert@spspme.ml.com> writes:
>>Hallvard B Furuseth <h.b.furuseth@usit.uio.no> writes:
> 
> assuming (pessimisticly) that we all move to 64 bit
> machine within ten years (and square MAX_INT), I would have to receive
> about 37000 messages per day in order to wrap around.
> 
> Even my black-hole newsgroup doesn't get more than 10k messages in a day
> (and that is a very bad day indeed).

Well, I *have* seen suspiciously low article numbers in occational
newsgroups, so I assume the numbers do get wrapped some times --
regardless of whether or not it is *necessary* to do so.

On the other hand, the original problem in this thread was about mail
groups, so I guess this wrap-around thing is not very relevant in this
context anyway.


Regards,

Hallvard