slow checking for new mail

slow checking for new mail

[stephen farrell]

Ok... so I wrote a bit about this before--it's really really slow for
me to check for new mail.  I have about 80 nnml groups, nfs mounted,
which are priority 1.  when i punch 1 g, it takes about 10-15 seconds
to incorperate new mail (even if only 1 new email).

The thing is, that this i exactly as long as it takes for me to check
all new news (i.e., hit g).  The thing is, to check all new news, it
needs to hit an nntp server for a couple of dozen groups AND use nnmh
backend to check the "active file" from about 20 folders, ballpark
5000 individual files.  I expect this to take 15 seconds over nfs.

I do not expect it to take 15 seconds when just checking out the nnml
groups.  In particular, it would appear that 1 g does the same backend
work as g (for me).

Any thoughts on what I could be doing wrong?  My .gnus and a var dump
should be in the mailing list archive.

Thanks!


--

Steve Farrell

Re: slow checking for new mail

[Mark Eichin]

is it running movemail once for each nnml group?  I've seen that
happen to people before, but never understood why...

Re: slow checking for new mail

[Ken Raeburn]

Mark Eichin <eichin@cygnus.com> writes:

> is it running movemail once for each nnml group?  I've seen that
> happen to people before, but never understood why...

I think the "<number> g" mechanism was to go through all the groups,
and if the group was at level N or below, fetch new mail.  Which means
with 100 level-N groups, you fetch mail 100 times.

People not using POP wouldn't notice, because it would translate to
(maybe) one movemail invocation, and 99 stat calls that find that the
incoming mail file is zero length; since there's no mail to fetch,
movemail isn't run after the first time.  That optimization can't be
done in the POP case, because you can't see into the mail spool.  If
that optimization were removed, I bet everyone would notice "<number>
g" being slow.

Gnus could remember where it has fetched mail from (movemail source
pathname, or post office reference, or whatever) so far in each pass.
Then it wouldn't need to check the same source twice in one pass.

Re: slow checking for new mail

[Lars Magne Ingebrigtsen]

Ken Raeburn <raeburn@cygnus.com> writes:

> People not using POP wouldn't notice, because it would translate to
> (maybe) one movemail invocation, and 99 stat calls that find that the
> incoming mail file is zero length; since there's no mail to fetch,
> movemail isn't run after the first time.  That optimization can't be
> done in the POP case, because you can't see into the mail spool.  If
> that optimization were removed, I bet everyone would notice "<number>
> g" being slow.

nnmail keeps track of what mailboxen it has tried to move in each
"pass", so you won't see this behavior in Gnus 5.4.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: slow checking for new mail

[David Kågedal]

Ken Raeburn <raeburn@cygnus.com> writes:

> People not using POP wouldn't notice, because it would translate to
> (maybe) one movemail invocation, and 99 stat calls that find that the
> incoming mail file is zero length; since there's no mail to fetch,
> movemail isn't run after the first time.

It's still slow. I use nnfolder, and have around 60 mail groups är 40
news groups. When I press "2 g", it hase to go through all of them,
even if only one or two of them needs to be updated. Looking at the
mail spool manually and guessing what folder needs updating, and then
pressing M-g on that group is infinitely faster.

-- 
David Kågedal     Lysator Academic Computer Society       davidk@lysator.liu.se
http://www.lysator.liu.se/~davidk/                              +46-13 17 65 89

Re: slow checking for new mail

[Lars Magne Ingebrigtsen]

stephen farrell <stephen@farrell.org> writes:

> Ok... so I wrote a bit about this before--it's really really slow for
> me to check for new mail.  I have about 80 nnml groups, nfs mounted,
> which are priority 1.  when i punch 1 g, it takes about 10-15 seconds
> to incorperate new mail (even if only 1 new email).

If you have lots of level 1 nnml groups, `1 g' will be slow.  nnml
checks each group seperately in these cases.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: slow checking for new mail

[Colin Rafferty]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]

Lars Magne Ingebrigtsen writes:
> stephen farrell <stephen@farrell.org> writes:

>> Ok... so I wrote a bit about this before--it's really really slow for
>> me to check for new mail.  I have about 80 nnml groups, nfs mounted,
>> which are priority 1.  when i punch 1 g, it takes about 10-15 seconds
>> to incorperate new mail (even if only 1 new email).

> If you have lots of level 1 nnml groups, `1 g' will be slow.  nnml
> checks each group seperately in these cases.

My solution was to have a command that gets that just gets the new mail
once, and updates the *Group* lines of the groups that have new mail.

This works for nnml only:


[-- Attachment #2: Type: text/plain, Size: 758 bytes --]

(define-key gnus-group-mode-map [(meta control g)] 'gnus-group-get-new-nnml)

(defun gnus-group-get-new-nnml ()
  "Get new mail only."
  (interactive)
  (let ((gnus-group-new-mail-list nil)
	(nnml-prepare-save-mail-hook
	 (cons '(lambda ()
		  (setq gnus-group-new-mail-list
			(union gnus-group-new-mail-list
			       (mapcar 'car group-art))))
	       nnml-prepare-save-mail-hook)))
    (nnml-request-scan)
    (let ((gnus-group-marked (mapcar '(lambda (group)
					(concat "nnml:" group))
				     gnus-group-new-mail-list)))
      (gnus-group-get-new-news-this-group))
    (when gnus-goto-next-group-when-activating
      (gnus-group-next-unread-group 1 t))
    (gnus-summary-position-point)
    (gnus-group-list-groups)
    gnus-group-new-mail-list))


[-- Attachment #3: Type: text/plain, Size: 10 bytes --]

-- 
Colin

Re: slow checking for new mail

[stephen farrell]

Colin Rafferty <craffert@ml.com> writes:

> My solution was to have a command that gets that just gets the new mail
> once, and updates the *Group* lines of the groups that have new mail.

Cool.  This is the way it should work.

thanks,

--

Steve Farrell

Re: slow checking for new mail

[stephen farrell]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> stephen farrell <stephen@farrell.org> writes:
> 
> > Ok... so I wrote a bit about this before--it's really really slow for
> > me to check for new mail.  I have about 80 nnml groups, nfs mounted,
> > which are priority 1.  when i punch 1 g, it takes about 10-15 seconds
> > to incorperate new mail (even if only 1 new email).
> 
> If you have lots of level 1 nnml groups, `1 g' will be slow.  nnml
> checks each group seperately in these cases.

nah--it's _really_ checking the nnmh groups (either that or checking
nnmh groups with thousands of files in them takes no time, which I
consider highly unlikely).  The key point is that it takes just as
long to check "1 g" as it does to check "g", where "g" gets a couple
of dozen nntp groups, a couple of dozen nnmh groups (with thousands of
entries), and ~ 50-100 nnml groups.

Gnus v5.4.64; nntp 5.0; nnml 1.0; nnmh 1.0


--

Steve Farrell