mail fetching speed problems 5.2.25

mail fetching speed problems 5.2.25

[David Moore]

	If this has been fixed, please let me know, I'll be upgrading to
red gnus this weekend anyways.

	I've noticed that often pressing 'M-2 g' in the group buffer is
pretty speedy at filing all of my new mail and saying that it was done
fetching the new mail.  But my emacs would then sit looping for another
1-2 minutes.  This delay mostly only happens when I really do have new
mail.  Using elp and the debugger, and peering at the source code, I
notice the following problem:

*** This is a structural problem with how gnus queries backends for new
*** messages, but might be fixable with adding some smarts to
*** nnmail-get-active or nnml-request-list.


gnus-get-unread-articles calls (gnus-activate-group group 'scan) for
every nnml mail group.

gnus-activate-group calls nnml-request-group

nnml-request-scan calls nnmail-get-new-mail for each of those calls.
This means that my new mail is fetched and distributed on the first
call, but the system still attempts for each of the following calls.

nnmail-get-new-mail calls (nnmail-activate 'nnml) on each of these
calls, before it even bothers to see if there was anything in the spool
file, which is obviously empty since it was cleared by the first group.

nnmail-activate then goes through every group checking the time stamps
on the active files.  And the second time through it updates the
internal active list information for the disk active files which were
modifed when the first group caused the spool file to be split.

gnus-get-unread-articles also calls gnus-get-unread-articles-in-group
for every group.

gnus-activate-group _also_ calls gnus-request-group, which hands off to
nnml-request-group.

nnml-request-group then calls (nnmail-activate 'nnml) on every group

The end result here is that for my 104 mail groups: I get 104 calls to
several routines which might be reasonable.  208 calls to
nnmail-activate.  414 calls to nnmail-get-active.  104 calls to
nnmail-get-new-mail.  207 calls to nnml-request-list (which cause half
of the calls to nnmail-get-active).

I call nnmail-get-active 414 times which parses my entire top level nnml
active file 414 times, each time creating a list with 312 elements by
use of a complex regexp.  And each of these 414 times we through the
list in the trash.  So to fetch a single new mail message, I generate
129168 con cells in nnmail-get-active alone.


Please tell me this is fixed.  I suppose a smart thing to do would be to
only have nnml-request-list reparse the active file if it's changed
since the last time it did so.

Of course, it might be nice to change how Gnus talks to it's backends,
so that it could basically just activate everything once, then split the
new mail up.  Then go through and update the unread listings.  As it is
now, it does that to every group with the same method.



-- 
David Moore <dmoore@ucsd.edu>       | Computer Systems Lab      __o
UCSD Dept. Computer Science - 0114  | Work: (619) 534-8604    _ \<,_
La Jolla, CA 92093-0114             | Fax:  (619) 534-1445   (_)/ (_)
<URL:http://oj.egbt.org/dmoore/>    | Solo Furnace Creek 508 -- 1996!

Re: mail fetching speed problems 5.2.25

[David Moore]

David Moore <dmoore@ucsd.edu> writes:
>
> 	If this has been fixed, please let me know, I'll be upgrading to
> red gnus this weekend anyways.

> *** This is a structural problem with how gnus queries backends for new
> *** messages, but might be fixable with adding some smarts to
> *** nnmail-get-active or nnml-request-list.

Oh boy, a self reply.

	Looking at the current rgnus source, the same problem seems to
exist.  nnmail-activate looks very wrong to me, since the nnml and other
backends seem to set their own BACKEND-group-alist by calling
nnmail-get-active directly.  nnmail-activate calls BACKEND-request-list
which generates the list, and then calls nnmail-get-active a second time
on the same data.

	Changing nnmail-activate to not make the unneeded call and also
changing the stored timestamp from (current-time) to 'file-time'
(because of possible nfs and other interactions), I've now cut the time
to fetch a single piece of new mail from 61 seconds to about 15, when
using a non-byte-compiled version of nnmail-activate.  Of course, a lot
of time is still being wasted here.


New version:

(defun nnmail-activate (backend &optional force)
  (let (file timestamp file-time)
    (if (or (not (symbol-value (intern (format "%s-group-alist" backend))))
	    force
	    (and (setq file (condition-case ()
				(symbol-value (intern (format "%s-active-file" 
							      backend)))
			      (error nil)))
		 (setq file-time (nth 5 (file-attributes file)))
		 (or (not
		      (setq timestamp
			    (condition-case ()
				(symbol-value (intern
					       (format "%s-active-timestamp" 
						       backend)))
			      (error 'none))))
		     (not (consp timestamp))
		     (equal timestamp '(0 0))
		     (> (nth 0 file-time) (nth 0 timestamp))
		     (and (= (nth 0 file-time) (nth 0 timestamp))
			  (> (nth 1 file-time) (nth 1 timestamp))))))
	(save-excursion
	  (or (eq timestamp 'none)
	      (set (intern (format "%s-active-timestamp" backend)) 
;;; dmoore@ucsd.edu 25.10.96
;;; it's not always the case that current-time
;;; does correspond to changes in the file's time.  So just compare
;;; the file's new time against its own previous time.
;;;		   (current-time)
		   file-time
		   ))
	  (funcall (intern (format "%s-request-list" backend)))
;;; dmoore@ucsd.edu 25.10.96
;;; BACKEND-request-list already does this itself!
;;;	  (set (intern (format "%s-group-alist" backend)) 
;;;	       (nnmail-get-active))
	  ))
    t))

-- 
David Moore <dmoore@ucsd.edu>       | Computer Systems Lab      __o
UCSD Dept. Computer Science - 0114  | Work: (619) 534-8604    _ \<,_
La Jolla, CA 92093-0114             | Fax:  (619) 534-1445   (_)/ (_)
<URL:http://oj.egbt.org/dmoore/>    | Solo Furnace Creek 508 -- 1996!