Getting new mail really slow with nnml marks

Getting new mail really slow with nnml marks

[David Z Maze]

Since the nnml not-in-newsrc.eld mark code got added, sucking in new
mail has been really slow for me.  Setting (setq gnus-verbose-backends
10) and pressing 'g' from the group buffer reveals that Gnus is
reading, updating, and writing the mark files for every single group,
regardless of whether or not it has new mail.  This is probably even
slower in my case, since (a) my mail is on a networked filesystem, (b)
I'm using XEmacs, and (c) I appear to have 637 nnml groups.  Is it
possible to cut down the "update marks" phase to only update the mark
files for groups where there actually is new mail?

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Getting new mail really slow with nnml marks

[Simon Josefsson]

On Wed, 17 Oct 2001, David Z Maze wrote:

> Since the nnml not-in-newsrc.eld mark code got added, sucking in new
> mail has been really slow for me.  Setting (setq gnus-verbose-backends
> 10) and pressing 'g' from the group buffer reveals that Gnus is
> reading, updating, and writing the mark files for every single group,
> regardless of whether or not it has new mail.  This is probably even
> slower in my case, since (a) my mail is on a networked filesystem, (b)
> I'm using XEmacs, and (c) I appear to have 637 nnml groups.  Is it
> possible to cut down the "update marks" phase to only update the mark
> files for groups where there actually is new mail?

Since 2001-09-08 this is precisely what should be happening.  Have you
checked out since then?

If so, could you quote the exact messages from the backend?  Could you
also do a `stat ~/Mail/somennmlgroup/.marks' to see if perhaps there is
something weird with the modtime field? (I guess some network filesystems
could skip that field.  What fs are you using?)

Re: Getting new mail really slow with nnml marks

[David Z Maze]

Simon Josefsson <jas@extundo.com> writes:
SJ> On Wed, 17 Oct 2001, David Z Maze wrote:
 DZM> Since the nnml not-in-newsrc.eld mark code got added, sucking in new
 DZM> mail has been really slow for me.  Setting (setq gnus-verbose-backends
 DZM> 10) and pressing 'g' from the group buffer reveals that Gnus is
 DZM> reading, updating, and writing the mark files for every single group,
 DZM> regardless of whether or not it has new mail.
SJ> 
SJ> Since 2001-09-08 this is precisely what should be happening.  Have you
SJ> checked out since then?

Yeah; I typically update once a week or so, depending on how broken
ding makes the CVS version sound.  :-)  Just updated again, and the
problem is still there.

SJ> If so, could you quote the exact messages from the backend?

Reading active file from news via nntp...
Reading active file via nnml...
nnml: Reading incoming mail from pop...
Wrote /afs/sipb.mit.edu/user/dmaze/Mail/mail/lists/gnus/ding/6328
Wrote /afs/sipb.mit.edu/user/dmaze/Mail/mail/lists/handhelds/familiar/936
Wrote /afs/sipb.mit.edu/user/dmaze/Mail/mail/lists/reuse/15551
Wrote /afs/sipb.mit.edu/user/dmaze/Mail/mail/lists/gnus/ding/6329
Wrote /afs/sipb.mit.edu/user/dmaze/Mail/mail/misc/impersonal/8355
Wrote /afs/sipb.mit.edu/user/dmaze/Mail/mail/lists/handhelds/familiar/937
Wrote /afs/sipb.mit.edu/user/dmaze/Mail/mail/lists/debian/user/177193
nnml: Reading incoming mail (8 new)...done
Reading active file via nnml...
Reading active file via nnml...done
Reading active file via nndsc...
nndsc: Reading 53 groups...
nndsc: Reading groups... 53 left (waiting for info-afs)
nndsc: Reading groups... 52 left (waiting for openafs-announce)
(...)
nndsc: Reading groups... 2 left (waiting for ca)
nndsc: Reading groups... 1 left (waiting for vi)
nndsc: Reading groups...done
Checking new news...
Updating marks for mail.cag.admin...
Updating marks for mail.cag.admin...done
Updating marks for mail.classes.17...
Updating marks for mail.classes.17...done
Updating marks for mail.classes.18...
Updating marks for mail.classes.18...done
Updating marks for mail.classes.6.035.cvs-log...
Updating marks for mail.classes.6.035.cvs-log...done
(...)
Updating marks for mail.cag.projects.streamit...
Updating marks for mail.cag.projects.streamit...done
Updating marks for mail.cag.misc...
Updating marks for mail.cag.misc...done
Checking new news...done

(There are also quite a few garbage collects while doing the "updating
marks" bit, but these don't seem to show up in the XEmacs log.)

SJ> Could you also do a `stat ~/Mail/somennmlgroup/.marks' to see if
SJ> perhaps there is something weird with the modtime field?

{5} Mail% stat mail/cag/misc/.marks
  File: "mail/cag/misc/.marks"
  Size: 84              Blocks: 2          IO Block: 4096   Regular File
Device: ah/10d  Inode: 704955028   Links: 1    
Access: (0600/-rw-------)  Uid: ( 2367/   dmaze)   Gid: (  101/     gdm)
Access: Tue Oct 16 19:01:23 2001
Modify: Tue Oct 16 19:01:23 2001
Change: Tue Oct 16 19:01:23 2001

(That's a little weird; I'd expect at least one of those times to be
Wed Oct 17 16:50:00 2001 or thereabouts.)

SJ> (I guess some network filesystems could skip that field.  What fs
SJ> are you using?)

Transarc AFS, with both the client and servers being on Suns running
Solaris, I believe.

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Getting new mail really slow with nnml marks

[David Z Maze]

David Z Maze <dmaze@mit.edu> writes:
 SJ> Could you also do a `stat ~/Mail/somennmlgroup/.marks' to see if
 SJ> perhaps there is something weird with the modtime field?
DZM> 
DZM> {5} Mail% stat mail/cag/misc/.marks
DZM>   File: "mail/cag/misc/.marks"
DZM>   Size: 84              Blocks: 2          IO Block: 4096   Regular File
DZM> Device: ah/10d  Inode: 704955028   Links: 1    
DZM> Access: (0600/-rw-------)  Uid: ( 2367/   dmaze)   Gid: (  101/     gdm)
DZM> Access: Tue Oct 16 19:01:23 2001
DZM> Modify: Tue Oct 16 19:01:23 2001
DZM> Change: Tue Oct 16 19:01:23 2001
DZM> 
DZM> (That's a little weird; I'd expect at least one of those times to be
DZM> Wed Oct 17 16:50:00 2001 or thereabouts.)

Checking on another group that actually did get new mail
(mail.lists.debian.devel is a good pick), the access/modify/change
times are current, though.  (And stat's output is in local time, not
UTC, FWIW.)

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Getting new mail really slow with nnml marks

[Simon Josefsson]

David Z Maze <dmaze@MIT.EDU> writes:

> SJ> Could you also do a `stat ~/Mail/somennmlgroup/.marks' to see if
> SJ> perhaps there is something weird with the modtime field?
> 
> {5} Mail% stat mail/cag/misc/.marks
>   File: "mail/cag/misc/.marks"
>   Size: 84              Blocks: 2          IO Block: 4096   Regular File
> Device: ah/10d  Inode: 704955028   Links: 1    
> Access: (0600/-rw-------)  Uid: ( 2367/   dmaze)   Gid: (  101/     gdm)
> Access: Tue Oct 16 19:01:23 2001
> Modify: Tue Oct 16 19:01:23 2001
> Change: Tue Oct 16 19:01:23 2001
> 
> (That's a little weird; I'd expect at least one of those times to be
> Wed Oct 17 16:50:00 2001 or thereabouts.)

Yes, AFS has only one value for mtime/atime/ctime but it should have
the mtime semantics (with Transarc at least, Arla on Linux fakes
atime/mtime/ctime somehow).  But anyway, it shouldn't matter as long
as the mtime value isn't updated by reading alone.  Is it?

Re: Getting new mail really slow with nnml marks

[David Z Maze]

Simon Josefsson <jas@extundo.com> writes:
SJ> Yes, AFS has only one value for mtime/atime/ctime but it should have
SJ> the mtime semantics (with Transarc at least, Arla on Linux fakes
SJ> atime/mtime/ctime somehow).  But anyway, it shouldn't matter as long
SJ> as the mtime value isn't updated by reading alone.  Is it?

*checks*  Nope.  (Did 'stat $i; cat $i; stat $i', and the mtime is old
and didn't change.)

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Getting new mail really slow with nnml marks

[Simon Josefsson]

David Z Maze <dmaze@MIT.EDU> writes:

> Simon Josefsson <jas@extundo.com> writes:
> SJ> Yes, AFS has only one value for mtime/atime/ctime but it should have
> SJ> the mtime semantics (with Transarc at least, Arla on Linux fakes
> SJ> atime/mtime/ctime somehow).  But anyway, it shouldn't matter as long
> SJ> as the mtime value isn't updated by reading alone.  Is it?
> 
> *checks*  Nope.  (Did 'stat $i; cat $i; stat $i', and the mtime is old
> and didn't change.)

Could you evaluate (nnml-marks-changed-p "nnml+foo:bar") to see if it
always return t?  Maybe you could edebug through `nnml-save-marks' to
see what happens?

On theory would be that the `stat' call Gnus does comes too quick
after writing the file, maybe the writing takes the path through the
local AFS cache stuff and the stat goes directly to the server so the
stat call will see the old modtime that is updated seconds later?
Maybe you could compare the modtime returned in `nnml-save-marks' with
the value you get if you do this a few seconds later.

But I don't know anything about AFS, maybe someone else knows more.

Re: Getting new mail really slow with nnml marks

[David Z Maze]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]

Simon Josefsson <jas@extundo.com> writes:
SJ> Could you evaluate (nnml-marks-changed-p "nnml+foo:bar") to see if it
SJ> always return t?

Yes, it does.  My theory is that this happens because
nnml-marks-changed-p tests whether the value gnus-gethash returns is
eq to the modtime returned by file-attributes; since they're
presumably not going to be the same object (I'd expect file-attributes
to return a new list every time), you want to test equality with
equal, rather than eq.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: eq->equal patch to nnml.el --]
[-- Type: text/x-patch, Size: 739 bytes --]

Index: nnml.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnml.el,v
retrieving revision 6.29
diff -u -r6.29 nnml.el
--- nnml.el	2001/10/17 15:20:30	6.29
+++ nnml.el	2001/10/17 21:03:14
@@ -979,8 +979,8 @@
 				(nnmail-group-pathname group nnml-directory))))
     (if (null (gnus-gethash file nnml-marks-modtime))
 	t ;; never looked at marks file, assume it has changed
-      (not (eq (gnus-gethash file nnml-marks-modtime)
-	       (nth 5 (file-attributes file)))))))
+      (not (equal (gnus-gethash file nnml-marks-modtime)
+		  (nth 5 (file-attributes file)))))))
 
 (defun nnml-save-marks (group server)
   (let ((file-name-coding-system nnmail-pathname-coding-system)

[-- Attachment #3: Type: text/plain, Size: 354 bytes --]


If I do this, then I get the whole updating-marks thing run at
startup, but not at all afterwards when I run 'g' (even if there is
new mail or I touch a .marks file).  Is this expected?

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Getting new mail really slow with nnml marks

[Simon Josefsson]

David Z Maze <dmaze@MIT.EDU> writes:

> Yes, it does.  My theory is that this happens because
> nnml-marks-changed-p tests whether the value gnus-gethash returns is
> eq to the modtime returned by file-attributes; since they're
> presumably not going to be the same object (I'd expect file-attributes
> to return a new list every time), you want to test equality with
> equal, rather than eq.

Aha!  Right.

> If I do this, then I get the whole updating-marks thing run at
> startup, but not at all afterwards when I run 'g' (even if there is
> new mail or I touch a .marks file).  Is this expected?

If you touch a .marks file, `nnml-marks-changed-p' must return
non-nil!  New mail shouldn't matter, but iff the .marks file is
modified n-m-c-p return non-nil.

But the change look right, can you find out why n-m-c-p doesn't return
non-nil when you touch the .marks file?

Re: Getting new mail really slow with nnml marks

[David Z Maze]

Simon Josefsson <jas@extundo.com> writes:
SJ> If you touch a .marks file, `nnml-marks-changed-p' must return
SJ> non-nil!  New mail shouldn't matter, but iff the .marks file is
SJ> modified n-m-c-p return non-nil.

*checks*  Yup, the patched version works correctly on this front.

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Getting new mail really slow with nnml marks

[Simon Josefsson]

David Z Maze <dmaze@MIT.EDU> writes:

> Simon Josefsson <jas@extundo.com> writes:
> SJ> If you touch a .marks file, `nnml-marks-changed-p' must return
> SJ> non-nil!  New mail shouldn't matter, but iff the .marks file is
> SJ> modified n-m-c-p return non-nil.
> 
> *checks*  Yup, the patched version works correctly on this front.

I've applied your patch, thank you.