nnmaildir now has persistent article numbers

nnmaildir now has persistent article numbers

[Paul Jarc]

And it's (somewhat) concurrency-friendly, too: you can run two Gnusae
running nnmaildir-request-scan on the same maildir at the same time,
and they won't assign the same article number to two different
messages.  In the spirit of maildir, this is done without needing file
locks.  (However, it could happen that a single message gets two
numbers assigned to it.  Only one number will be stored in the
maildir, but the two Gnusae will disagree for as long as they re both
running.  So don't do that after all.)

Article numbers are stored in .nnmaildir/nov/*.  So caching and
agentizing should work now.  In the past, you could remove the
.nnmaildir/nov/... file to force regeneration of NOV data (although it
should always happen automatically when needed).  Beware that this
will now also cause the message to get a new article number - unless
you do it while Gnus is running and do M-g on the group afterwards, in
which case the old number will be remembered and stored back in the
new NOV file.


paul

Re: nnmaildir now has persistent article numbers

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> Article numbers are stored in .nnmaildir/nov/*.  So caching and
> agentizing should work now.  In the past, you could remove the
> .nnmaildir/nov/... file to force regeneration of NOV data (although
> it should always happen automatically when needed).  Beware that
> this will now also cause the message to get a new article number -
> unless you do it while Gnus is running and do M-g on the group
> afterwards, in which case the old number will be remembered and
> stored back in the new NOV file.

Cool --

After updating, I couldn't start Gnus, perhaps this is the right fix:

Index: lisp/nnmaildir.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnmaildir.el,v
retrieving revision 6.18
diff -u -r6.18 nnmaildir.el
--- lisp/nnmaildir.el	2002/09/23 19:07:57	6.18
+++ lisp/nnmaildir.el	2002/09/23 20:07:21
@@ -388,7 +388,7 @@
 		nov-end (mapconcat
 			  (lambda (extra)
 			    (setq field (symbol-name (car extra))
-				  val (cdr field))
+				  val (cdr extra))
 			    (nnmaildir--tab-to-space field)
 			    (nnmaildir--tab-to-space val)
 			    (concat field ": " val))


-- 
Josh Huber

Re: nnmaildir now has persistent article numbers

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> After updating, I couldn't start Gnus, perhaps this is the right fix:

Yup.  Committed, thanks.


paul

Re: nnmaildir now has persistent article numbers

[Josh Huber]

I'm noticing something odd --

With the current nnmaildir code I'm seeing the following behavior:

1. enter group which has unread messages
2. catchup group (c)
3. refresh groups (g) in group buffer
4. group shows same number of unread messages again
5. hit RET on group, unread message count returns to 0 without
   entering the group.
6. refresh groups (g) again
7. *now* the article count is correct.

Is anyone else seeing this?

-- 
Josh Huber

Re: nnmaildir now has persistent article numbers

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> I'm noticing something odd --
>
> With the current nnmaildir code I'm seeing the following behavior:
>
> 1. enter group which has unread messages
> 2. catchup group (c)
> 3. refresh groups (g) in group buffer
> 4. group shows same number of unread messages again
> 5. hit RET on group, unread message count returns to 0 without
>    entering the group.
> 6. refresh groups (g) again
> 7. *now* the article count is correct.
>
> Is anyone else seeing this?

Yes.  I assume it's the same problem that other backends have always
had - Gnus ignores the backend's article count in favor of the min/max
numbers, and so holes in that range will cause overestimates.  Holes
are more likely now for nnmaildir because article numbers aren't
automatically reassigned for each session.


paul

Re: nnmaildir now has persistent article numbers

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> Josh Huber <huber@alum.wpi.edu> wrote:
>> I'm noticing something odd --
>>
>> With the current nnmaildir code I'm seeing the following behavior:
>>
>> 1. enter group which has unread messages
>> 2. catchup group (c)
>> 3. refresh groups (g) in group buffer
>> 4. group shows same number of unread messages again
>> 5. hit RET on group, unread message count returns to 0 without
>>    entering the group.
>> 6. refresh groups (g) again
>> 7. *now* the article count is correct.
>>
>> Is anyone else seeing this?
>
> Yes.  I assume it's the same problem that other backends have always
> had - Gnus ignores the backend's article count in favor of the
> min/max numbers, and so holes in that range will cause
> overestimates.  Holes are more likely now for nnmaildir because
> article numbers aren't automatically reassigned for each session.

I don't think this is it...the article count is not the issue here,
it's the new message count (unread, unticked article count: %y in the
gnus-group-line-format).

Basically, after catching up the group in the summary buffer, when I'm
returned to the Group buffer the group shows zero unread messages.
After gnus-group-get-new-news, it shows some other (sometimes much
higher, like 100) number of unread messages when in fact there are
none!

-- 
Josh Huber

Re: nnmaildir now has persistent article numbers

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> I don't think this is it...the article count is not the issue here,
> it's the new message count (unread, unticked article count: %y in the
> gnus-group-line-format).

I think that count is computed as (total-read).  Total is (max-min+1).


paul

Re: nnmaildir now has persistent article numbers

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> I think that count is computed as (total-read).  Total is
> (max-min+1).

Hmm...but it's odd.  I've never seen this happen with any other
backend...

Isn't it odd that count would be incorrect, then correct after
attempting to enter the group.  It stays correct after that, even when
gnus-group-get-new-news is called!

Seems a little odd to me, especially since these are mailing list
groups which I have not moved/deleted any messages to/from.

-- 
Josh Huber

Re: nnmaildir now has persistent article numbers

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> prj@po.cwru.edu (Paul Jarc) writes:
>> I think that count is computed as (total-read).  Total is
>> (max-min+1).
>
> Hmm...but it's odd.  I've never seen this happen with any other
> backend...

Assuming we're working with the group "ding" on the (nnmaildir "" ...)
server, this will tell you which article numbers exist in the group:
(let* ((group (nnmaildir--prepare "" "ding"))
       (nlist (nnmaildir--grp-nlist group))
       (articles-descending (mapcar 'car nlist))
       (articles-ascending (reverse articles-descending))
       (ranges (gnus-compress-sequence articles-ascending 'always-list)))
  ranges)

If (> (length ranges) 1), then you have holes.  Count how many
articles are missing and see if that matches the number of extra
articles reported in the group buffer.


paul

Re: nnmaildir now has persistent article numbers

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> Assuming we're working with the group "ding" on the (nnmaildir "" ...)
> server, this will tell you which article numbers exist in the group:

> (let* ((group (nnmaildir--prepare "" "ding"))
>        (nlist (nnmaildir--grp-nlist group))
>        (articles-descending (mapcar 'car nlist))
>        (articles-ascending (reverse articles-descending))
>        (ranges (gnus-compress-sequence articles-ascending 'always-list)))
>   ranges)
>
> If (> (length ranges) 1), then you have holes.  Count how many
> articles are missing and see if that matches the number of extra
> articles reported in the group buffer.

I used this to get a list of all nnmaildir group ranges:

(defun show-nnmaildir-ranges ()
  (mapc '(lambda (item)
             (let ((grp (car item)))
               (when (string-match "^nnmaildir:\\(.*\\)" grp)
                 (let* ((group (nnmaildir--prepare "" (match-string 1 grp)))
                        (nlist (nnmaildir--grp-nlist group))
                        (articles-descending (mapcar 'car nlist))
                        (articles-ascending (reverse articles-descending))
                        (ranges (gnus-compress-sequence articles-ascending 'always-list)))
                   (insert (format "%s: %s\n" grp ranges))))))
                 gnus-newsrc-alist))

Here is a (trimmed) list of the output when run on my machine:

nnmaildir:mail.inbox.debian: ((1 . 11))
nnmaildir:mail.inbox.main: ((1 . 400))
nnmaildir:mail.inbox.mclx: ((1 . 2))
nnmaildir:sent-mail: ((1 . 582))
nnmaildir:sent-news: ((1 . 80))
nnmaildir:mail.lists.debian-kerberos: ((1 . 7))
nnmaildir:mail.lists.debian.private: ((41 . 340) (342 . 358))
nnmaildir:mail.lists.vr6: ((11 . 28))
nnmaildir:mail.lists.monadlug: ((1 . 5))
nnmaildir:mail.lists.gnupg: ((19 . 131))
nnmaildir:mail.lists.wlug: ((5 . 41))
nnmaildir:mail.lists.ruby-talk: (1 (109 . 833))
nnmaildir:mail.lists.tmda-users: ((20 . 80))
nnmaildir:mail.lists.tmda-workers: ((20 . 41))
nnmaildir:mail.lists.debian.security: ((1 . 6))
nnmaildir:mail.lists.openafs-devel: ((1 . 24))

Many of these are reporting incorrect new mail when there is none.

For example, when I fire up gnus for the first time, it always tells
me there are 2 new messages in mail.inbox.debian and 3 new messages in
mail.inbox.main, when in fact there are no new messages in either of
these folders...

Is there anything else I can do to further debug this problem?

Oh, and here's an odd one, the unread message count is greater than
the total:

 23/   18[0]: nnmaildir:mail.lists.vr6

-- 
Josh Huber

Re: nnmaildir now has persistent article numbers

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> nnmaildir:mail.lists.debian.private: ((41 . 340) (342 . 358))

I would expect this one to have its new article count too high by 1.
The others ought to be ok.  For all my groups with no holes, I get
correct counts in the *Group* buffer.

> Is there anything else I can do to further debug this problem?

You could debug-on-entry
nnmaildir-{request-group,request-list,retrieve-groups} to see whether
the data it's putting in the " *nntpd*" buffer is accurate.

> Oh, and here's an odd one, the unread message count is greater than
> the total:
>
>  23/   18[0]: nnmaildir:mail.lists.vr6

Weird.
(defun report-nnmaildir-group (pgname)
  (let* ((gname (gnus-group-real-name pgname))
         (server-name (gnus-group-real-prefix pgname))
         (method (gnus-server-to-method server-name))
         (server-address (nth 1 method))
         (group (nnmaildir--prepare server-address gname))
         (server nnmaildir--cur-server)
         (max (nnmaildir--group-maxnum server group))
         (min (nnmaildir--grp-min group))
         (count (nnmaildir--grp-count group)))
    (insert pgname "\n"
            "\tmax: " (number-to-string max) "\n"
            "\tmin: " (number-to-string min) "\n"
            "\tcount: " (number-to-string count) "\n")))
(report-nnmaildir-group "nnmaildir:mail.lists.vr6")

You might also debug-on-entry nnmaildir-update-info and make sure the
marks it passes back look reasonable.


paul

Re: nnmaildir now has persistent article numbers

[Paul Jarc]

I wrote:
> For all my groups with no holes, I get correct counts in the *Group*
> buffer.

Actually, I found one where the count is too high but there are no
holes.  This group has only a single article, number 2.  nnmaildir was
reporting a minimum number of 1 in the " *nntpd*" buffer.  I just
committed a fix for that; maybe it helps for you.  I still have a
wrong initial count in the Group buffer for this group, even though
nnmaildir is now reporting the right data in " *nntpd*" and for
nnmaildir-request-update-info.


paul

Re: nnmaildir now has persistent article numbers

[Josh Huber]

I'll try debugging further later, but here's some more output,
including high/low/count values:

nnmaildir:mail.inbox.debian: ((1 . 11) 14), max: 14, min: 1, count: 12
nnmaildir:mail.inbox.main: ((1 . 400) (407 . 408)), max: 408, min: 1, count: 402
nnmaildir:mail.lists.debian-kerberos: ((1 . 7)), max: 8, min: 1, count: 7
nnmaildir:mail.lists.debian.private: ((102 . 340) (342 . 358) (360 . 361) (364 . 367)), max: 367, min: 102, count: 262
nnmaildir:mail.lists.vr6: ((11 . 29)), max: 29, min: 11, count: 19
nnmaildir:mail.lists.gnupg: ((23 . 152)), max: 152, min: 23, count: 130
nnmaildir:mail.lists.ruby-talk: (1 162 (194 . 954)), max: 954, min: 1, count: 763
nnmaildir:mail.lists.tmda-users: ((31 . 84)), max: 84, min: 31, count: 54
nnmaildir:mail.lists.tmda-workers: ((20 . 43)), max: 43, min: 20, count: 24
nnmaildir:mail.lists.debian.security: ((1 . 6)), max: 6, min: 1, count: 6
nnmaildir:mail.lists.openafs-devel: ((15 . 26)), max: 26, min: 15, count: 12


Right now in my *Group* buffer I see the following two groups which
look odd:

     14/   12[0]: nnmaildir:mail.lists.openafs-devel 
     24/   19[0]: nnmaildir:mail.lists.vr6 


The count listed above for the groups is correct, but how could the
new article count be so messed up?  I guess I'm going to have to dive
into the code :)

These are the only two groups with new counts above the total count,
but there are (many) other groups which have incorrect new article
counts.

Thanks for the help, btw.

-- 
Josh Huber

Re: nnmaildir now has persistent article numbers

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> These are the only two groups with new counts above the total count,
> but there are (many) other groups which have incorrect new article
> counts.

Have you checked out the fix I put in yesterday?


paul

Re: nnmaildir now has persistent article numbers

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> Have you checked out the fix I put in yesterday?

Yes, it doesn't seem to fix the problem for me, unfortunately.

-- 
Josh Huber

Re: nnmaildir now has persistent article numbers

[Kai Großjohann]

Josh Huber <huber@alum.wpi.edu> writes:

> I don't think this is it...the article count is not the issue here,
> it's the new message count (unread, unticked article count: %y in the
> gnus-group-line-format).

The estimate _is_ for the %y number, AFAIK.

kai
-- 
~/.signature is: umop 3p!sdn    (Frank Nobis)