nnmaildir's article counts are better now

nnmaildir's article counts are better now

[Paul Jarc]

I think I finally found the problem with nnmaildir's article counts in
the *Group* buffer.  So now it should be no worse than the other mail
backends.  (You'll still have overestimates for groups with holes in
the article ranges.)


paul

Re: nnmaildir's article counts are better now

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> I think I finally found the problem with nnmaildir's article counts
> in the *Group* buffer.  So now it should be no worse than the other
> mail backends.  (You'll still have overestimates for groups with
> holes in the article ranges.)

Should I have to do anything special to get the counts to be correct,
other than updating the cvs tree?  They still seem to be broken for
me.  (i.e. the unread count on many groups is higher than the total
count, or the unread count is >0 when there are no unread messages in
the group)

-- 
Josh Huber

Re: nnmaildir's article counts are better now

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> Should I have to do anything special to get the counts to be correct,
> other than updating the cvs tree?

No.

> (i.e. the unread count on many groups is higher than the total
> count,

That remains a mystery, I guess.  Does M-g in the *Group* buffer fix
it?  If not, what does nnmaildir-request-group put in the " *nntpd*"
buffer?

> or the unread count is >0 when there are no unread messages in
> the group)

This has improved somewhat for me.  When I first start Gnus, or after
g in the *Group* buffer, the counts are high (presumably due to
holes), but after some combination of M-g and entering the group, the
counts are right, and stay right at least until new mail arrives.


paul

Re: nnmaildir's article counts are better now

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> That remains a mystery, I guess.  Does M-g in the *Group* buffer fix
> it?  If not, what does nnmaildir-request-group put in the " *nntpd*"
> buffer?

No M-g does not fix the problem.

As an example, mail.lists.info-cyrus claims (in my *Group* buffer) to
have 1244 new messages, out of a possible 126 messages:

   1244/  126[0]: nnmaildir:mail.lists.info-cyrus

The line in " *nntpd*" after nnmaildir-request-group is:

211 126 1197 1322 mail.lists.info-cyrus

Does that help?  I'm not familiar with the format of the server
buffer.

> This has improved somewhat for me.  When I first start Gnus, or
> after g in the *Group* buffer, the counts are high (presumably due
> to holes), but after some combination of M-g and entering the group,
> the counts are right, and stay right at least until new mail
> arrives.

M-g does not help me at all.  For me, after entering a group and
exiting, the count is correct. (but, like you, a new mail causes the
count to be invalid)  With nnml I always had the problem where the
total count was high (because of holes in the article numbers), but I
never had the new message count be wrong...how is the new article
count calculated?

-- 
Josh Huber

Re: nnmaildir's article counts are better now

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> As an example, mail.lists.info-cyrus claims (in my *Group* buffer) to
> have 1244 new messages, out of a possible 126 messages:
>
>    1244/  126[0]: nnmaildir:mail.lists.info-cyrus

What are you using in gnus-group-line-format to produce that 1244?

> 211 126 1197 1322 mail.lists.info-cyrus
>
> Does that help?  I'm not familiar with the format of the server
> buffer.

It's "211 count min max groupname".  So this looks reaonable.  Try
debugging nnmaildir-request-update-info.  It updates the group info
(marks, in particular) at the end.  You can trigger it with M-g.  What
does the group info look like before and after the update?  Try when
there is new mail as well as when there isn't.

> M-g does not help me at all.  For me, after entering a group and
> exiting, the count is correct.

I think it's the same for me.  After entering and exiting, the correct
count is preserved even after M-g, right?


paul

Re: nnmaildir's article counts are better now

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> Josh Huber <huber@alum.wpi.edu> wrote:
>> As an example, mail.lists.info-cyrus claims (in my *Group* buffer) to
>> have 1244 new messages, out of a possible 126 messages:
>>
>>    1244/  126[0]: nnmaildir:mail.lists.info-cyrus
>
> What are you using in gnus-group-line-format to produce that 1244?

gnus-group-line-format is set to "%M%S%5y/%5t[%T]: %(%g%)\n"

%y = number of unread, unticked articles.
%t = estimated total
%T = number of ticked articles.

Does this seem reasonable to you?

> It's "211 count min max groupname".  So this looks reaonable.  Try
> debugging nnmaildir-request-update-info.  It updates the group info
> (marks, in particular) at the end.  You can trigger it with M-g.
> What does the group info look like before and after the update?  Try
> when there is new mail as well as when there isn't.

This group correctly displays the number of new messages (only because
I've entered/exited the group):

("nnmaildir:mail.lists.openafs-devel" 3
 ((1 . 305))
 nil
 "nnmaildir:"
 ((to-list . "openafs-devel@openafs.org")))


mail.lists.gnupg has the following *Group* buffer line:

    978/  197[1]: nnmaildir:mail.lists.gnupg 

The contents of "info" at the end of nnmaildir-request-update-info is:

("nnmaildir:mail.lists.gnupg" 3
 ((979 . 1054)
  (1055 . 1175))
 ((reply 1048)
  (tick 1164))
 "nnmaildir:"
 ((to-list . "gnupg-users@gnupg.org")))

After entering/exiting the group, it correctly displays that there are
in fact NO new messages:

*     0/  197[1]: nnmaildir:mail.lists.gnupg 

...at the end of n-r-u-i, the info is:

("nnmaildir:mail.lists.gnupg" 3
 ((1 . 1175))
 ((reply 1048)
  (tick 1164))
 "nnmaildir:"
 ((to-list . "gnupg-users@gnupg.org")))


Finally, for a group which actually has new mail, but is incorrectly
reporting a high new count:

   147/  744[10]: nnmaildir:mail.inbox.main

Has info (wow, this is pretty big...):

("nnmaildir:mail.inbox.main" 1
 ((1 . 2)
  (3 . 400)
  (407 . 408)
  (412 . 413)
  417
  (423 . 424)
  (426 . 432)
  (446 . 447)
  449
  (453 . 459)
  (461 . 462)
  467
  (471 . 472)
  (479 . 485)
  (487 . 497)
  (500 . 502)
  (504 . 509)
  (518 . 521)
  (523 . 524)
  (527 . 540)
  (542 . 546)
  549
  (551 . 552)
  (554 . 559)
  562
  (566 . 567)
  (571 . 573)
  578
  (582 . 586)
  591
  (594 . 598)
  (600 . 602)
  (604 . 611)
  615
  (617 . 618)
  (630 . 633)
  (635 . 637)
  (639 . 640)
  (647 . 649)
  (651 . 654)
  (656 . 661)
  (663 . 671)
  (673 . 679)
  (681 . 688)
  (690 . 691)
  697
  (703 . 704)
  708
  (711 . 713)
  715
  (717 . 718)
  (720 . 722)
  724
  (727 . 731)
  (734 . 740)
  743)
 ((reply 4
         21
         41
         51
         (53 . 54)
         57
         59
         61
         63
         (68 . 73)
         (75 . 78)
         80
         (91 . 97)
         102
         (104 . 107)
         (109 . 113)
         (116 . 117)
         (127 . 129)
         (133 . 135)
         (142 . 146)
         148
         (151 . 153)
         155
         157
         159
         (162 . 166)
         (169 . 170)
         173
         179
         187
         191
         199
         201
         203
         205
         (210 . 211)
         213
         217
         (220 . 222)
         (224 . 226)
         228
         (233 . 235)
         237
         239
         242
         245
         (248 . 249)
         (251 . 252)
         255
         257
         259
         (262 . 264)
         (266 . 268)
         (270 . 271)
         274
         (276 . 280)
         (282 . 284)
         (288 . 291)
         293
         295
         298
         302
         310
         312
         317
         320
         (333 . 335)
         337
         (346 . 348)
         (351 . 354)
         (358 . 360)
         370
         (373 . 377)
         (381 . 384)
         (395 . 396)
         400
         (407 . 408)
         417
         428
         430
         449
         (453 . 454)
         (456 . 457)
         479
         485
         (489 . 490)
         509
         518
         520
         524
         527
         529
         537
         540
         (544 . 546)
         549
         551
         554
         557
         562
         567
         (571 . 572)
         582
         (585 . 586)
         594
         (596 . 598)
         602
         604
         (606 . 609)
         617
         632
         (635 . 637)
         (652 . 654)
         (657 . 660)
         (663 . 665)
         (670 . 671)
         676
         678
         682
         697
         703
         (712 . 713)
         715
         (717 . 718)
         (720 . 722)
         734
         742)
  (tick 16 269 457 542 615 711 (729 . 731) 743)
  (forward 118 130 201 (243 . 244) (274 . 275) 303 357 371 488 (491 . 494)))
 "nnmaildir:")

After entering/exiting the group, it correctly shows 7 unread
messages:

      7/  744[10]: nnmaildir:mail.inbox.main 

And (get ready for it ;), the info looks like this now:

("nnmaildir:mail.inbox.main" 1
 ((1 . 661)
  (663 . 671)
  (673 . 722)
  (724 . 732)
  (734 . 740)
  743)
 ((reply 4
         21
         41
         51
         (53 . 54)
         57
         59
         61
         63
         (68 . 73)
         (75 . 78)
         80
         (91 . 97)
         102
         (104 . 107)
         (109 . 113)
         (116 . 117)
         (127 . 129)
         (133 . 135)
         (142 . 146)
         148
         (151 . 153)
         155
         157
         159
         (162 . 166)
         (169 . 170)
         173
         179
         187
         191
         199
         201
         203
         205
         (210 . 211)
         213
         217
         (220 . 222)
         (224 . 226)
         228
         (233 . 235)
         237
         239
         242
         245
         (248 . 249)
         (251 . 252)
         255
         257
         259
         (262 . 264)
         (266 . 268)
         (270 . 271)
         274
         (276 . 280)
         (282 . 284)
         (288 . 291)
         293
         295
         298
         302
         310
         312
         317
         320
         (333 . 335)
         337
         (346 . 348)
         (351 . 354)
         (358 . 360)
         370
         (373 . 377)
         (381 . 384)
         (395 . 396)
         400
         (407 . 408)
         417
         428
         430
         449
         (453 . 454)
         (456 . 457)
         479
         485
         (489 . 490)
         509
         518
         520
         524
         527
         529
         537
         540
         (544 . 546)
         549
         551
         554
         557
         562
         567
         (571 . 572)
         582
         (585 . 586)
         594
         (596 . 598)
         602
         604
         (606 . 609)
         617
         632
         (635 . 637)
         (652 . 654)
         (657 . 660)
         (663 . 665)
         (670 . 671)
         676
         678
         682
         697
         703
         (712 . 713)
         715
         (717 . 718)
         (720 . 722)
         734
         742)
  (tick 16 269 457 542 615 711 (729 . 731) 743)
  (forward 118 130 201 (243 . 244) (274 . 275) 303 357 371 488 (491 . 494)))
 "nnmaildir:")

Hopefully, something in the above information helps :)

> I think it's the same for me.  After entering and exiting, the
> correct count is preserved even after M-g, right?

Yes, that's correct.

-- 
Josh Huber

gnus marks nonexistent articles (was: nnmaildir's article counts are better now)

[Paul Jarc]

I think I've figured it out now.  When Gnus finds that an article does
not exist, it marks it as read.  But nnmaildir doesn't store marks for
nonexistent articles.  (Does nnimap?  nnml?)
nnmaildir-request-update-info returns only the marks it knows about;
it removes marks supplied by Gnus that weren't stored by nnmaildir.
So after it's called, all the nonexistent articles seem to exist as
unread, as far as Gnus is concerned.  Even if nnmaildir is the only
backend currently affected by this problem, I think it makes sense to
fix this in Gnus:  gnus-request-update-info could artificially add
(1 . (1- min)) to the read range after getting marks from the backend.
Thoughts?  Objections?  Other ideas?


paul

Re: gnus marks nonexistent articles

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> I think I've figured it out now.  When Gnus finds that an article does
> not exist, it marks it as read.  But nnmaildir doesn't store marks for
> nonexistent articles.  (Does nnimap?  nnml?)

Nnimap tries hard to not reset Gnus' marks but only update them with
correct information from the server.  So if Gnus thinks a non-existing
article has some set of marks, nnimap never changes it.  This is what
is slowing down group entry, btw..  relevant code from
nnimap-request-update-info-internal:

	(when (nnimap-mark-permanent-p 'read)
	  (let (seen unseen)
	    ;; read info could contain articles marked unread by other
	    ;; imap clients!  we correct this
	    (setq seen (gnus-uncompress-range (gnus-info-read info))
		  unseen (imap-search "UNSEEN UNDELETED")
		  seen (gnus-set-difference seen unseen)
		  ;; seen might lack articles marked as read by other
		  ;; imap clients! we correct this
		  seen (append seen (imap-search "SEEN"))
		  ;; remove dupes
		  seen (sort seen '<)
		  seen (gnus-compress-sequence seen t)
		  ;; we can't return '(1) since this isn't a "list of ranges",
		  ;; and we can't return '((1)) since g-list-of-unread-articles
		  ;; is buggy so we return '((1 . 1)).
		  seen (if (and (integerp (car seen))
				(null (cdr seen)))
			   (list (cons (car seen) (car seen)))
			 seen))
	    (gnus-info-set-read info seen)))

I guess similar code could be used in nnmaildir too.

> nnmaildir-request-update-info returns only the marks it knows about;
> it removes marks supplied by Gnus that weren't stored by nnmaildir.
> So after it's called, all the nonexistent articles seem to exist as
> unread, as far as Gnus is concerned.  Even if nnmaildir is the only
> backend currently affected by this problem, I think it makes sense to
> fix this in Gnus:  gnus-request-update-info could artificially add
> (1 . (1- min)) to the read range after getting marks from the backend.
> Thoughts?  Objections?  Other ideas?

If it is that simple to fix it, I'm for it.  OTOH it might be a good
property that backends should never update marks on
server-non-existing articles; it would allow Gnus to have local
articles using those article numbers.

Re: gnus marks nonexistent articles

[Paul Jarc]

Simon Josefsson <jas@extundo.com> wrote:
> Nnimap tries hard to not reset Gnus' marks but only update them with
> correct information from the server.  So if Gnus thinks a non-existing
> article has some set of marks, nnimap never changes it.

Ok.  nnmaildir is pushier.

> 		  ;; we can't return '(1) since this isn't a "list of ranges",
> 		  ;; and we can't return '((1)) since g-list-of-unread-articles
> 		  ;; is buggy so we return '((1 . 1)).
> 		  seen (if (and (integerp (car seen))
> 				(null (cdr seen)))
> 			   (list (cons (car seen) (car seen)))
> 			 seen))

Are you sure '(1) wouldn't work?  A list of ranges can look like
'(1 (3 . 4)).  nnmaildir doesn't defend against this, and I don't
think I've seen any problems because of it.

> If it is that simple to fix it, I'm for it.

Will do.

> OTOH it might be a good property that backends should never update
> marks on server-non-existing articles; it would allow Gnus to have
> local articles using those article numbers.

I think if that ever happens, Gnus should not trust the backend about
marks for those article numbers anyway, so it shouldn't matter which
marks the backend says those article numbers have.  Gnus can always
restore the missing marks that it knows about after asking the backend
for the marks it knows about.


paul

Re: gnus marks nonexistent articles

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

>> 		  ;; we can't return '(1) since this isn't a "list of ranges",
>> 		  ;; and we can't return '((1)) since g-list-of-unread-articles
>> 		  ;; is buggy so we return '((1 . 1)).
>> 		  seen (if (and (integerp (car seen))
>> 				(null (cdr seen)))
>> 			   (list (cons (car seen) (car seen)))
>> 			 seen))
>
> Are you sure '(1) wouldn't work?  A list of ranges can look like
> '(1 (3 . 4)).  nnmaildir doesn't defend against this, and I don't
> think I've seen any problems because of it.

I don't know, that code was written some time ago, probably Gnus
improved meanwhile.

>> OTOH it might be a good property that backends should never update
>> marks on server-non-existing articles; it would allow Gnus to have
>> local articles using those article numbers.
>
> I think if that ever happens, Gnus should not trust the backend about
> marks for those article numbers anyway, so it shouldn't matter which
> marks the backend says those article numbers have.  Gnus can always
> restore the missing marks that it knows about after asking the backend
> for the marks it knows about.

Hm, yes.  How are articles stored only in the cache handled today?
Gnus remembers marks for them, doesn't it?

Re: gnus marks nonexistent articles

[Paul Jarc]

I wrote:
> Simon Josefsson <jas@extundo.com> wrote:
>> If it is that simple to fix it, I'm for it.
>
> Will do.

Did.  Didn't help.  Argh.


paul