nnml/nnfolder marks faster

nnml/nnfolder marks faster

[Simon Josefsson]

The marks stuff should now be faster as the marks file is only read
when it is necessary.

A little implementation note on how the marks stuff works follows, if
anyone is interested.  Two new functions were added:

*-request-update-info: Reads marks file if it has changed (or if it
  has not been previously read, e.g. on startup of Gnus) and updates
  Gnus' info from the marks file.

*-request-set-mark: Reads marks file, applies the changes, writes
  marks file.  Updates modtime.

(Nothing else was changed.)

Re: nnml/nnfolder marks faster

[Dan Christensen]

I have a few groups that have really large .mrk files, and this is
causing slow performance.  I've noticed this more since the new
.mrk code was introduced, and it is in particular noticeable when
I *exit* a group with a large .mrk file.  But I don't know if the
new .mrk code is causing any problems, or whether it just makes
them more visible.

As an example, my nnfolder:debian.big group has about 1000 articles
in it, and the .mrk file is somehow using about 840 bytes per article,
which seems really excessive:

-rw-------    1 jdc      jdc       4231285 Sep 12 13:56 Gnus/Mail/debian.big
-rw-r--r--    1 jdc      jdc        840761 Sep 12 14:46 Gnus/Mail/debian.big.mrk
-rw-------    1 jdc      jdc        310299 Sep 12 14:42 Gnus/Mail/debian.big.nov

Here are a few excerpts from the .mrk file which look suspicious:

There are screenfuls and screenfuls of "seen" marks that look like:

  (184755 . 184758) (184759 . 184762)

which should be merged into a single long list I would think.

Also, the score caching has stuff like:

  (score (185776 . -1) (185820 . -1) (185962 . -1) (185831 . -1)
  (185991 . -1)

which isn't even stored in numerical order (how is lookup done??).  
I also wonder if this could be stored in a collapsed form using ranges

  (score ((12345 . 15000) . -1)    

etc. (in cases where there are consecutive articles with the same
score.)

Hmm, looking further, it is the score entries that are taking up 99%
of the .mrk file, and most of them don't correspond to articles that
exist in the group!  Shouldn't score cache entries be removed when
the article is deleted from the group (e.g. by the expiry mechanism)?
I smell a bug.

Dan

-- 
Dan Christensen
jdc+news@uwo.ca

Re: nnml/nnfolder marks faster

[Simon Josefsson]

Dan Christensen <jdc+news@uwo.ca> writes:

> I have a few groups that have really large .mrk files, and this is
> causing slow performance.  I've noticed this more since the new
> .mrk code was introduced, and it is in particular noticeable when
> I *exit* a group with a large .mrk file.  But I don't know if the
> new .mrk code is causing any problems, or whether it just makes
> them more visible.

I guess the latter, I think the problem you mention existed earlier
(because the backend marks code hasn't touched that code).

> As an example, my nnfolder:debian.big group has about 1000 articles
> in it, and the .mrk file is somehow using about 840 bytes per article,
> which seems really excessive:

Yes, too much.

> Here are a few excerpts from the .mrk file which look suspicious:
>
> There are screenfuls and screenfuls of "seen" marks that look like:
>
>   (184755 . 184758) (184759 . 184762)
>
> which should be merged into a single long list I would think.

That is probably better yes, but I don't know the seen mark, it seem
to be handled specially somehow.  Lars?

> Also, the score caching has stuff like:
>
>   (score (185776 . -1) (185820 . -1) (185962 . -1) (185831 . -1)
>   (185991 . -1)
>
> which isn't even stored in numerical order (how is lookup done??).  
> I also wonder if this could be stored in a collapsed form using ranges
>
>   (score ((12345 . 15000) . -1)    
>
> etc. (in cases where there are consecutive articles with the same
> score.)
>
> Hmm, looking further, it is the score entries that are taking up 99%
> of the .mrk file, and most of them don't correspond to articles that
> exist in the group!  Shouldn't score cache entries be removed when
> the article is deleted from the group (e.g. by the expiry mechanism)?
> I smell a bug.

I'm not sure score marks should be in the file at all.  Isn't the
scores calculated each time you enter the group?  Maybe the score is
cached in .newsrc/.mark?

Likewise, `downloaded', `cache' and `unsent' should probably never be
propagated to the backend.

Re: nnml/nnfolder marks faster

[Paul Jarc]

Simon Josefsson <jas@extundo.com> wrote:
> Dan Christensen <jdc+news@uwo.ca> writes:
>> There are screenfuls and screenfuls of "seen" marks that look like:
>>
>>   (184755 . 184758) (184759 . 184762)
>>
>> which should be merged into a single long list I would think.
>
> That is probably better yes, but I don't know the seen mark, it seem
> to be handled specially somehow.  Lars?

I think (gnus-compress-sequence (gnus-uncompress-range range)) would
do it.  How is 'seen special?

> I'm not sure score marks should be in the file at all.  Isn't the
> scores calculated each time you enter the group?  Maybe the score is
> cached in .newsrc/.mark?
>
> Likewise, `downloaded', `cache' and `unsent' should probably never be
> propagated to the backend.

This would mean Gnus should remove those marks before passing the info
right?  I don't think backends should have to know which marks they
aren't supposed to know about.


paul

Re: nnml/nnfolder marks faster

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> Simon Josefsson <jas@extundo.com> wrote:
> > Dan Christensen <jdc+news@uwo.ca> writes:
>>> There are screenfuls and screenfuls of "seen" marks that look like:
>>>
>>>   (184755 . 184758) (184759 . 184762)
>>>
>>> which should be merged into a single long list I would think.
>>
>> That is probably better yes, but I don't know the seen mark, it seem
>> to be handled specially somehow.  Lars?
>
> I think (gnus-compress-sequence (gnus-uncompress-range range)) would
> do it. How is 'seen special?

It is a range instead of a list, which most other marks is.  And there
is some special logic to set the variable.  Where should the snippet
you suggest be put?

>> I'm not sure score marks should be in the file at all.  Isn't the
>> scores calculated each time you enter the group?  Maybe the score is
>> cached in .newsrc/.mark?
>>
>> Likewise, `downloaded', `cache' and `unsent' should probably never be
>> propagated to the backend.
>
> This would mean Gnus should remove those marks before passing the info
> right?  I don't think backends should have to know which marks they
> aren't supposed to know about.

Right.

How about this?

2001-09-14  Simon Josefsson  <jas@extundo.com>

	* gnus-sum.el (gnus-article-unpropagatable-p): New function.
	(gnus-update-marks): Use it.
	(gnus-update-marks): Use `gnus-article-mark-to-type' instead of
	hardcoded list.

	* gnus.el (gnus-article-special-mark-lists): Add killed.
	(gnus-article-unpropagated-mark-lists): New constant.

Index: gnus.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus.el,v
retrieving revision 6.54
diff -u -r6.54 gnus.el
--- gnus.el	2001/08/23 18:56:59	6.54
+++ gnus.el	2001/09/14 17:31:21
@@ -1800,8 +1800,19 @@
 
 (defconst gnus-article-special-mark-lists
   '((seen range)
+    (killed range)
     (bookmark tuple)
     (score tuple)))
+
+;; Propagate flags to server, with the following exceptions:
+;; `cache' is a internal gnus flag
+;; `download' is a agent flag
+;; `unsend' are for nndraft groups only
+(defconst gnus-article-unpropagated-mark-lists '(cache download unsend)
+  "Marks that shouldn't be propagated to backends.
+Typical marks are those that make no sense in a standalone backend,
+such as marks that says if a article is stored in the cache or not
+(which doesn't make sense in a standalone backend).")
 
 (defvar gnus-headers-retrieved-by nil)
 (defvar gnus-article-reply nil)
Index: gnus-sum.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus-sum.el,v
retrieving revision 6.106
diff -u -r6.106 gnus-sum.el
--- gnus-sum.el	2001/09/10 09:29:50	6.106
+++ gnus-sum.el	2001/09/14 17:31:23
@@ -4811,6 +4811,10 @@
   (or (cadr (assq mark gnus-article-special-mark-lists))
       'list))
 
+(defun gnus-article-unpropagatable-p (mark)
+  "Return whether MARK should be propagated to backend."
+  (memq mark gnus-article-unpropagated-mark-lists))
+
 (defun gnus-adjust-marked-articles (info)
   "Set all article lists and remove all marks that are no longer valid."
   (let* ((marked-lists (gnus-info-marks info))
@@ -4870,7 +4874,6 @@
   "Enter the various lists of marked articles into the newsgroup info list."
   (let ((types gnus-article-mark-lists)
 	(info (gnus-get-info gnus-newsgroup-name))
-	(uncompressed '(score bookmark killed seen))
 	type list newmarked symbol delta-marks)
     (when info
       ;; Add all marks lists to the list of marks lists.
@@ -4902,27 +4905,20 @@
 		    (gnus-add-to-range list gnus-newsgroup-unseen)
 		  (gnus-compress-sequence gnus-newsgroup-articles))))
 
-	(unless (memq (cdr type) uncompressed)
+	(when (eq (gnus-article-mark-to-type (cdr type)) 'list)
 	  (setq list (gnus-compress-sequence (set symbol (sort list '<)) t)))
 
-	(when (gnus-check-backend-function
-	       'request-set-mark gnus-newsgroup-name)
-	  ;; propagate flags to server, with the following exceptions:
-	  ;; uncompressed:s are not proper flags (they are cons cells)
-	  ;; cache is a internal gnus flag
-	  ;; download are local to one gnus installation (well)
-	  ;; unsend are for nndraft groups only
-	  ;; xxx: generality of this?  this suits nnimap anyway
-	  (unless (memq (cdr type) (append '(cache download unsend)
-					   uncompressed))
-	    (let* ((old (cdr (assq (cdr type) (gnus-info-marks info))))
-		   (del (gnus-remove-from-range (gnus-copy-sequence old) list))
-		   (add (gnus-remove-from-range
-			 (gnus-copy-sequence list) old)))
-	      (when add
-		(push (list add 'add (list (cdr type))) delta-marks))
-	      (when del
-		(push (list del 'del (list (cdr type))) delta-marks)))))
+	(when (and (gnus-check-backend-function
+		    'request-set-mark gnus-newsgroup-name)
+		   (not (gnus-article-unpropagatable-p (cdr type))))
+	  (let* ((old (cdr (assq (cdr type) (gnus-info-marks info))))
+		 (del (gnus-remove-from-range (gnus-copy-sequence old) list))
+		 (add (gnus-remove-from-range
+		       (gnus-copy-sequence list) old)))
+	    (when add
+	      (push (list add 'add (list (cdr type))) delta-marks))
+	    (when del
+	      (push (list del 'del (list (cdr type))) delta-marks))))
 
 	(when list
 	  (push (cons (cdr type) list) newmarked)))

Re: nnml/nnfolder marks faster

[Paul Jarc]

Simon Josefsson <jas@extundo.com> wrote:
> prj@po.cwru.edu (Paul Jarc) writes:
>> I think (gnus-compress-sequence (gnus-uncompress-range range)) would
>> do it. How is 'seen special?
>
> It is a range instead of a list, which most other marks is.

They all look like lists of ranges to me, at least in .newsrc.eld.
Does Gnus still use uncompressed lists because there are backends that
can't handle rages yet?

> Where should the snippet you suggest be put?

In nn*-request-set-mark, you could calculate the new range, apply the
above to it, and store the result.  But it would probably be
significantly faster to just manipulate the list of ranges by looking
for adjacent/overlapping ranges, without recompressing.

> +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend)
> +  "Marks that shouldn't be propagated to backends.
> +Typical marks are those that make no sense in a standalone backend,
> +such as marks that says if a article is stored in the cache or not

Better, I think:
such as a mark that says whether an article is stored in the cache

Are the unpropagatable marks stored in .newsrc.eld?  If they're always
recalculated, it seems they shouldn't be.  Otherwise it seems they
should be, but then they should also be stored in the backend.


paul

Re: nnml/nnfolder marks faster

[Lars Magne Ingebrigtsen]

prj@po.cwru.edu (Paul Jarc) writes:

>>> I think (gnus-compress-sequence (gnus-uncompress-range range)) would
>>> do it. How is 'seen special?
>>
>> It is a range instead of a list, which most other marks is.
>
> They all look like lists of ranges to me, at least in .newsrc.eld.
> Does Gnus still use uncompressed lists because there are backends that
> can't handle rages yet?

`seen' is special because it's the only marks lists that's never
uncompressed.  All the other ones -- `tick', `dormant', etc -- are
uncompressed on group entry and then compressed on group exit.  `seen'
is not, and that's because it's assumed that `seen' will grow bounded
only by the size of the group, while the others will have a more
moderate growth pattern.  `seen' will typically be something like
'(1 . 45787839)' in a suitably active group.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnml/nnfolder marks faster

[Paul Jarc]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
>>>> I think (gnus-compress-sequence (gnus-uncompress-range range)) would
>>>> do it. How is 'seen special?
...
> `seen' is special because it's the only marks lists that's never
> uncompressed.  All the other ones -- `tick', `dormant', etc -- are
> uncompressed on group entry and then compressed on group exit.  `seen'
> is not, and that's because it's assumed that `seen' will grow bounded
> only by the size of the group, while the others will have a more
> moderate growth pattern.

Well, then, there could be a function that recompresses a range by
looking for consecutive entries, rather than by expanding and
unexpanding.  That could be applied to 'seen.  Right?


paul

Re: nnml/nnfolder marks faster

[Lars Magne Ingebrigtsen]

prj@po.cwru.edu (Paul Jarc) writes:

> Well, then, there could be a function that recompresses a range by
> looking for consecutive entries, rather than by expanding and
> unexpanding.  That could be applied to 'seen.  Right?

There are functions to merge ranges without uncompressing them, if
that's what you mean.

But there's no problem in the `seen' handling (except the .marks
stuff), I think.  I was just explaining why they had to be treated
specially. 

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnml/nnfolder marks faster

[Paul Jarc]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> prj@po.cwru.edu (Paul Jarc) writes:
>> Well, then, there could be a function that recompresses a range by
>> looking for consecutive entries, rather than by expanding and
>> unexpanding.  That could be applied to 'seen.  Right?
>
> There are functions to merge ranges without uncompressing them, if
> that's what you mean.

I don't think it is.  I'm thinking of a function which takes a single
range list and returns the smallest range list representing the same
set of numbers, and which doesn't eat as much memory as
(gnus-compress-sequence (gnus-uncompress-range range)).

> But there's no problem in the `seen' handling (except the .marks
> stuff), I think.  I was just explaining why they had to be treated
> specially.

Right, that's the problem this function would help solve.


paul

Re: nnml/nnfolder marks faster

[Lars Magne Ingebrigtsen]

prj@po.cwru.edu (Paul Jarc) writes:

> I don't think it is.  I'm thinking of a function which takes a single
> range list and returns the smallest range list representing the same
> set of numbers, and which doesn't eat as much memory as
> (gnus-compress-sequence (gnus-uncompress-range range)).

That's true.  But why would you need something like that?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnml/nnfolder marks faster

[Paul Jarc]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> prj@po.cwru.edu (Paul Jarc) writes:
>> I'm thinking of a function which takes a single range list and
>> returns the smallest range list representing the same set of
>> numbers, and which doesn't eat as much memory as
>> (gnus-compress-sequence (gnus-uncompress-range range)).
>
> That's true.  But why would you need something like that?

To make the 'seen range list smaller - in .newsrc.eld, in nnml and
nnfolder's mark storage, and in memory.  There's wasted space there.


paul

Re: nnml/nnfolder marks faster

[Lars Magne Ingebrigtsen]

prj@po.cwru.edu (Paul Jarc) writes:

> To make the 'seen range list smaller - in .newsrc.eld, in nnml and
> nnfolder's mark storage, and in memory.  There's wasted space there.

When are these non-optimal ranges created?  If they're not optimal,
the range functions are buggy.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnml/nnfolder marks faster

[Paul Jarc]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> prj@po.cwru.edu (Paul Jarc) writes:
>> To make the 'seen range list smaller - in .newsrc.eld, in nnml and
>> nnfolder's mark storage, and in memory.  There's wasted space there.
>
> When are these non-optimal ranges created?  If they're not optimal,
> the range functions are buggy.

I don't know, but that's the problem that was reported in this thread
3 months ago, as well as more recently in the thread "\"seen\" marks
create adjacent lists".  Maybe it's fixed now.  The recompression
might still be useful to fix old range lists left over from when the
range functions were buggy.  Or not.


paul

Re: nnml/nnfolder marks faster

[Dan Christensen]

Simon Josefsson <jas@extundo.com> writes:

> Dan Christensen <jdc+news@uwo.ca> writes:
>
>> Hmm, looking further, it is the score entries that are taking up 99%
>> of the .mrk file

> I'm not sure score marks should be in the file at all.  Isn't the
> scores calculated each time you enter the group?  Maybe the score is
> cached in .newsrc/.mark?

I have gnus-save-score set to t, in the hope that this speeds up
summary generation.  But maybe it doesn't.  In any case, this
feature should only store scores for articles that exist:

>> and most of them don't correspond to articles that
>> exist in the group!  Shouldn't score cache entries be removed when
>> the article is deleted from the group (e.g. by the expiry mechanism)?
>> I smell a bug.

-- 
Dan Christensen
jdc+news@uwo.ca

Re: nnml/nnfolder marks faster

[Dan Christensen]

Dan Christensen <jdc+news@uwo.ca> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Dan Christensen <jdc+news@uwo.ca> writes:
>>
>>> Hmm, looking further, it is the score entries that are taking up 99%
>>> of the .mrk file
>
>> I'm not sure score marks should be in the file at all.  Isn't the
>> scores calculated each time you enter the group?  Maybe the score is
>> cached in .newsrc/.mark?
>
> I have gnus-save-score set to t, in the hope that this speeds up
> summary generation.

I now have gnus-save-score set to nil.  After entering and exiting
a group, the massive (score ...) entries are removed from that
group's list in .newsrc.eld, but they remain in that group's .mrk
file.  Is this a bug in the .mrk code?

By the way, is the eventual goal of the .mrk files to remove this
information from .newsrc.eld?  I can see benefits to this, like
faster performance because only the marks relevant to the groups
you access get read from disk, etc.  Or will the .mrk information
always be duplicated?

Dan

-- 
Dan Christensen
jdc+news@uwo.ca

Re: nnml/nnfolder marks faster

[Simon Josefsson]

Dan Christensen <jdc+news@uwo.ca> writes:

> I now have gnus-save-score set to nil.  After entering and exiting
> a group, the massive (score ...) entries are removed from that
> group's list in .newsrc.eld, but they remain in that group's .mrk
> file.  Is this a bug in the .mrk code?

Possibly.  Did you try my patch in
<iluheu5adba.fsf@barbar.josefsson.org>?  I think it keeps `score' out
of .mrk.

> By the way, is the eventual goal of the .mrk files to remove this
> information from .newsrc.eld?  I can see benefits to this, like
> faster performance because only the marks relevant to the groups
> you access get read from disk, etc.

As a long term goal, yes.  Using Gnus with nnml or IMAP without
.newsrc.eld would be nice, but backwards compatibility is probably
nicer so I don't expect it will be possibly soon.

Re: nnml/nnfolder marks faster

[Dan Christensen]

Simon Josefsson <jas@extundo.com> writes:

> Dan Christensen <jdc+news@uwo.ca> writes:
>
>> I now have gnus-save-score set to nil.  After entering and exiting
>> a group, the massive (score ...) entries are removed from that
>> group's list in .newsrc.eld, but they remain in that group's .mrk
>> file.  Is this a bug in the .mrk code?
>
> Possibly.  Did you try my patch in
> <iluheu5adba.fsf@barbar.josefsson.org>?  I think it keeps `score' out
> of .mrk.

It doesn't work.  Score entries remain in the .mrk file.  Even worse,
they are changed to be incorrect.  For example, the score pair
(579 . 2000) was changed to (579 . 1092)  (and lots of other 2000's
were changed to 1092's as well).

If I delete the .mrk file and set gnus-save-score to nil, then no
score entries reappear.  But if I set gnus-save-score to t, they
appear again in the newly created .mrk file.

While I'm writing, let me mention again the two other bugs:

- seen entries aren't as compressed as they could be
- score entries aren't purged when the article they refer to is
  deleted/expired

-- 
Dan Christensen
jdc+news@uwo.ca

Re: nnml/nnfolder marks faster

[Simon Josefsson]

On Fri, 21 Sep 2001, Dan Christensen wrote:

> Simon Josefsson <jas@extundo.com> writes:
>
> > Dan Christensen <jdc+news@uwo.ca> writes:
> >
> >> I now have gnus-save-score set to nil.  After entering and exiting
> >> a group, the massive (score ...) entries are removed from that
> >> group's list in .newsrc.eld, but they remain in that group's .mrk
> >> file.  Is this a bug in the .mrk code?
> >
> > Possibly.  Did you try my patch in
> > <iluheu5adba.fsf@barbar.josefsson.org>?  I think it keeps `score' out
> > of .mrk.
>
> It doesn't work.  Score entries remain in the .mrk file.

Oops, I just found a problem with the patch.  Please change

+(defconst gnus-article-unpropagated-mark-lists '(cache download unsend)

into

+(defconst gnus-article-unpropagated-mark-lists '(cache download unsend score)

and hopefully score doesn't end up in .mrk.

> Even worse, they are changed to be incorrect.  For example, the score pair
> (579 . 2000) was changed to (579 . 1092)  (and lots of other 2000's
> were changed to 1092's as well).

I suspect Gnus tries to sort the score marks as ranges somehow, which is a
bad idea.

> While I'm writing, let me mention again the two other bugs:
>
> - seen entries aren't as compressed as they could be

I'll try to understand how seen are supposed to work and fix it.

> - score entries aren't purged when the article they refer to is
>   deleted/expired

Once we get the score mark out of the .mrk file, this and your main
problem should vanish, I hope.

Thanks for the help.

Re: nnml/nnfolder marks faster

[Dan Christensen]

Simon Josefsson <jas@extundo.com> writes:

> On Fri, 21 Sep 2001, Dan Christensen wrote:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>> > Possibly.  Did you try my patch in
>> > <iluheu5adba.fsf@barbar.josefsson.org>?  I think it keeps `score' out
>> > of .mrk.
>>
>> It doesn't work.  Score entries remain in the .mrk file.
>
> Oops, I just found a problem with the patch.  Please change
>
> +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend)
>
> into
>
> +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend score)

Still doesn't work.  western.mrk included below.  (I made sure to
change a mark in this group so the .mrk file got rewritten.)  The
.mrk file also contains seen entries.  If it matters, I'm using
nnfolder.

More testing...  Ok, if I delete the western.mrk file, then no 
score entries get put there, even if gnus-save-score is set to t.
On the other hand, if there are already score entries in the
western.mrk file, then they remain there when the western.mrk
file gets rewritten, even with the patch applied.

In every case I've tried, the seen marks are written to the
western.mrk file.

>> Even worse, they are changed to be incorrect.  For example, the score pair
>> (579 . 2000) was changed to (579 . 1092)  (and lots of other 2000's
>> were changed to 1092's as well).
>
> I suspect Gnus tries to sort the score marks as ranges somehow, which is a
> bad idea.

But this only happens in the .mrk file, not in the .newsrc.eld file.
(Included file illustrates this.)

>> - score entries aren't purged when the article they refer to is
>>   deleted/expired
>
> Once we get the score mark out of the .mrk file, this and your main
> problem should vanish, I hope.

I think this is a separate problem.  The score entries aren't
expired from .newsrc.eld either.  So this bug will still need to
be fixed even if score entries don't propagate to the .mrk files.

Dan

western.mrk:

((tick 282 298 307) (score (82 . 84) (81 . 84) (78 . 84) (79 . 84) (80 . 84) (76 . 84) (77 . 84) (68 . 84) (69 . 84) (70 . 84) (65 . 84) (63 . 84) (62 . 84) (64 . 84) (60 . 84) (58 . 84) (59 . 84) (56 . 84) (44 . 84) (48 . 84) (39 . 84) (37 . 84) (43 . 84) (41 . 84) (40 . 84) (32 . 84) (29 . 84) (45 . 84) (49 . 84) (47 . 84) (38 . 84) (36 . 84) (31 . 84)) (read (1 . 312)) (seen (27 . 45) (47 . 106) (109 . 114) (116 . 274) 280) (forward 268 285 287) (reply 33 37 42 50 55 63 68 (77 . 78) 85 (90 . 91) (101 . 102) 105 (110 . 111) (113 . 114) 117 119 122 (124 . 125) (128 . 131) 143 146 156 166 (176 . 177) (179 . 180) 185 203 208 231 233 238 240 263 (284 . 285) (294 . 295)))

western.mrk recreated after deletion:

((read (1 . 312)) (seen (27 . 45) (47 . 106) (109 . 114) (116 . 274) 280 282 298 307) (forward 268 285 287) (reply 33 37 42 50 55 63 68 (77 . 78) 85 (90 . 91) (101 . 102) 105 (110 . 111) (113 . 114) 117 119 122 (124 . 125) (128 . 131) 143 146 156 166 (176 . 177) (179 . 180) 185 203 208 231 233 238 240 263 (284 . 285) (294 . 295)) (tick 282 298 307))

Re: nnml/nnfolder marks faster

[Simon Josefsson]

Dan Christensen <jdc+news@uwo.ca> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> On Fri, 21 Sep 2001, Dan Christensen wrote:
>>
>>> Simon Josefsson <jas@extundo.com> writes:
>>>
>>> > Possibly.  Did you try my patch in
>>> > <iluheu5adba.fsf@barbar.josefsson.org>?  I think it keeps `score' out
>>> > of .mrk.
>>>
>>> It doesn't work.  Score entries remain in the .mrk file.
>>
>> Oops, I just found a problem with the patch.  Please change
>>
>> +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend)
>>
>> into
>>
>> +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend score)
>
> Still doesn't work.  western.mrk included below.  (I made sure to
> change a mark in this group so the .mrk file got rewritten.)  The
> .mrk file also contains seen entries.  If it matters, I'm using
> nnfolder.
>
> More testing...  Ok, if I delete the western.mrk file, then no 
> score entries get put there, even if gnus-save-score is set to t.

Ok, good.

> On the other hand, if there are already score entries in the
> western.mrk file, then they remain there when the western.mrk
> file gets rewritten, even with the patch applied.

The patches below fixes this by making sure existing marks are removed
as well.

> In every case I've tried, the seen marks are written to the
> western.mrk file.

Yup, the patch didn't care about seen.

One could probably argue that seen marks should be specific to each
Gnus installation -- you could use the seen mark to see which articles
YOU have read in a shared mailbox in which some articles may have been
marked as read by other people -- so I added `seen' to
`gnus-article-unpropagated-mark-lists' as well.

>>> Even worse, they are changed to be incorrect.  For example, the score pair
>>> (579 . 2000) was changed to (579 . 1092)  (and lots of other 2000's
>>> were changed to 1092's as well).
>>
>> I suspect Gnus tries to sort the score marks as ranges somehow, which is a
>> bad idea.
>
> But this only happens in the .mrk file, not in the .newsrc.eld file.
> (Included file illustrates this.)

I think I found the problem, nnml/nnfolder calls some gnus-range
functions on those ranges, and since they aren't proper ranges they
are messed up.  Gnus takes care to not do that for score marks.  This
is fixed by not storing score in .mrk.

>>> - score entries aren't purged when the article they refer to is
>>>   deleted/expired
>>
>> Once we get the score mark out of the .mrk file, this and your main
>> problem should vanish, I hope.
>
> I think this is a separate problem.  The score entries aren't
> expired from .newsrc.eld either.  So this bug will still need to
> be fixed even if score entries don't propagate to the .mrk files.

Yup.  Let's take care of it once .mrk works.

Ok, here is the complete set of patches that fixes things, include
Paul's doc fix.  Please test it and I'll commit it.

Are there any outstanding issues with the .mrk/.marks after this?  How
is the performance for you?  Profiling results of
`nnfolder-request-update-info' and `nnfolder-request-set-mark' for
largish groups would be interesting.

2001-09-22  Simon Josefsson  <jas@extundo.com>

	* nnml.el (nnml-open-marks): Remove unpropagatable marks.

	* nnfolder.el (nnfolder-open-marks): Ditto.

	* gnus-sum.el (gnus-article-unpropagatable-p): New function.
	(gnus-update-marks): Use it.
	(gnus-update-marks): Use `gnus-article-mark-to-type' instead of
	hardcoded list.

	* gnus.el (gnus-article-special-mark-lists): Add killed.
	(gnus-article-unpropagated-mark-lists): New constant.

Index: gnus.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus.el,v
retrieving revision 6.54
diff -u -r6.54 gnus.el
--- gnus.el	2001/08/23 18:56:59	6.54
+++ gnus.el	2001/09/22 16:49:47
@@ -1800,8 +1800,22 @@
 
 (defconst gnus-article-special-mark-lists
   '((seen range)
+    (killed range)
     (bookmark tuple)
     (score tuple)))
+
+;; Propagate flags to server, with the following exceptions:
+;; `seen' is private to each gnus installation
+;; `cache' is a internal gnus flag for each gnus installation
+;; `download' is a agent flag private to each gnus installation
+;; `unsend' are for nndraft groups only
+;; `score' is not a proper mark
+(defconst gnus-article-unpropagated-mark-lists 
+  '(seen cache download unsend score)
+  "Marks that shouldn't be propagated to backends.
+Typical marks are those that make no sense in a standalone backend,
+such as a mark that says whether an article is stored in the cache
+(which doesn't make sense in a standalone backend).")
 
 (defvar gnus-headers-retrieved-by nil)
 (defvar gnus-article-reply nil)
Index: gnus-sum.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus-sum.el,v
retrieving revision 6.106
diff -u -r6.106 gnus-sum.el
--- gnus-sum.el	2001/09/10 09:29:50	6.106
+++ gnus-sum.el	2001/09/22 16:49:48
@@ -4811,6 +4811,10 @@
   (or (cadr (assq mark gnus-article-special-mark-lists))
       'list))
 
+(defun gnus-article-unpropagatable-p (mark)
+  "Return whether MARK should be propagated to backend."
+  (memq mark gnus-article-unpropagated-mark-lists))
+
 (defun gnus-adjust-marked-articles (info)
   "Set all article lists and remove all marks that are no longer valid."
   (let* ((marked-lists (gnus-info-marks info))
@@ -4870,7 +4874,6 @@
   "Enter the various lists of marked articles into the newsgroup info list."
   (let ((types gnus-article-mark-lists)
 	(info (gnus-get-info gnus-newsgroup-name))
-	(uncompressed '(score bookmark killed seen))
 	type list newmarked symbol delta-marks)
     (when info
       ;; Add all marks lists to the list of marks lists.
@@ -4902,27 +4905,20 @@
 		    (gnus-add-to-range list gnus-newsgroup-unseen)
 		  (gnus-compress-sequence gnus-newsgroup-articles))))
 
-	(unless (memq (cdr type) uncompressed)
+	(when (eq (gnus-article-mark-to-type (cdr type)) 'list)
 	  (setq list (gnus-compress-sequence (set symbol (sort list '<)) t)))
 
-	(when (gnus-check-backend-function
-	       'request-set-mark gnus-newsgroup-name)
-	  ;; propagate flags to server, with the following exceptions:
-	  ;; uncompressed:s are not proper flags (they are cons cells)
-	  ;; cache is a internal gnus flag
-	  ;; download are local to one gnus installation (well)
-	  ;; unsend are for nndraft groups only
-	  ;; xxx: generality of this?  this suits nnimap anyway
-	  (unless (memq (cdr type) (append '(cache download unsend)
-					   uncompressed))
-	    (let* ((old (cdr (assq (cdr type) (gnus-info-marks info))))
-		   (del (gnus-remove-from-range (gnus-copy-sequence old) list))
-		   (add (gnus-remove-from-range
-			 (gnus-copy-sequence list) old)))
-	      (when add
-		(push (list add 'add (list (cdr type))) delta-marks))
-	      (when del
-		(push (list del 'del (list (cdr type))) delta-marks)))))
+	(when (and (gnus-check-backend-function
+		    'request-set-mark gnus-newsgroup-name)
+		   (not (gnus-article-unpropagatable-p (cdr type))))
+	  (let* ((old (cdr (assq (cdr type) (gnus-info-marks info))))
+		 (del (gnus-remove-from-range (gnus-copy-sequence old) list))
+		 (add (gnus-remove-from-range
+		       (gnus-copy-sequence list) old)))
+	    (when add
+	      (push (list add 'add (list (cdr type))) delta-marks))
+	    (when del
+	      (push (list del 'del (list (cdr type))) delta-marks))))
 
 	(when list
 	  (push (cons (cdr type) list) newmarked)))
Index: nnml.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnml.el,v
retrieving revision 6.26
diff -u -r6.26 nnml.el
--- nnml.el	2001/09/15 21:18:36	6.26
+++ nnml.el	2001/09/22 16:49:49
@@ -1000,16 +1000,17 @@
 	       nnml-marks-file-name 
 	       (nnmail-group-pathname group nnml-directory))))
     (if (file-exists-p file)
-	(setq nnml-marks (condition-case err
-			     (with-temp-buffer
-			       (gnus-sethash file
-					     (nth 5 (file-attributes file))
-					     nnml-marks-modtime)
-			       (nnheader-insert-file-contents file)
-			       (read (current-buffer)))
-			   (error (or (gnus-yes-or-no-p
-				       (format "Error reading nnml marks file %s (%s).  Continuing will use marks from .newsrc.eld.  Continue? " file err))
-				      (error "Cannot read nnml marks file %s (%s)" file err)))))
+	(condition-case err
+	    (with-temp-buffer
+	      (gnus-sethash file (nth 5 (file-attributes file))
+			    nnml-marks-modtime)
+	      (nnheader-insert-file-contents file)
+	      (setq nnml-marks (read (current-buffer)))
+	      (dolist (el gnus-article-unpropagated-mark-lists)
+		(setq nnml-marks (gnus-remassoc el nnml-marks))))
+	  (error (or (gnus-yes-or-no-p
+		      (format "Error reading nnml marks file %s (%s).  Continuing will use marks from .newsrc.eld.  Continue? " file err))
+		     (error "Cannot read nnml marks file %s (%s)" file err))))
       ;; User didn't have a .marks file.  Probably first time
       ;; user of the .marks stuff.  Bootstrap it from .newsrc.eld.
       (let ((info (gnus-get-info
@@ -1019,6 +1020,8 @@
 	(nnheader-message 7 "Bootstrapping marks for %s..." group)
 	(setq nnml-marks (gnus-info-marks info))
 	(push (cons 'read (gnus-info-read info)) nnml-marks)
+	(dolist (el gnus-article-unpropagated-mark-lists)
+	  (setq nnml-marks (gnus-remassoc el nnml-marks)))
 	(nnml-save-marks group server)))))
 
 (provide 'nnml)
Index: nnfolder.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnfolder.el,v
retrieving revision 6.22
diff -u -r6.22 nnfolder.el
--- nnfolder.el	2001/09/08 20:55:16	6.22
+++ nnfolder.el	2001/09/22 16:49:49
@@ -1224,16 +1224,17 @@
 (defun nnfolder-open-marks (group server)
   (let ((file (nnfolder-group-marks-pathname group)))
     (if (file-exists-p file)
-	(setq nnfolder-marks (condition-case err
-				 (with-temp-buffer
-				   (gnus-sethash file
-						 (nth 5 (file-attributes file))
-						 nnfolder-marks-modtime)
-				   (nnheader-insert-file-contents file)
-				   (read (current-buffer)))
-			       (error (or (gnus-yes-or-no-p
-					   (format "Error reading nnfolder marks file %s (%s).  Continuing will use marks from .newsrc.eld.  Continue? " file err))
-					  (error "Cannot read nnfolder marks file %s (%s)" file err)))))
+	(condition-case err
+	    (with-temp-buffer
+	      (gnus-sethash file (nth 5 (file-attributes file)) 
+			    nnfolder-marks-modtime)
+	      (nnheader-insert-file-contents file)
+	      (setq nnfolder-marks (read (current-buffer)))
+	      (dolist (el gnus-article-unpropagated-mark-lists)
+		(setq nnfolder-marks (gnus-remassoc el nnfolder-marks))))
+	  (error (or (gnus-yes-or-no-p
+		      (format "Error reading nnfolder marks file %s (%s).  Continuing will use marks from .newsrc.eld.  Continue? " file err))
+		     (error "Cannot read nnfolder marks file %s (%s)" file err))))
       ;; User didn't have a .marks file.  Probably first time
       ;; user of the .marks stuff.  Bootstrap it from .newsrc.eld.
       (let ((info (gnus-get-info
@@ -1243,6 +1244,8 @@
 	(nnheader-message 7 "Bootstrapping marks for %s..." group)
 	(setq nnfolder-marks (gnus-info-marks info))
 	(push (cons 'read (gnus-info-read info)) nnfolder-marks)
+	(dolist (el gnus-article-unpropagated-mark-lists)
+	  (setq nnfolder-marks (gnus-remassoc el nnfolder-marks)))
 	(nnfolder-save-marks group server)))))
 
 (provide 'nnfolder)

Re: nnml/nnfolder marks faster

[Paul Jarc]

Simon Josefsson <jas@extundo.com> wrote:
> One could probably argue that seen marks should be specific to each
> Gnus installation -- you could use the seen mark to see which articles
> YOU have read in a shared mailbox in which some articles may have been
> marked as read by other people -- so I added `seen' to
> `gnus-article-unpropagated-mark-lists' as well.

I think everything stored in newsrc.eld should also be stored in the
backend.  Sharing seems to be a storage issue, so if a mailbox is
shared, it should be up to the backend to decide whether to share
marks as well, and which marks should be shared.

nnmaildir allows this by storing all 'read marks in
group-dir/.nnmaildir/marks/read, 'expire in .../expire, etc.  The
group-dir itself is never shared, but the subdirectory containing
messages may be a symlink to a shared message directory.  If you want
to share the 'read mark, you make .../read a symlink to a shared
location too.  If other backends aren't this flexible, that's a little
bit bad, but if Gnus caters to them at the expense of a more flexible
backend, I think that's worse.


paul

Re: nnml/nnfolder marks faster

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> Simon Josefsson <jas@extundo.com> wrote:
> > One could probably argue that seen marks should be specific to each
>> Gnus installation -- you could use the seen mark to see which articles
>> YOU have read in a shared mailbox in which some articles may have been
>> marked as read by other people -- so I added `seen' to
>> `gnus-article-unpropagated-mark-lists' as well.
>
> I think everything stored in newsrc.eld should also be stored in the
> backend.

Erm, even e.g. `cache', `download' and `unsend'?

I mean, if those marks are stored in the backend and the backend
somehow change these marks (e.g. by backup/restore) the marks will be
inaccurate and Gnus will break.

If an article is cached or not is not something a backend should (or
even could!) worry about, only the Gnus Cache stuff should worry about
that.

> Sharing seems to be a storage issue, so if a mailbox is
> shared, it should be up to the backend to decide whether to share
> marks as well, and which marks should be shared.

The marks I want to prevent being stored in the backend are those that
would break Gnus if they are changed at whim by the backend (by
e.g. backup/restore).

Consider the scenario where you mark an article as cached, take a
backup of your nnml server, remove the article from the cache, restore
the nnml server.  Now Gnus will think the article is cached, but it
isn't.

The long term solution is of course that the scoring system, Gnus
Cache and Gnus Agent keeps track of their internal states themselves,
instead of pushing this into the backend which isn't safe.
Fortunately, this doesn't seem impossible to achieve.

Do you still disagree?

Re: nnml/nnfolder marks faster

[Paul Jarc]

Simon Josefsson <jas@extundo.com> wrote:
> prj@po.cwru.edu (Paul Jarc) writes:
>> I think everything stored in newsrc.eld should also be stored in the
>> backend.
>
> Erm, even e.g. `cache', `download' and `unsend'?

Um.  Well, everything that would still be accurate if the group were
moved to another Gnus.  (I think this would include 'seen.)  For other
things, I wonder if it might make sense to make them not be marks.

> I mean, if those marks are stored in the backend and the backend
> somehow change these marks (e.g. by backup/restore) the marks will be
> inaccurate and Gnus will break.

Right.

> If an article is cached or not is not something a backend should (or
> even could!) worry about, only the Gnus Cache stuff should worry about
> that.

That's why I wonder if it might be good to, e.g., store all the cache
information in one place, separate from marks.

> The long term solution is of course that the scoring system, Gnus
> Cache and Gnus Agent keeps track of their internal states themselves,
> instead of pushing this into the backend which isn't safe.

Ah - I guess you have the same idea.  Convenient, that. :)


paul

Re: nnml/nnfolder marks faster

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> (I think this would include 'seen.)

I don't think so, right now we have three seenness marks: `unread',
`recent' and `seen'.

Unread and Recent is managed by the backend.  Seen is handled by Gnus.
Making Seen handled by the backend would make it equivalent to Recent,
which seems sort of useless.

(And getting rid of Seen in favor of Recent, or vice versa, would
exclude some possible uses of the marks.)

> For other things, I wonder if it might make sense to make them not
> be marks.

Yup.

Re: nnml/nnfolder marks faster

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:

> prj@po.cwru.edu (Paul Jarc) writes:
>
>> (I think this would include 'seen.)
>
> I don't think so, right now we have three seenness marks: `unread',
> `recent' and `seen'.
>
> Unread and Recent is managed by the backend.  Seen is handled by Gnus.
> Making Seen handled by the backend would make it equivalent to Recent,
> which seems sort of useless.
>
> (And getting rid of Seen in favor of Recent, or vice versa, would
> exclude some possible uses of the marks.)
>
>> For other things, I wonder if it might make sense to make them not
>> be marks.
>
> Yup.

It seems that marks, seen, recent, et all has introduced quite a few
problems.  For me it has been problems that seem to have solved
themselves at least temporarily.

Probably heresy but I'm wondering if the value of these things is so
great as to merit not just dropping it as a nice idea but not that
workable.

My understanding is that this started as a way to allow for backing up
readedness marks easily.  For my usage that is not now nor has it ever
been a big issue.  When I've had occasion to lose readedness marks it
has never been a major calamity.  Usually I can reconstruct close
enough without too much trouble, or just start over.

How serious of a problem is it to simply start over with the marks?
About the only ones that has immediate effects is the persistent,
dormant and tick mark, which depending on setting might mean a bunch
of stuff gets uncached and deleted or unmarked and deleted.  And of
course this could happen anyway, even with the new files in place.
Like maybe a random rogue M-c in group buffer.

I've used a backup system *outside* gnus for a long time.  It does not
backup marks but it does backup messages which is far more important.
It seem to me that marks are really not a core need.  That is, nothing
really bad happens if you loose them.

It seems further that the things that do represent significant effort
to reconstruct are still no better than ever. That is, the stuff in 
`G p' settings still resides in .newsrc.eld and is therefore no more
backupable than it ever was.

(Running for the root cellar....)
 

Re: nnml/nnfolder marks faster

[Nevin Kapur]

Harry Putnam <reader@newsguy.com> writes:

> Probably heresy but I'm wondering if the value of these things is so
> great as to merit not just dropping it as a nice idea but not that
> workable.

As an end-user, I would like to weigh in in favor of the new marks
code. I find the recently introduced Recent mark very useful. Also,
the ability to rsync a directory and have one's previous state
restored is very handy.

Whatever is decided, I hope there is a user-configurable variable that
allows one to use the current behavior.

-- 
Nevin Kapur
nevin@jhu.edu 

Re: nnml/nnfolder marks faster

[Simon Josefsson]

Nevin Kapur <nevin@jhu.edu> writes:

>> Probably heresy but I'm wondering if the value of these things is so
>> great as to merit not just dropping it as a nice idea but not that
>> workable.
>
> As an end-user, I would like to weigh in in favor of the new marks
> code. I find the recently introduced Recent mark very useful. Also,
> the ability to rsync a directory and have one's previous state
> restored is very handy.
>
> Whatever is decided, I hope there is a user-configurable variable that
> allows one to use the current behavior.

The `nnml-marks-is-evil' and `nnfolder-marks-is-evil' variables
exists, and disabling the nnml/nnfolder marks stuff should be as easy
as changing the default value of that variable.

You can't disable the recent or seen marks AFAIK though.

Re: nnml/nnfolder marks faster

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

> It seems that marks, seen, recent, et all has introduced quite a few
> problems. For me it has been problems that seem to have solved themselves at
> least temporarily.
>
> Probably heresy but I'm wondering if the value of these things is so
> great as to merit not just dropping it as a nice idea but not that
> workable.

If noone feel these changes serve any purpose, I guess we could back
them out.  But if someone finds them useful, maybe it is nice to keep
it and isolate any possible problems.

Most features have problems in the beginning, and we are talking about
several new features here -- seen, recent, self-contained nnml and
self-contained nnfolder are four separate features, with their own
separate set of bugs.  Thinking of them as one new feature is not
fair, I think.

> My understanding is that this started as a way to allow for backing up
> readedness marks easily.  For my usage that is not now nor has it ever
> been a big issue.  When I've had occasion to lose readedness marks it
> has never been a major calamity.  Usually I can reconstruct close
> enough without too much trouble, or just start over.
>
> How serious of a problem is it to simply start over with the marks?
> About the only ones that has immediate effects is the persistent,
> dormant and tick mark, which depending on setting might mean a bunch
> of stuff gets uncached and deleted or unmarked and deleted.  And of
> course this could happen anyway, even with the new files in place.
> Like maybe a random rogue M-c in group buffer.
>
> I've used a backup system *outside* gnus for a long time.  It does not
> backup marks but it does backup messages which is far more important.
> It seem to me that marks are really not a core need.  That is, nothing
> really bad happens if you loose them.

Marks in nnml and nnfolder had this goal, yes.  It sounds as if you
don't need the feature, and modulo bugs the absence or presence of the
feature shouldn't matter then.

The seen and recent marks had different goals though.

> It seems further that the things that do represent significant effort
> to reconstruct are still no better than ever. That is, the stuff in 
> `G p' settings still resides in .newsrc.eld and is therefore no more
> backupable than it ever was.

Right, I think we could get to that as well, in time.

Re: nnml/nnfolder marks faster

[Kai Großjohann]

Harry Putnam <reader@newsguy.com> writes:

> Probably heresy but I'm wondering if the value of these things is so
> great as to merit not just dropping it as a nice idea but not that
> workable.

I think this marks stuff is important.  For example, it would allow
you to rsync your ~/Mail directory between two machines with no fuss,
even if you otherwise use different servers.

I would be happy if an effort was made to replace the .newsrc.eld file
with per-server (or per-group, even) files.  That would be way cool.

kai
-- 
Symbol's function definition is void: signature

Re: nnml/nnfolder marks faster

[Paul Jarc]

Simon Josefsson <jas@extundo.com> wrote:
> prj@po.cwru.edu (Paul Jarc) writes:
>> (I think this would include 'seen.)
>
> I don't think so, right now we have three seenness marks: `unread',
> `recent' and `seen'.
> 
> Unread and Recent is managed by the backend.  Seen is handled by Gnus.

Currently, yes.  I'm not convinced that it needs or ought to be that
way.

> Making Seen handled by the backend would make it equivalent to Recent,
> which seems sort of useless.

You're saying that Gnus is capable of storing Seen in a way that
preserves the semantic difference from Recent, but backends are not?
Maybe that's true for the current version of nnimap (either all marks
are shared or all are private, right?), but it can't be generally
true: if nothing else, backends could potentially store Seen the same
way Gnus does, or nearly the same way.

I wouldn't object to hiding Seen from nnimap if nnimap can't handle it
properly, but such hiding should not be forced on all backends.  The
list of hidden marks should be configured at least per-backend, if not
per-server or maybe even per-group.  The backend interface should not
assume limitations in backends' capabilities, and semantic issues
should be kept orthogonal to storage issues as much as possible.


paul

Re: nnml/nnfolder marks faster

[Simon Josefsson]

On Mon, 24 Sep 2001, Paul Jarc wrote:

> > Making Seen handled by the backend would make it equivalent to Recent,
> > which seems sort of useless.
>
> You're saying that Gnus is capable of storing Seen in a way that
> preserves the semantic difference from Recent, but backends are not?

Well, sort of.  I don't doubt that back ends are _capable_ of storing the
mark, but I think back ends should not store marks that don't have any
semantic meaning in the back end. Or maybe rather, back ends should not
store marks that have a semantic meaning somewhere ELSE. Just like the
Gnus Cache or Gnus Agent marks. This is because it prevents backup and
stuff like that..

> Maybe that's true for the current version of nnimap (either all marks
> are shared or all are private, right?)

It depends on the server, I think.  Some servers allows sharing marks,
and some allows even fine-grained control over what marks are shared using
IMAP ACLs, per group.  Some doesn't allow non-standard marks at all.  And
some servers just forgets marks randomly..

> I wouldn't object to hiding Seen from nnimap if nnimap can't handle it
> properly, but such hiding should not be forced on all backends.  The
> list of hidden marks should be configured at least per-backend, if not
> per-server or maybe even per-group.  The backend interface should not
> assume limitations in backends' capabilities, and semantic issues
> should be kept orthogonal to storage issues as much as possible.

I think I agree -- but I view the Seen mark similar to the Cache or Agent
marks, so I don't want the mark in the back end.  (Hmm, this new "back
end" feels a bit funny..)

Ok, so are you suggesting that the Seen mark should be stored in the
backend as well?  In this case, what makes Seen different from Recent?
I.e., why should we have both then?  Maybe I'm just missing something.

Re: nnml/nnfolder marks faster

[Paul Jarc]

Simon Josefsson <jas@extundo.com> wrote:
> I don't doubt that back ends are _capable_ of storing the mark, but
> I think back ends should not store marks that don't have any
> semantic meaning in the back end.

I don't think *any* marks have semantic meaning in the backend.

> Ok, so are you suggesting that the Seen mark should be stored in the
> backend as well?  In this case, what makes Seen different from Recent?

I'm not sure I know what the semantic difference between them is.
Tell me that, and I'll tell you how that difference can be preserved
when storing them in the backend.


paul

Re: nnml/nnfolder marks faster

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

>> I don't doubt that back ends are _capable_ of storing the mark, but
>> I think back ends should not store marks that don't have any
>> semantic meaning in the back end.
>
> I don't think *any* marks have semantic meaning in the backend.

Maybe I chosed a poor word, I meant the marks that the back end sanely
can modify.  Read, Tick, Answered etc marks all have a well defined
meaning and can be modified, without causing harm in other parts of
Gnus, by the backend.

>> Ok, so are you suggesting that the Seen mark should be stored in the
>> backend as well?  In this case, what makes Seen different from Recent?
>
> I'm not sure I know what the semantic difference between them is.
> Tell me that, and I'll tell you how that difference can be preserved
> when storing them in the backend.

As I think it is right now:

Seen: Each Gnus instance keeps track of which articles you have seen
and displays a `.' secondary mark on "unseen" articles.

Recent: The back end keeps track of which articles you haven't seen,
and Gnus displays a `N' secondary mark of "recent" articles.

Both marks are different from the usual "unread" because the unread
mark can be set and cleared by the user.

Re: nnml/nnfolder marks faster

[Paul Jarc]

Simon Josefsson <jas@extundo.com> wrote:
> Seen: Each Gnus instance keeps track of which articles you have seen
> and displays a `.' secondary mark on "unseen" articles.
>
> Recent: The back end keeps track of which articles you haven't seen,
> and Gnus displays a `N' secondary mark of "recent" articles.

So changes to Seen marks are initiated by Gnus, and changes to Recent
marks are initiated by servers (not backends, I think - nnimap just
relays the Recent marks it gets from the server, it doesn't track them
internally, right?).

I think backends just pass marks between servers and Gnus.  They don't
know where changes are initiated; they just keep everybody
synchronized.  If backends store Seen, nothing changes - it just makes
it easier for users to manipulate those marks from outside Gnus.  They
wouldn't normally do this, but if they want to, why try to stop them?


paul

Re: nnml/nnfolder marks faster

[Dan Christensen]

Simon Josefsson <jas@extundo.com> writes:

> Ok, here is the complete set of patches that fixes things, include
> Paul's doc fix.  Please test it and I'll commit it.

It works for me.  Thanks!

> Are there any outstanding issues with the .mrk/.marks after this?  

Not that I can think of.

> How is the performance for you?  

It seems as fast as usual now.  Thanks.

> Profiling results of
> `nnfolder-request-update-info' and `nnfolder-request-set-mark' for
> largish groups would be interesting.

How do you want me to call those functions, and from what buffer?
Should I elp-instrument-package all of gnus?

Dan

-- 
Dan Christensen
jdc+news@uwo.ca

Re: nnml/nnfolder marks faster

[Simon Josefsson]

Dan Christensen <jdc+news@uwo.ca> writes:

>> Profiling results of
>> `nnfolder-request-update-info' and `nnfolder-request-set-mark' for
>> largish groups would be interesting.
>
> How do you want me to call those functions, and from what buffer?
> Should I elp-instrument-package all of gnus?

Yes, please elp-instrument-package "gnus" and "nn" and evaluate (setq
nnfolder-marks-modtime (gnus-make-hashtable)) to clear the modtime
cache.  Then M-x elp-reset-all RET, enter a large nnfolder, and M-x
elp-results RET.  Then change one mark, M-x elp-reset-all RET, quit
the group with `q' and M-x elp-results RET.

I've committed the patch.

Re: nnml/nnfolder marks faster

[Dan Christensen]

[-- Attachment #1: Type: text/plain, Size: 743 bytes --]

Simon Josefsson <jas@extundo.com> writes:

> Profiling results of
> `nnfolder-request-update-info' and `nnfolder-request-set-mark' for
> largish groups would be interesting.

Ok, attached are the results from a enter and the results from an
exit, done as you suggested.

One oddity:  The first time I hit SPC after elp-reset-all, I get

  Retrieving newsgroup: nnfolder:geg...
  Key sequence A P uses invalid prefix characters

The second time I hit SPC, Gnus enters the group without a problem.
That's why the results show two calls to some functions.  So take the
elapsed time, not the average time, for these functions.

If I do elp-results after the error message, then when I hit SPC
a second time, emacs dies!  Completely reproducible.


[-- Attachment #2: enter.results --]
[-- Type: application/octet-stream, Size: 15508 bytes --]

Function Name                                           Call Count  Elapsed Time  Average Time
======================================================  ==========  ============  ============
gnus-topic-read-group                                   2           6.91517       3.457585
gnus-group-read-group                                   2           6.914893      3.4574465
gnus-summary-read-group                                 2           6.91401       3.457005
gnus-summary-read-group-1                               2           6.913954      3.456977
gnus-summary-prepare                                    1           3.517201      3.517201
gnus-summary-prepare-threads                            1           3.1134560000  3.1134560000
gnus-select-newsgroup                                   1           2.643356      2.643356
gnus-fetch-headers                                      1           2.1616239999  2.1616239999
gnus-retrieve-headers                                   2           1.9280249999  0.9640124999
gnus-get-newsgroup-headers-xover                        1           1.194242      1.194242
gnus-summary-from-or-to-or-newsgroups                   381         1.0954090000  0.0028750892
gnus-cache-retrieve-headers                             1           0.964245      0.964245
nnfolder-retrieve-headers                               1           0.963349      0.963349
nnfolder-possibly-change-group                          3           0.94385       0.3146166666
nnfolder-read-folder                                    1           0.922397      0.922397
gnus-split-references                                   764         0.5518150000  0.0007222709
gnus-correct-substring                                  45          0.5452049999  0.0121156666
gnus-correct-length                                     512         0.528213      0.0010316660
nnheader-find-file-noselect                             1           0.500325      0.500325
gnus-gather-threads-by-subject                          1           0.346348      0.346348
gnus-adjust-marked-articles                             1           0.342421      0.342421
gnus-uncompress-range                                   9           0.304108      0.0337897777
gnus-thread-total-score                                 199         0.294596      0.0014803819
gnus-user-format-function-t                             127         0.2894059999  0.0022787874
gnus-thread-total-score-1                               199         0.2866779999  0.0014405929
gnus-dd-mmm                                             127         0.2491480000  0.0019617952
gnus-simplify-subject-fuzzy                             467         0.2024439999  0.0004334989
gnus-summary-limit-children                             306         0.2016169999  0.0006588790
gnus-summary-initial-limit                              1           0.112051      0.112051
gnus-article-marked-p                                   69          0.0892289999  0.0012931739
gnus-member-of-range                                    306         0.0890710000  0.0002910816
gnus-sort-threads-1                                     71          0.050934      0.0007173802
gnus-sort-threads                                       1           0.049246      0.049246
gnus-run-hooks                                          134         0.0483230000  0.0003606194
gnus-char-width                                         8341        0.0432159999  5.181...e-06
gnus-summary-highlight-line                             126         0.040602      0.0003222380
nnfolder-retrieve-headers-with-nov                      1           0.039379      0.039379
nnheader-insert-file-contents                           1           0.037986      0.037986
gnus-set-work-buffer                                    485         0.0270190000  5.570...e-05
nnfolder-save-active                                    1           0.026995      0.026995
nnmail-save-active                                      1           0.026927      0.026927
nnmail-generate-active                                  1           0.025707      0.025707
gnus-point-at-eol                                       1155        0.0210250000  1.820...e-05
gnus-request-group                                      1           0.020602      0.020602
nnmail-activate                                         3           0.020492      0.0068306666
nnfolder-request-group                                  1           0.020346      0.020346
gnus-make-threads                                       3           0.019247      0.0064156666
nnfolder-request-list                                   1           0.01885       0.01885
nnmail-get-active                                       1           0.017745      0.017745
nnmail-parse-active                                     1           0.017681      0.017681
gnus-message                                            12          0.0144269999  0.00120225
gnus-put-text-property                                  509         0.013577      2.667...e-05
gnus-put-text-property-excluding-characters-with-faces  127         0.0127679999  0.0001005354
gnus-thread-sort-by-number                              1148        0.0124979999  1.088...e-05
gnus-group-find-parameter                               6           0.011593      0.0019321666
gnus-group-topic-parameters                             6           0.0101129999  0.0016854999
gnus-parse-format                                       5           0.009327      0.0018654000
gnus-point-at-bol                                       631         0.0081069999  1.284...e-05
gnus-update-format-specifications                       1           0.007939      0.007939
gnus-topic-hierarchical-parameters                      6           0.0077969999  0.0012994999
gnus-parse-complex-format                               3           0.0071330000  0.0023776666
gnus-complex-form-to-spec                               6           0.006403      0.0010671666
gnus-topic-parent-topic                                 126         0.0063800000  5.063...e-05
gnus-current-topics                                     6           0.0052109999  0.0008684999
gnus-parse-simple-format                                13          0.0050780000  0.0003906153
gnus-update-summary-mark-positions                      1           0.004676      0.004676
gnus-summary-setup-default-charset                      1           0.004659      0.004659
gnus-summary-insert-line                                1           0.004307      0.004307
gnus-articles-to-read                                   1           0.004191      0.004191
gnus-thread-loop-p                                      267         0.0036759999  1.376...e-05
gnus-update-read-articles                               1           0.003428      0.003428
gnus-id-to-thread                                       199         0.0032639999  1.640...e-05
gnus-set-format                                         2           0.003155      0.0015775
gnus-configure-frame                                    5           0.002849      0.0005698
gnus-parameter-charset                                  1           0.002456      0.002456
gnus-remove-from-range                                  2           0.00228       0.00114
gnus-parameter-ignored-charsets                         1           0.002068      0.002068
gnus-group-auto-expirable-p                             1           0.002002      0.002002
gnus-configure-windows                                  1           0.00197       0.00197
gnus-topic-parameters                                   18          0.0015940000  8.855...e-05
gnus-copy-sequence                                      31          0.0015260000  4.922...e-05
gnus-sorted-complement                                  2           0.0015119999  0.0007559999
gnus-topic-find-topology                                36          0.001399      3.886...e-05
gnus-range-length                                       4           0.001287      0.00032175
gnus-sort-gathered-threads                              1           0.001242      0.001242
gnus-killed-articles                                    1           0.001024      0.001024
nnmail-find-file                                        1           0.000909      0.000909
gnus-set-mode-line                                      1           0.000891      0.000891
gnus-continuum-version                                  1           0.000817      0.000817
gnus-set-sorted-intersection                            1           0.000743      0.000743
gnus-apply-kill-file                                    1           0.000715      0.000715
nnfolder-group-pathname                                 6           0.0006840000  0.0001140000
gnus-group-goto-group                                   7           0.0005939999  8.485...e-05
nnheader-nov-delete-outside-range                       1           0.000582      0.000582
gnus-sorted-intersection                                1           0.000581      0.000581
nnheader-init-server-buffer                             3           0.000568      0.0001893333
gnus-get-buffer-create                                  8           0.000562      7.025e-05
gnus-set-global-variables                               3           0.000543      0.0001809999
gnus-summary-make-local-variables                       2           0.000539      0.0002695
gnus-current-topic                                      6           0.000529      8.816...e-05
gnus-cache-articles-in-group                            1           0.000502      0.000502
nnheader-find-nov-line                                  2           0.000487      0.0002435
gnus-newsgroup-kill-file                                2           0.000411      0.0002055
gnus-summary-set-display-table                          1           0.000392      0.000392
gnus-group-decoded-name                                 4           0.0003590000  8.975...e-05
gnus-find-method-for-group                              10          0.000355      3.55e-05
gnus-group-topic-p                                      2           0.000332      0.000166
gnus-cache-file-name                                    1           0.000323      0.000323
nnfolder-group-nov-pathname                             2           0.000315      0.0001575
gnus-list-of-unread-articles                            1           0.000311      0.000311
nnfolder-server-opened                                  3           0.000292      9.733...e-05
gnus-last-element                                       1           0.000276      0.000276
gnus-short-group-name                                   1           0.00027       0.00027
gnus-get-unread-articles-in-group                       1           0.000257      0.000257
gnus-group-topic-name                                   2           0.0002510000  0.0001255000
gnus-check-server                                       1           0.000234      0.000234
gnus-group-get-parameter                                7           0.0002320000  3.314...e-05
gnus-make-hashtable                                     2           0.000228      0.000114
gnus-all-windows-visible-p                              1           0.000205      0.000205
gnus-group-parameter-value                              6           0.0002000000  3.333...e-05
gnus-server-opened                                      1           0.000197      0.000197
gnus-summary-setup-buffer                               2           0.000197      9.85e-05
gnus-newsgroup-savable-name                             3           0.000195      6.5e-05
nnoo-server-opened                                      3           0.000174      5.8e-05
nnmail-message-id                                       1           0.00016       0.00016
gnus-group-group-name                                   2           0.000146      7.3e-05
gnus-summary-position-point                             1           0.000145      0.000145
gnus-summary-make-menu-bar                              1           0.00013       0.00013
gnus-group-quit-config                                  1           0.000122      0.000122
gnus-file-newer-than                                    1           0.000116      0.000116
gnus-goto-colon                                         1           0.000109      0.000109
gnus-simplify-mode-line                                 2           9.2e-05       4.6e-05
gnus-virtual-group-p                                    1           8.9e-05       8.9e-05
gnus-turn-off-edit-menu                                 1           8.7e-05       8.7e-05
nnheader-report                                         1           7.4e-05       7.4e-05
gnus-set-difference                                     2           7.099...e-05  3.549...e-05
gnus-article-mark-to-type                               6           6.8e-05       1.133...e-05
gnus-parameters-get-parameter                           6           6.7e-05       1.116...e-05
gnus-mouse-face-function                                3           6.400...e-05  2.133...e-05
gnus-undo-register                                      1           6.3e-05       6.3e-05
gnus-create-hash-size                                   2           5.9e-05       2.95e-05
gnus-use-long-file-name                                 5           5.699...e-05  1.14e-05
gnus-summary-buffer-name                                2           5.200...e-05  2.600...e-05
gnus-visual-p                                           4           5.2e-05       1.3e-05
gnus-tilde-max-form                                     2           5.1e-05       2.55e-05
gnus-mode-line-buffer-identification                    1           4.9e-05       4.9e-05
gnus-window-to-buffer-helper                            5           4.8e-05       9.6e-06
nnheader-translate-file-chars                           4           4.700...e-05  1.175...e-05
gnus-undo-boundary                                      4           4.4e-05       1.1e-05
gnus-correct-pad-form                                   2           3.6e-05       1.8e-05
nnoo-current-server                                     3           3.1e-05       1.033...e-05
gnus-summary-display-make-predicate                     1           2.9e-05       2.9e-05
gnus-cut-threads                                        1           2.8e-05       2.8e-05
gnus-get-function                                       1           2.4e-05       2.4e-05
gnus-spec-tab                                           1           2.4e-05       2.4e-05
gnus-undo-register-1                                    1           2e-05         2e-05
gnus-make-sort-function                                 1           1.7e-05       1.7e-05
gnus-server-status                                      1           1.4e-05       1.4e-05
gnus-add-text-properties                                1           1.3e-05       1.3e-05
gnus-windows-old-to-new                                 1           1.2e-05       1.2e-05
gnus-summary-make-tool-bar                              1           1.2e-05       1.2e-05
gnus-server-to-method                                   1           1.2e-05       1.2e-05
gnus-make-thread-indent-array                           1           1.1e-05       1.1e-05
gnus-set-default-directory                              1           9e-06         9e-06
gnus-update-missing-marks                               1           8e-06         8e-06
gnus-summary-mode                                       1           0.0           0.0

[-- Attachment #3: exit.results --]
[-- Type: application/octet-stream, Size: 10613 bytes --]

Function Name                                           Call Count  Elapsed Time  Average Time
======================================================  ==========  ============  ============
gnus-summary-exit                                       1           0.581116      0.581116
gnus-summary-update-info                                1           0.5557259999  0.5557259999
gnus-update-marks                                       1           0.5105040000  0.5105040000
gnus-check-group                                        1           0.276002      0.276002
gnus-copy-sequence                                      961         0.0577880000  6.013...e-05
gnus-remove-from-range                                  21          0.044493      0.0021187142
gnus-compress-sequence                                  10          0.030644      0.0030644
gnus-mark-xrefs-as-read                                 1           0.026937      0.026937
gnus-create-xref-hashtb                                 1           0.026588      0.026588
gnus-request-set-mark                                   1           0.015017      0.015017
nnfolder-request-set-mark                               1           0.014665      0.014665
gnus-group-update-group                                 1           0.014194      0.014194
gnus-topic-update-topic-line                            3           0.0081170000  0.0027056666
nnfolder-open-marks                                     1           0.007009      0.007009
nnfolder-save-marks                                     1           0.006007      0.006007
gnus-prin1-to-string                                    1           0.004997      0.004997
gnus-topic-update-topics-containing-group               1           0.004803      0.004803
gnus-remassoc                                           39          0.004116      0.0001055384
gnus-update-read-articles                               1           0.003412      0.003412
gnus-group-insert-group-line-info                       1           0.002573      0.002573
gnus-add-to-range                                       1           0.002516      0.002516
gnus-group-insert-group-line                            1           0.002429      0.002429
gnus-write-buffer                                       2           0.002343      0.0011715
gnus-configure-windows                                  1           0.002065      0.002065
gnus-close-group                                        1           0.001689      0.001689
nnheader-insert-file-contents                           1           0.001559      0.001559
nnfolder-close-group                                    1           0.001449      0.001449
gnus-deaden-summary                                     1           0.001065      0.001065
gnus-topic-insert-topic-line                            2           0.001058      0.000529
gnus-topic-parent-topic                                 21          0.001056      5.028...e-05
gnus-run-hooks                                          5           0.000965      0.000193
gnus-range-add                                          1           0.000941      0.000941
gnus-prin1                                              1           0.000874      0.000874
gnus-group-highlight-line                               1           0.000836      0.000836
gnus-range-length                                       3           0.0008060000  0.0002686666
gnus-topic-find-groups                                  3           0.00076       0.0002533333
gnus-uncompress-range                                   3           0.0006889999  0.0002296666
gnus-find-method-for-group                              21          0.000686      3.266...e-05
gnus-group-set-mode-line                                2           0.000611      0.0003055
gnus-dribble-enter                                      1           0.0006        0.0006
gnus-make-directory                                     4           0.000496      0.000124
gnus-dead-summary-mode                                  1           0.000495      0.000495
gnus-group-get-parameter                                5           0.0004840000  9.680...e-05
gnus-configure-frame                                    2           0.000459      0.0002295
gnus-group-jump-to-group                                1           0.000455      0.000455
gnus-ephemeral-group-p                                  3           0.0004310000  0.0001436666
gnus-kill-save-kill-buffer                              1           0.000424      0.000424
gnus-add-minor-mode                                     1           0.000419      0.000419
nnfolder-group-marks-pathname                           2           0.000417      0.0002085
gnus-set-global-variables                               2           0.000342      0.000171
gnus-group-next-unread-group                            1           0.000328      0.000328
nnfolder-group-pathname                                 2           0.000319      0.0001595
gnus-group-position-point                               2           0.000312      0.000156
gnus-cache-possibly-remove-articles                     1           0.000282      0.000282
gnus-newsgroup-kill-file                                1           0.000278      0.000278
gnus-group-search-forward                               1           0.00027       0.00027
gnus-topic-goto-topic                                   4           0.000265      6.625e-05
gnus-topic-find-topology                                6           0.00026       4.333...e-05
gnus-virtual-group-p                                    3           0.000255      8.5e-05
gnus-get-unread-articles-in-group                       1           0.000255      0.000255
gnus-goto-colon                                         2           0.000241      0.0001205
gnus-group-topic-unread                                 5           0.000233      4.66e-05
gnus-server-opened                                      1           0.000226      0.000226
nnfolder-server-opened                                  2           0.000201      0.0001005
gnus-mode-line-buffer-identification                    2           0.000194      9.7e-05
gnus-update-alist-soft                                  2           0.000166      8.3e-05
gnus-article-mark-to-type                               14          0.000142      1.014...e-05
gnus-group-timestamp-delta                              1           0.000139      0.000139
gnus-group-parameter-value                              5           0.000138      2.76e-05
gnus-point-at-bol                                       13          0.000134      1.030...e-05
gnus-all-windows-visible-p                              1           0.000131      0.000131
gnus-group-group-name                                   2           0.000129      6.45e-05
gnus-cache-possibly-remove-articles-1                   1           0.00012       0.00012
gnus-group-quit-config                                  1           0.00012       0.00012
nnoo-server-opened                                      2           0.0001189999  5.949...e-05
gnus-add-text-properties                                5           0.0001130000  2.26e-05
nnheader-set-temp-buffer                                1           0.000112      0.000112
gnus-group-group-indentation                            2           0.0001089999  5.449...e-05
gnus-article-unpropagatable-p                           14          0.0001049999  7.499...e-06
gnus-put-text-property-excluding-characters-with-faces  1           9e-05         9e-05
gnus-topic-visible-p                                    2           8.5e-05       4.25e-05
gnus-group-topic-level                                  2           8.4e-05       4.2e-05
gnus-cache-fully-p                                      1           8e-05         8e-05
gnus-get-function                                       3           7.8e-05       2.6e-05
gnus-group-new-mail                                     1           7.8e-05       7.8e-05
gnus-newsgroup-savable-name                             1           7.3e-05       7.3e-05
gnus-asynchronous-p                                     1           7.2e-05       7.2e-05
gnus-undo-register                                      1           6.3e-05       6.3e-05
nnfolder-save-buffer                                    1           6e-05         6e-05
gnus-put-text-property                                  3           5.9e-05       1.966...e-05
nnheader-translate-file-chars                           6           5.8e-05       9.666...e-06
gnus-group-real-prefix                                  1           5.6e-05       5.6e-05
gnus-group-goto-group                                   1           5e-05         5e-05
gnus-server-to-method                                   3           4.7e-05       1.566...e-05
gnus-point-at-eol                                       2           4.6e-05       2.3e-05
gnus-delete-alist                                       2           4.200...e-05  2.100...e-05
gnus-make-hashtable                                     1           4.1e-05       4.1e-05
gnus-topic-update-unreads                               2           4.1e-05       2.05e-05
gnus-window-to-buffer-helper                            2           2.2e-05       1.1e-05
gnus-undo-register-1                                    1           2e-05         2e-05
nnoo-current-server                                     2           1.999...e-05  9.999...e-06
gnus-server-status                                      1           1.7e-05       1.7e-05
nnfolder-save-nov                                       1           1.6e-05       1.6e-05
gnus-get-buffer-create                                  1           1.4e-05       1.4e-05
gnus-windows-old-to-new                                 1           1.3e-05       1.3e-05
nnmail-purge-split-history                              1           1.3e-05       1.3e-05
gnus-cache-write-active                                 1           1.2e-05       1.2e-05
gnus-undo-force-boundary                                1           1.1e-05       1.1e-05
nnmail-new-mail-p                                       1           1.1e-05       1.1e-05
gnus-undo-boundary                                      1           9e-06         9e-06
gnus-use-long-file-name                                 1           8e-06         8e-06
gnus-extent-start-open                                  1           7e-06         7e-06
gnus-cache-save-buffers                                 1           7e-06         7e-06
gnus-group-remove-excess-properties                     1           7e-06         7e-06

[-- Attachment #4: Type: text/plain, Size: 37 bytes --]


-- 
Dan Christensen
jdc+news@uwo.ca

Re: nnml/nnfolder marks faster

[Simon Josefsson]

Dan Christensen <jdc+news@uwo.ca> writes:

>> Profiling results of
>> `nnfolder-request-update-info' and `nnfolder-request-set-mark' for
>> largish groups would be interesting.
>
> Ok, attached are the results from a enter and the results from an
> exit, done as you suggested.

Thanks!  It seems the gnus-range functions are the major culprit.
nnml/nnfolder-request-set-mark consumed 1-2 % of the time.
nn*-request-update-info wasn't called though, maybe it is only called
when doing M-g instead of entering the group?

Maybe someone could try replacing the gnus-range-* functions with
simple list manipulation functions to see how much the range functions
really increase performance.  They seem to consume quite some time..
Take a backup first, though. :)

> One oddity:  The first time I hit SPC after elp-reset-all, I get
>
>   Retrieving newsgroup: nnfolder:geg...
>   Key sequence A P uses invalid prefix characters
>
> The second time I hit SPC, Gnus enters the group without a problem.

Beats me.  Anyone?

> If I do elp-results after the error message, then when I hit SPC
> a second time, emacs dies!  Completely reproducible.

Perhaps the emacs developers would like a bug report. Assuming the
latest pretest exhibits the same problem.