* nnml/nnfolder marks faster @ 2001-09-08 19:38 Simon Josefsson 2001-09-12 19:42 ` Dan Christensen 0 siblings, 1 reply; 39+ messages in thread From: Simon Josefsson @ 2001-09-08 19:38 UTC (permalink / raw) The marks stuff should now be faster as the marks file is only read when it is necessary. A little implementation note on how the marks stuff works follows, if anyone is interested. Two new functions were added: *-request-update-info: Reads marks file if it has changed (or if it has not been previously read, e.g. on startup of Gnus) and updates Gnus' info from the marks file. *-request-set-mark: Reads marks file, applies the changes, writes marks file. Updates modtime. (Nothing else was changed.) ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-08 19:38 nnml/nnfolder marks faster Simon Josefsson @ 2001-09-12 19:42 ` Dan Christensen 2001-09-14 13:43 ` Simon Josefsson 0 siblings, 1 reply; 39+ messages in thread From: Dan Christensen @ 2001-09-12 19:42 UTC (permalink / raw) I have a few groups that have really large .mrk files, and this is causing slow performance. I've noticed this more since the new .mrk code was introduced, and it is in particular noticeable when I *exit* a group with a large .mrk file. But I don't know if the new .mrk code is causing any problems, or whether it just makes them more visible. As an example, my nnfolder:debian.big group has about 1000 articles in it, and the .mrk file is somehow using about 840 bytes per article, which seems really excessive: -rw------- 1 jdc jdc 4231285 Sep 12 13:56 Gnus/Mail/debian.big -rw-r--r-- 1 jdc jdc 840761 Sep 12 14:46 Gnus/Mail/debian.big.mrk -rw------- 1 jdc jdc 310299 Sep 12 14:42 Gnus/Mail/debian.big.nov Here are a few excerpts from the .mrk file which look suspicious: There are screenfuls and screenfuls of "seen" marks that look like: (184755 . 184758) (184759 . 184762) which should be merged into a single long list I would think. Also, the score caching has stuff like: (score (185776 . -1) (185820 . -1) (185962 . -1) (185831 . -1) (185991 . -1) which isn't even stored in numerical order (how is lookup done??). I also wonder if this could be stored in a collapsed form using ranges (score ((12345 . 15000) . -1) etc. (in cases where there are consecutive articles with the same score.) Hmm, looking further, it is the score entries that are taking up 99% of the .mrk file, and most of them don't correspond to articles that exist in the group! Shouldn't score cache entries be removed when the article is deleted from the group (e.g. by the expiry mechanism)? I smell a bug. Dan -- Dan Christensen jdc+news@uwo.ca ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-12 19:42 ` Dan Christensen @ 2001-09-14 13:43 ` Simon Josefsson 2001-09-14 14:51 ` Paul Jarc 2001-09-14 18:11 ` Dan Christensen 0 siblings, 2 replies; 39+ messages in thread From: Simon Josefsson @ 2001-09-14 13:43 UTC (permalink / raw) Cc: ding Dan Christensen <jdc+news@uwo.ca> writes: > I have a few groups that have really large .mrk files, and this is > causing slow performance. I've noticed this more since the new > .mrk code was introduced, and it is in particular noticeable when > I *exit* a group with a large .mrk file. But I don't know if the > new .mrk code is causing any problems, or whether it just makes > them more visible. I guess the latter, I think the problem you mention existed earlier (because the backend marks code hasn't touched that code). > As an example, my nnfolder:debian.big group has about 1000 articles > in it, and the .mrk file is somehow using about 840 bytes per article, > which seems really excessive: Yes, too much. > Here are a few excerpts from the .mrk file which look suspicious: > > There are screenfuls and screenfuls of "seen" marks that look like: > > (184755 . 184758) (184759 . 184762) > > which should be merged into a single long list I would think. That is probably better yes, but I don't know the seen mark, it seem to be handled specially somehow. Lars? > Also, the score caching has stuff like: > > (score (185776 . -1) (185820 . -1) (185962 . -1) (185831 . -1) > (185991 . -1) > > which isn't even stored in numerical order (how is lookup done??). > I also wonder if this could be stored in a collapsed form using ranges > > (score ((12345 . 15000) . -1) > > etc. (in cases where there are consecutive articles with the same > score.) > > Hmm, looking further, it is the score entries that are taking up 99% > of the .mrk file, and most of them don't correspond to articles that > exist in the group! Shouldn't score cache entries be removed when > the article is deleted from the group (e.g. by the expiry mechanism)? > I smell a bug. I'm not sure score marks should be in the file at all. Isn't the scores calculated each time you enter the group? Maybe the score is cached in .newsrc/.mark? Likewise, `downloaded', `cache' and `unsent' should probably never be propagated to the backend. ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-14 13:43 ` Simon Josefsson @ 2001-09-14 14:51 ` Paul Jarc 2001-09-14 17:32 ` Simon Josefsson 2001-09-14 18:11 ` Dan Christensen 1 sibling, 1 reply; 39+ messages in thread From: Paul Jarc @ 2001-09-14 14:51 UTC (permalink / raw) Simon Josefsson <jas@extundo.com> wrote: > Dan Christensen <jdc+news@uwo.ca> writes: >> There are screenfuls and screenfuls of "seen" marks that look like: >> >> (184755 . 184758) (184759 . 184762) >> >> which should be merged into a single long list I would think. > > That is probably better yes, but I don't know the seen mark, it seem > to be handled specially somehow. Lars? I think (gnus-compress-sequence (gnus-uncompress-range range)) would do it. How is 'seen special? > I'm not sure score marks should be in the file at all. Isn't the > scores calculated each time you enter the group? Maybe the score is > cached in .newsrc/.mark? > > Likewise, `downloaded', `cache' and `unsent' should probably never be > propagated to the backend. This would mean Gnus should remove those marks before passing the info right? I don't think backends should have to know which marks they aren't supposed to know about. paul ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-14 14:51 ` Paul Jarc @ 2001-09-14 17:32 ` Simon Josefsson 2001-09-14 18:08 ` Paul Jarc 0 siblings, 1 reply; 39+ messages in thread From: Simon Josefsson @ 2001-09-14 17:32 UTC (permalink / raw) prj@po.cwru.edu (Paul Jarc) writes: > Simon Josefsson <jas@extundo.com> wrote: > > Dan Christensen <jdc+news@uwo.ca> writes: >>> There are screenfuls and screenfuls of "seen" marks that look like: >>> >>> (184755 . 184758) (184759 . 184762) >>> >>> which should be merged into a single long list I would think. >> >> That is probably better yes, but I don't know the seen mark, it seem >> to be handled specially somehow. Lars? > > I think (gnus-compress-sequence (gnus-uncompress-range range)) would > do it. How is 'seen special? It is a range instead of a list, which most other marks is. And there is some special logic to set the variable. Where should the snippet you suggest be put? >> I'm not sure score marks should be in the file at all. Isn't the >> scores calculated each time you enter the group? Maybe the score is >> cached in .newsrc/.mark? >> >> Likewise, `downloaded', `cache' and `unsent' should probably never be >> propagated to the backend. > > This would mean Gnus should remove those marks before passing the info > right? I don't think backends should have to know which marks they > aren't supposed to know about. Right. How about this? 2001-09-14 Simon Josefsson <jas@extundo.com> * gnus-sum.el (gnus-article-unpropagatable-p): New function. (gnus-update-marks): Use it. (gnus-update-marks): Use `gnus-article-mark-to-type' instead of hardcoded list. * gnus.el (gnus-article-special-mark-lists): Add killed. (gnus-article-unpropagated-mark-lists): New constant. Index: gnus.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/gnus.el,v retrieving revision 6.54 diff -u -r6.54 gnus.el --- gnus.el 2001/08/23 18:56:59 6.54 +++ gnus.el 2001/09/14 17:31:21 @@ -1800,8 +1800,19 @@ (defconst gnus-article-special-mark-lists '((seen range) + (killed range) (bookmark tuple) (score tuple))) + +;; Propagate flags to server, with the following exceptions: +;; `cache' is a internal gnus flag +;; `download' is a agent flag +;; `unsend' are for nndraft groups only +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend) + "Marks that shouldn't be propagated to backends. +Typical marks are those that make no sense in a standalone backend, +such as marks that says if a article is stored in the cache or not +(which doesn't make sense in a standalone backend).") (defvar gnus-headers-retrieved-by nil) (defvar gnus-article-reply nil) Index: gnus-sum.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/gnus-sum.el,v retrieving revision 6.106 diff -u -r6.106 gnus-sum.el --- gnus-sum.el 2001/09/10 09:29:50 6.106 +++ gnus-sum.el 2001/09/14 17:31:23 @@ -4811,6 +4811,10 @@ (or (cadr (assq mark gnus-article-special-mark-lists)) 'list)) +(defun gnus-article-unpropagatable-p (mark) + "Return whether MARK should be propagated to backend." + (memq mark gnus-article-unpropagated-mark-lists)) + (defun gnus-adjust-marked-articles (info) "Set all article lists and remove all marks that are no longer valid." (let* ((marked-lists (gnus-info-marks info)) @@ -4870,7 +4874,6 @@ "Enter the various lists of marked articles into the newsgroup info list." (let ((types gnus-article-mark-lists) (info (gnus-get-info gnus-newsgroup-name)) - (uncompressed '(score bookmark killed seen)) type list newmarked symbol delta-marks) (when info ;; Add all marks lists to the list of marks lists. @@ -4902,27 +4905,20 @@ (gnus-add-to-range list gnus-newsgroup-unseen) (gnus-compress-sequence gnus-newsgroup-articles)))) - (unless (memq (cdr type) uncompressed) + (when (eq (gnus-article-mark-to-type (cdr type)) 'list) (setq list (gnus-compress-sequence (set symbol (sort list '<)) t))) - (when (gnus-check-backend-function - 'request-set-mark gnus-newsgroup-name) - ;; propagate flags to server, with the following exceptions: - ;; uncompressed:s are not proper flags (they are cons cells) - ;; cache is a internal gnus flag - ;; download are local to one gnus installation (well) - ;; unsend are for nndraft groups only - ;; xxx: generality of this? this suits nnimap anyway - (unless (memq (cdr type) (append '(cache download unsend) - uncompressed)) - (let* ((old (cdr (assq (cdr type) (gnus-info-marks info)))) - (del (gnus-remove-from-range (gnus-copy-sequence old) list)) - (add (gnus-remove-from-range - (gnus-copy-sequence list) old))) - (when add - (push (list add 'add (list (cdr type))) delta-marks)) - (when del - (push (list del 'del (list (cdr type))) delta-marks))))) + (when (and (gnus-check-backend-function + 'request-set-mark gnus-newsgroup-name) + (not (gnus-article-unpropagatable-p (cdr type)))) + (let* ((old (cdr (assq (cdr type) (gnus-info-marks info)))) + (del (gnus-remove-from-range (gnus-copy-sequence old) list)) + (add (gnus-remove-from-range + (gnus-copy-sequence list) old))) + (when add + (push (list add 'add (list (cdr type))) delta-marks)) + (when del + (push (list del 'del (list (cdr type))) delta-marks)))) (when list (push (cons (cdr type) list) newmarked))) ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-14 17:32 ` Simon Josefsson @ 2001-09-14 18:08 ` Paul Jarc 2001-12-30 3:52 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 39+ messages in thread From: Paul Jarc @ 2001-09-14 18:08 UTC (permalink / raw) Simon Josefsson <jas@extundo.com> wrote: > prj@po.cwru.edu (Paul Jarc) writes: >> I think (gnus-compress-sequence (gnus-uncompress-range range)) would >> do it. How is 'seen special? > > It is a range instead of a list, which most other marks is. They all look like lists of ranges to me, at least in .newsrc.eld. Does Gnus still use uncompressed lists because there are backends that can't handle rages yet? > Where should the snippet you suggest be put? In nn*-request-set-mark, you could calculate the new range, apply the above to it, and store the result. But it would probably be significantly faster to just manipulate the list of ranges by looking for adjacent/overlapping ranges, without recompressing. > +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend) > + "Marks that shouldn't be propagated to backends. > +Typical marks are those that make no sense in a standalone backend, > +such as marks that says if a article is stored in the cache or not Better, I think: such as a mark that says whether an article is stored in the cache Are the unpropagatable marks stored in .newsrc.eld? If they're always recalculated, it seems they shouldn't be. Otherwise it seems they should be, but then they should also be stored in the backend. paul ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-14 18:08 ` Paul Jarc @ 2001-12-30 3:52 ` Lars Magne Ingebrigtsen 2001-12-31 10:02 ` Paul Jarc 0 siblings, 1 reply; 39+ messages in thread From: Lars Magne Ingebrigtsen @ 2001-12-30 3:52 UTC (permalink / raw) prj@po.cwru.edu (Paul Jarc) writes: >>> I think (gnus-compress-sequence (gnus-uncompress-range range)) would >>> do it. How is 'seen special? >> >> It is a range instead of a list, which most other marks is. > > They all look like lists of ranges to me, at least in .newsrc.eld. > Does Gnus still use uncompressed lists because there are backends that > can't handle rages yet? `seen' is special because it's the only marks lists that's never uncompressed. All the other ones -- `tick', `dormant', etc -- are uncompressed on group entry and then compressed on group exit. `seen' is not, and that's because it's assumed that `seen' will grow bounded only by the size of the group, while the others will have a more moderate growth pattern. `seen' will typically be something like '(1 . 45787839)' in a suitably active group. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-12-30 3:52 ` Lars Magne Ingebrigtsen @ 2001-12-31 10:02 ` Paul Jarc 2001-12-31 10:07 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 39+ messages in thread From: Paul Jarc @ 2001-12-31 10:02 UTC (permalink / raw) Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: >>>> I think (gnus-compress-sequence (gnus-uncompress-range range)) would >>>> do it. How is 'seen special? ... > `seen' is special because it's the only marks lists that's never > uncompressed. All the other ones -- `tick', `dormant', etc -- are > uncompressed on group entry and then compressed on group exit. `seen' > is not, and that's because it's assumed that `seen' will grow bounded > only by the size of the group, while the others will have a more > moderate growth pattern. Well, then, there could be a function that recompresses a range by looking for consecutive entries, rather than by expanding and unexpanding. That could be applied to 'seen. Right? paul ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-12-31 10:02 ` Paul Jarc @ 2001-12-31 10:07 ` Lars Magne Ingebrigtsen 2001-12-31 10:28 ` Paul Jarc 0 siblings, 1 reply; 39+ messages in thread From: Lars Magne Ingebrigtsen @ 2001-12-31 10:07 UTC (permalink / raw) prj@po.cwru.edu (Paul Jarc) writes: > Well, then, there could be a function that recompresses a range by > looking for consecutive entries, rather than by expanding and > unexpanding. That could be applied to 'seen. Right? There are functions to merge ranges without uncompressing them, if that's what you mean. But there's no problem in the `seen' handling (except the .marks stuff), I think. I was just explaining why they had to be treated specially. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-12-31 10:07 ` Lars Magne Ingebrigtsen @ 2001-12-31 10:28 ` Paul Jarc 2001-12-31 10:34 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 39+ messages in thread From: Paul Jarc @ 2001-12-31 10:28 UTC (permalink / raw) Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: > prj@po.cwru.edu (Paul Jarc) writes: >> Well, then, there could be a function that recompresses a range by >> looking for consecutive entries, rather than by expanding and >> unexpanding. That could be applied to 'seen. Right? > > There are functions to merge ranges without uncompressing them, if > that's what you mean. I don't think it is. I'm thinking of a function which takes a single range list and returns the smallest range list representing the same set of numbers, and which doesn't eat as much memory as (gnus-compress-sequence (gnus-uncompress-range range)). > But there's no problem in the `seen' handling (except the .marks > stuff), I think. I was just explaining why they had to be treated > specially. Right, that's the problem this function would help solve. paul ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-12-31 10:28 ` Paul Jarc @ 2001-12-31 10:34 ` Lars Magne Ingebrigtsen 2001-12-31 10:56 ` Paul Jarc 0 siblings, 1 reply; 39+ messages in thread From: Lars Magne Ingebrigtsen @ 2001-12-31 10:34 UTC (permalink / raw) prj@po.cwru.edu (Paul Jarc) writes: > I don't think it is. I'm thinking of a function which takes a single > range list and returns the smallest range list representing the same > set of numbers, and which doesn't eat as much memory as > (gnus-compress-sequence (gnus-uncompress-range range)). That's true. But why would you need something like that? -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-12-31 10:34 ` Lars Magne Ingebrigtsen @ 2001-12-31 10:56 ` Paul Jarc 2001-12-31 10:59 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 39+ messages in thread From: Paul Jarc @ 2001-12-31 10:56 UTC (permalink / raw) Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: > prj@po.cwru.edu (Paul Jarc) writes: >> I'm thinking of a function which takes a single range list and >> returns the smallest range list representing the same set of >> numbers, and which doesn't eat as much memory as >> (gnus-compress-sequence (gnus-uncompress-range range)). > > That's true. But why would you need something like that? To make the 'seen range list smaller - in .newsrc.eld, in nnml and nnfolder's mark storage, and in memory. There's wasted space there. paul ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-12-31 10:56 ` Paul Jarc @ 2001-12-31 10:59 ` Lars Magne Ingebrigtsen 2001-12-31 11:20 ` Paul Jarc 0 siblings, 1 reply; 39+ messages in thread From: Lars Magne Ingebrigtsen @ 2001-12-31 10:59 UTC (permalink / raw) prj@po.cwru.edu (Paul Jarc) writes: > To make the 'seen range list smaller - in .newsrc.eld, in nnml and > nnfolder's mark storage, and in memory. There's wasted space there. When are these non-optimal ranges created? If they're not optimal, the range functions are buggy. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-12-31 10:59 ` Lars Magne Ingebrigtsen @ 2001-12-31 11:20 ` Paul Jarc 0 siblings, 0 replies; 39+ messages in thread From: Paul Jarc @ 2001-12-31 11:20 UTC (permalink / raw) Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: > prj@po.cwru.edu (Paul Jarc) writes: >> To make the 'seen range list smaller - in .newsrc.eld, in nnml and >> nnfolder's mark storage, and in memory. There's wasted space there. > > When are these non-optimal ranges created? If they're not optimal, > the range functions are buggy. I don't know, but that's the problem that was reported in this thread 3 months ago, as well as more recently in the thread "\"seen\" marks create adjacent lists". Maybe it's fixed now. The recompression might still be useful to fix old range lists left over from when the range functions were buggy. Or not. paul ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-14 13:43 ` Simon Josefsson 2001-09-14 14:51 ` Paul Jarc @ 2001-09-14 18:11 ` Dan Christensen 2001-09-14 18:23 ` Dan Christensen 1 sibling, 1 reply; 39+ messages in thread From: Dan Christensen @ 2001-09-14 18:11 UTC (permalink / raw) Simon Josefsson <jas@extundo.com> writes: > Dan Christensen <jdc+news@uwo.ca> writes: > >> Hmm, looking further, it is the score entries that are taking up 99% >> of the .mrk file > I'm not sure score marks should be in the file at all. Isn't the > scores calculated each time you enter the group? Maybe the score is > cached in .newsrc/.mark? I have gnus-save-score set to t, in the hope that this speeds up summary generation. But maybe it doesn't. In any case, this feature should only store scores for articles that exist: >> and most of them don't correspond to articles that >> exist in the group! Shouldn't score cache entries be removed when >> the article is deleted from the group (e.g. by the expiry mechanism)? >> I smell a bug. -- Dan Christensen jdc+news@uwo.ca ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-14 18:11 ` Dan Christensen @ 2001-09-14 18:23 ` Dan Christensen 2001-09-20 18:38 ` Simon Josefsson 0 siblings, 1 reply; 39+ messages in thread From: Dan Christensen @ 2001-09-14 18:23 UTC (permalink / raw) Dan Christensen <jdc+news@uwo.ca> writes: > Simon Josefsson <jas@extundo.com> writes: > >> Dan Christensen <jdc+news@uwo.ca> writes: >> >>> Hmm, looking further, it is the score entries that are taking up 99% >>> of the .mrk file > >> I'm not sure score marks should be in the file at all. Isn't the >> scores calculated each time you enter the group? Maybe the score is >> cached in .newsrc/.mark? > > I have gnus-save-score set to t, in the hope that this speeds up > summary generation. I now have gnus-save-score set to nil. After entering and exiting a group, the massive (score ...) entries are removed from that group's list in .newsrc.eld, but they remain in that group's .mrk file. Is this a bug in the .mrk code? By the way, is the eventual goal of the .mrk files to remove this information from .newsrc.eld? I can see benefits to this, like faster performance because only the marks relevant to the groups you access get read from disk, etc. Or will the .mrk information always be duplicated? Dan -- Dan Christensen jdc+news@uwo.ca ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-14 18:23 ` Dan Christensen @ 2001-09-20 18:38 ` Simon Josefsson 2001-09-21 15:00 ` Dan Christensen 0 siblings, 1 reply; 39+ messages in thread From: Simon Josefsson @ 2001-09-20 18:38 UTC (permalink / raw) Dan Christensen <jdc+news@uwo.ca> writes: > I now have gnus-save-score set to nil. After entering and exiting > a group, the massive (score ...) entries are removed from that > group's list in .newsrc.eld, but they remain in that group's .mrk > file. Is this a bug in the .mrk code? Possibly. Did you try my patch in <iluheu5adba.fsf@barbar.josefsson.org>? I think it keeps `score' out of .mrk. > By the way, is the eventual goal of the .mrk files to remove this > information from .newsrc.eld? I can see benefits to this, like > faster performance because only the marks relevant to the groups > you access get read from disk, etc. As a long term goal, yes. Using Gnus with nnml or IMAP without .newsrc.eld would be nice, but backwards compatibility is probably nicer so I don't expect it will be possibly soon. ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-20 18:38 ` Simon Josefsson @ 2001-09-21 15:00 ` Dan Christensen 2001-09-21 16:23 ` Simon Josefsson 0 siblings, 1 reply; 39+ messages in thread From: Dan Christensen @ 2001-09-21 15:00 UTC (permalink / raw) Simon Josefsson <jas@extundo.com> writes: > Dan Christensen <jdc+news@uwo.ca> writes: > >> I now have gnus-save-score set to nil. After entering and exiting >> a group, the massive (score ...) entries are removed from that >> group's list in .newsrc.eld, but they remain in that group's .mrk >> file. Is this a bug in the .mrk code? > > Possibly. Did you try my patch in > <iluheu5adba.fsf@barbar.josefsson.org>? I think it keeps `score' out > of .mrk. It doesn't work. Score entries remain in the .mrk file. Even worse, they are changed to be incorrect. For example, the score pair (579 . 2000) was changed to (579 . 1092) (and lots of other 2000's were changed to 1092's as well). If I delete the .mrk file and set gnus-save-score to nil, then no score entries reappear. But if I set gnus-save-score to t, they appear again in the newly created .mrk file. While I'm writing, let me mention again the two other bugs: - seen entries aren't as compressed as they could be - score entries aren't purged when the article they refer to is deleted/expired -- Dan Christensen jdc+news@uwo.ca ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-21 15:00 ` Dan Christensen @ 2001-09-21 16:23 ` Simon Josefsson 2001-09-22 15:29 ` Dan Christensen 0 siblings, 1 reply; 39+ messages in thread From: Simon Josefsson @ 2001-09-21 16:23 UTC (permalink / raw) Cc: ding On Fri, 21 Sep 2001, Dan Christensen wrote: > Simon Josefsson <jas@extundo.com> writes: > > > Dan Christensen <jdc+news@uwo.ca> writes: > > > >> I now have gnus-save-score set to nil. After entering and exiting > >> a group, the massive (score ...) entries are removed from that > >> group's list in .newsrc.eld, but they remain in that group's .mrk > >> file. Is this a bug in the .mrk code? > > > > Possibly. Did you try my patch in > > <iluheu5adba.fsf@barbar.josefsson.org>? I think it keeps `score' out > > of .mrk. > > It doesn't work. Score entries remain in the .mrk file. Oops, I just found a problem with the patch. Please change +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend) into +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend score) and hopefully score doesn't end up in .mrk. > Even worse, they are changed to be incorrect. For example, the score pair > (579 . 2000) was changed to (579 . 1092) (and lots of other 2000's > were changed to 1092's as well). I suspect Gnus tries to sort the score marks as ranges somehow, which is a bad idea. > While I'm writing, let me mention again the two other bugs: > > - seen entries aren't as compressed as they could be I'll try to understand how seen are supposed to work and fix it. > - score entries aren't purged when the article they refer to is > deleted/expired Once we get the score mark out of the .mrk file, this and your main problem should vanish, I hope. Thanks for the help. ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-21 16:23 ` Simon Josefsson @ 2001-09-22 15:29 ` Dan Christensen 2001-09-22 16:53 ` Simon Josefsson 0 siblings, 1 reply; 39+ messages in thread From: Dan Christensen @ 2001-09-22 15:29 UTC (permalink / raw) Cc: ding Simon Josefsson <jas@extundo.com> writes: > On Fri, 21 Sep 2001, Dan Christensen wrote: > >> Simon Josefsson <jas@extundo.com> writes: >> >> > Possibly. Did you try my patch in >> > <iluheu5adba.fsf@barbar.josefsson.org>? I think it keeps `score' out >> > of .mrk. >> >> It doesn't work. Score entries remain in the .mrk file. > > Oops, I just found a problem with the patch. Please change > > +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend) > > into > > +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend score) Still doesn't work. western.mrk included below. (I made sure to change a mark in this group so the .mrk file got rewritten.) The .mrk file also contains seen entries. If it matters, I'm using nnfolder. More testing... Ok, if I delete the western.mrk file, then no score entries get put there, even if gnus-save-score is set to t. On the other hand, if there are already score entries in the western.mrk file, then they remain there when the western.mrk file gets rewritten, even with the patch applied. In every case I've tried, the seen marks are written to the western.mrk file. >> Even worse, they are changed to be incorrect. For example, the score pair >> (579 . 2000) was changed to (579 . 1092) (and lots of other 2000's >> were changed to 1092's as well). > > I suspect Gnus tries to sort the score marks as ranges somehow, which is a > bad idea. But this only happens in the .mrk file, not in the .newsrc.eld file. (Included file illustrates this.) >> - score entries aren't purged when the article they refer to is >> deleted/expired > > Once we get the score mark out of the .mrk file, this and your main > problem should vanish, I hope. I think this is a separate problem. The score entries aren't expired from .newsrc.eld either. So this bug will still need to be fixed even if score entries don't propagate to the .mrk files. Dan western.mrk: ((tick 282 298 307) (score (82 . 84) (81 . 84) (78 . 84) (79 . 84) (80 . 84) (76 . 84) (77 . 84) (68 . 84) (69 . 84) (70 . 84) (65 . 84) (63 . 84) (62 . 84) (64 . 84) (60 . 84) (58 . 84) (59 . 84) (56 . 84) (44 . 84) (48 . 84) (39 . 84) (37 . 84) (43 . 84) (41 . 84) (40 . 84) (32 . 84) (29 . 84) (45 . 84) (49 . 84) (47 . 84) (38 . 84) (36 . 84) (31 . 84)) (read (1 . 312)) (seen (27 . 45) (47 . 106) (109 . 114) (116 . 274) 280) (forward 268 285 287) (reply 33 37 42 50 55 63 68 (77 . 78) 85 (90 . 91) (101 . 102) 105 (110 . 111) (113 . 114) 117 119 122 (124 . 125) (128 . 131) 143 146 156 166 (176 . 177) (179 . 180) 185 203 208 231 233 238 240 263 (284 . 285) (294 . 295))) western.mrk recreated after deletion: ((read (1 . 312)) (seen (27 . 45) (47 . 106) (109 . 114) (116 . 274) 280 282 298 307) (forward 268 285 287) (reply 33 37 42 50 55 63 68 (77 . 78) 85 (90 . 91) (101 . 102) 105 (110 . 111) (113 . 114) 117 119 122 (124 . 125) (128 . 131) 143 146 156 166 (176 . 177) (179 . 180) 185 203 208 231 233 238 240 263 (284 . 285) (294 . 295)) (tick 282 298 307)) ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-22 15:29 ` Dan Christensen @ 2001-09-22 16:53 ` Simon Josefsson 2001-09-22 21:19 ` Paul Jarc 2001-09-24 17:03 ` Dan Christensen 0 siblings, 2 replies; 39+ messages in thread From: Simon Josefsson @ 2001-09-22 16:53 UTC (permalink / raw) Dan Christensen <jdc+news@uwo.ca> writes: > Simon Josefsson <jas@extundo.com> writes: > >> On Fri, 21 Sep 2001, Dan Christensen wrote: >> >>> Simon Josefsson <jas@extundo.com> writes: >>> >>> > Possibly. Did you try my patch in >>> > <iluheu5adba.fsf@barbar.josefsson.org>? I think it keeps `score' out >>> > of .mrk. >>> >>> It doesn't work. Score entries remain in the .mrk file. >> >> Oops, I just found a problem with the patch. Please change >> >> +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend) >> >> into >> >> +(defconst gnus-article-unpropagated-mark-lists '(cache download unsend score) > > Still doesn't work. western.mrk included below. (I made sure to > change a mark in this group so the .mrk file got rewritten.) The > .mrk file also contains seen entries. If it matters, I'm using > nnfolder. > > More testing... Ok, if I delete the western.mrk file, then no > score entries get put there, even if gnus-save-score is set to t. Ok, good. > On the other hand, if there are already score entries in the > western.mrk file, then they remain there when the western.mrk > file gets rewritten, even with the patch applied. The patches below fixes this by making sure existing marks are removed as well. > In every case I've tried, the seen marks are written to the > western.mrk file. Yup, the patch didn't care about seen. One could probably argue that seen marks should be specific to each Gnus installation -- you could use the seen mark to see which articles YOU have read in a shared mailbox in which some articles may have been marked as read by other people -- so I added `seen' to `gnus-article-unpropagated-mark-lists' as well. >>> Even worse, they are changed to be incorrect. For example, the score pair >>> (579 . 2000) was changed to (579 . 1092) (and lots of other 2000's >>> were changed to 1092's as well). >> >> I suspect Gnus tries to sort the score marks as ranges somehow, which is a >> bad idea. > > But this only happens in the .mrk file, not in the .newsrc.eld file. > (Included file illustrates this.) I think I found the problem, nnml/nnfolder calls some gnus-range functions on those ranges, and since they aren't proper ranges they are messed up. Gnus takes care to not do that for score marks. This is fixed by not storing score in .mrk. >>> - score entries aren't purged when the article they refer to is >>> deleted/expired >> >> Once we get the score mark out of the .mrk file, this and your main >> problem should vanish, I hope. > > I think this is a separate problem. The score entries aren't > expired from .newsrc.eld either. So this bug will still need to > be fixed even if score entries don't propagate to the .mrk files. Yup. Let's take care of it once .mrk works. Ok, here is the complete set of patches that fixes things, include Paul's doc fix. Please test it and I'll commit it. Are there any outstanding issues with the .mrk/.marks after this? How is the performance for you? Profiling results of `nnfolder-request-update-info' and `nnfolder-request-set-mark' for largish groups would be interesting. 2001-09-22 Simon Josefsson <jas@extundo.com> * nnml.el (nnml-open-marks): Remove unpropagatable marks. * nnfolder.el (nnfolder-open-marks): Ditto. * gnus-sum.el (gnus-article-unpropagatable-p): New function. (gnus-update-marks): Use it. (gnus-update-marks): Use `gnus-article-mark-to-type' instead of hardcoded list. * gnus.el (gnus-article-special-mark-lists): Add killed. (gnus-article-unpropagated-mark-lists): New constant. Index: gnus.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/gnus.el,v retrieving revision 6.54 diff -u -r6.54 gnus.el --- gnus.el 2001/08/23 18:56:59 6.54 +++ gnus.el 2001/09/22 16:49:47 @@ -1800,8 +1800,22 @@ (defconst gnus-article-special-mark-lists '((seen range) + (killed range) (bookmark tuple) (score tuple))) + +;; Propagate flags to server, with the following exceptions: +;; `seen' is private to each gnus installation +;; `cache' is a internal gnus flag for each gnus installation +;; `download' is a agent flag private to each gnus installation +;; `unsend' are for nndraft groups only +;; `score' is not a proper mark +(defconst gnus-article-unpropagated-mark-lists + '(seen cache download unsend score) + "Marks that shouldn't be propagated to backends. +Typical marks are those that make no sense in a standalone backend, +such as a mark that says whether an article is stored in the cache +(which doesn't make sense in a standalone backend).") (defvar gnus-headers-retrieved-by nil) (defvar gnus-article-reply nil) Index: gnus-sum.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/gnus-sum.el,v retrieving revision 6.106 diff -u -r6.106 gnus-sum.el --- gnus-sum.el 2001/09/10 09:29:50 6.106 +++ gnus-sum.el 2001/09/22 16:49:48 @@ -4811,6 +4811,10 @@ (or (cadr (assq mark gnus-article-special-mark-lists)) 'list)) +(defun gnus-article-unpropagatable-p (mark) + "Return whether MARK should be propagated to backend." + (memq mark gnus-article-unpropagated-mark-lists)) + (defun gnus-adjust-marked-articles (info) "Set all article lists and remove all marks that are no longer valid." (let* ((marked-lists (gnus-info-marks info)) @@ -4870,7 +4874,6 @@ "Enter the various lists of marked articles into the newsgroup info list." (let ((types gnus-article-mark-lists) (info (gnus-get-info gnus-newsgroup-name)) - (uncompressed '(score bookmark killed seen)) type list newmarked symbol delta-marks) (when info ;; Add all marks lists to the list of marks lists. @@ -4902,27 +4905,20 @@ (gnus-add-to-range list gnus-newsgroup-unseen) (gnus-compress-sequence gnus-newsgroup-articles)))) - (unless (memq (cdr type) uncompressed) + (when (eq (gnus-article-mark-to-type (cdr type)) 'list) (setq list (gnus-compress-sequence (set symbol (sort list '<)) t))) - (when (gnus-check-backend-function - 'request-set-mark gnus-newsgroup-name) - ;; propagate flags to server, with the following exceptions: - ;; uncompressed:s are not proper flags (they are cons cells) - ;; cache is a internal gnus flag - ;; download are local to one gnus installation (well) - ;; unsend are for nndraft groups only - ;; xxx: generality of this? this suits nnimap anyway - (unless (memq (cdr type) (append '(cache download unsend) - uncompressed)) - (let* ((old (cdr (assq (cdr type) (gnus-info-marks info)))) - (del (gnus-remove-from-range (gnus-copy-sequence old) list)) - (add (gnus-remove-from-range - (gnus-copy-sequence list) old))) - (when add - (push (list add 'add (list (cdr type))) delta-marks)) - (when del - (push (list del 'del (list (cdr type))) delta-marks))))) + (when (and (gnus-check-backend-function + 'request-set-mark gnus-newsgroup-name) + (not (gnus-article-unpropagatable-p (cdr type)))) + (let* ((old (cdr (assq (cdr type) (gnus-info-marks info)))) + (del (gnus-remove-from-range (gnus-copy-sequence old) list)) + (add (gnus-remove-from-range + (gnus-copy-sequence list) old))) + (when add + (push (list add 'add (list (cdr type))) delta-marks)) + (when del + (push (list del 'del (list (cdr type))) delta-marks)))) (when list (push (cons (cdr type) list) newmarked))) Index: nnml.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/nnml.el,v retrieving revision 6.26 diff -u -r6.26 nnml.el --- nnml.el 2001/09/15 21:18:36 6.26 +++ nnml.el 2001/09/22 16:49:49 @@ -1000,16 +1000,17 @@ nnml-marks-file-name (nnmail-group-pathname group nnml-directory)))) (if (file-exists-p file) - (setq nnml-marks (condition-case err - (with-temp-buffer - (gnus-sethash file - (nth 5 (file-attributes file)) - nnml-marks-modtime) - (nnheader-insert-file-contents file) - (read (current-buffer))) - (error (or (gnus-yes-or-no-p - (format "Error reading nnml marks file %s (%s). Continuing will use marks from .newsrc.eld. Continue? " file err)) - (error "Cannot read nnml marks file %s (%s)" file err))))) + (condition-case err + (with-temp-buffer + (gnus-sethash file (nth 5 (file-attributes file)) + nnml-marks-modtime) + (nnheader-insert-file-contents file) + (setq nnml-marks (read (current-buffer))) + (dolist (el gnus-article-unpropagated-mark-lists) + (setq nnml-marks (gnus-remassoc el nnml-marks)))) + (error (or (gnus-yes-or-no-p + (format "Error reading nnml marks file %s (%s). Continuing will use marks from .newsrc.eld. Continue? " file err)) + (error "Cannot read nnml marks file %s (%s)" file err)))) ;; User didn't have a .marks file. Probably first time ;; user of the .marks stuff. Bootstrap it from .newsrc.eld. (let ((info (gnus-get-info @@ -1019,6 +1020,8 @@ (nnheader-message 7 "Bootstrapping marks for %s..." group) (setq nnml-marks (gnus-info-marks info)) (push (cons 'read (gnus-info-read info)) nnml-marks) + (dolist (el gnus-article-unpropagated-mark-lists) + (setq nnml-marks (gnus-remassoc el nnml-marks))) (nnml-save-marks group server))))) (provide 'nnml) Index: nnfolder.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/nnfolder.el,v retrieving revision 6.22 diff -u -r6.22 nnfolder.el --- nnfolder.el 2001/09/08 20:55:16 6.22 +++ nnfolder.el 2001/09/22 16:49:49 @@ -1224,16 +1224,17 @@ (defun nnfolder-open-marks (group server) (let ((file (nnfolder-group-marks-pathname group))) (if (file-exists-p file) - (setq nnfolder-marks (condition-case err - (with-temp-buffer - (gnus-sethash file - (nth 5 (file-attributes file)) - nnfolder-marks-modtime) - (nnheader-insert-file-contents file) - (read (current-buffer))) - (error (or (gnus-yes-or-no-p - (format "Error reading nnfolder marks file %s (%s). Continuing will use marks from .newsrc.eld. Continue? " file err)) - (error "Cannot read nnfolder marks file %s (%s)" file err))))) + (condition-case err + (with-temp-buffer + (gnus-sethash file (nth 5 (file-attributes file)) + nnfolder-marks-modtime) + (nnheader-insert-file-contents file) + (setq nnfolder-marks (read (current-buffer))) + (dolist (el gnus-article-unpropagated-mark-lists) + (setq nnfolder-marks (gnus-remassoc el nnfolder-marks)))) + (error (or (gnus-yes-or-no-p + (format "Error reading nnfolder marks file %s (%s). Continuing will use marks from .newsrc.eld. Continue? " file err)) + (error "Cannot read nnfolder marks file %s (%s)" file err)))) ;; User didn't have a .marks file. Probably first time ;; user of the .marks stuff. Bootstrap it from .newsrc.eld. (let ((info (gnus-get-info @@ -1243,6 +1244,8 @@ (nnheader-message 7 "Bootstrapping marks for %s..." group) (setq nnfolder-marks (gnus-info-marks info)) (push (cons 'read (gnus-info-read info)) nnfolder-marks) + (dolist (el gnus-article-unpropagated-mark-lists) + (setq nnfolder-marks (gnus-remassoc el nnfolder-marks))) (nnfolder-save-marks group server))))) (provide 'nnfolder) ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-22 16:53 ` Simon Josefsson @ 2001-09-22 21:19 ` Paul Jarc 2001-09-22 22:50 ` Simon Josefsson 2001-09-24 17:03 ` Dan Christensen 1 sibling, 1 reply; 39+ messages in thread From: Paul Jarc @ 2001-09-22 21:19 UTC (permalink / raw) Simon Josefsson <jas@extundo.com> wrote: > One could probably argue that seen marks should be specific to each > Gnus installation -- you could use the seen mark to see which articles > YOU have read in a shared mailbox in which some articles may have been > marked as read by other people -- so I added `seen' to > `gnus-article-unpropagated-mark-lists' as well. I think everything stored in newsrc.eld should also be stored in the backend. Sharing seems to be a storage issue, so if a mailbox is shared, it should be up to the backend to decide whether to share marks as well, and which marks should be shared. nnmaildir allows this by storing all 'read marks in group-dir/.nnmaildir/marks/read, 'expire in .../expire, etc. The group-dir itself is never shared, but the subdirectory containing messages may be a symlink to a shared message directory. If you want to share the 'read mark, you make .../read a symlink to a shared location too. If other backends aren't this flexible, that's a little bit bad, but if Gnus caters to them at the expense of a more flexible backend, I think that's worse. paul ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-22 21:19 ` Paul Jarc @ 2001-09-22 22:50 ` Simon Josefsson 2001-09-23 6:17 ` Paul Jarc 0 siblings, 1 reply; 39+ messages in thread From: Simon Josefsson @ 2001-09-22 22:50 UTC (permalink / raw) prj@po.cwru.edu (Paul Jarc) writes: > Simon Josefsson <jas@extundo.com> wrote: > > One could probably argue that seen marks should be specific to each >> Gnus installation -- you could use the seen mark to see which articles >> YOU have read in a shared mailbox in which some articles may have been >> marked as read by other people -- so I added `seen' to >> `gnus-article-unpropagated-mark-lists' as well. > > I think everything stored in newsrc.eld should also be stored in the > backend. Erm, even e.g. `cache', `download' and `unsend'? I mean, if those marks are stored in the backend and the backend somehow change these marks (e.g. by backup/restore) the marks will be inaccurate and Gnus will break. If an article is cached or not is not something a backend should (or even could!) worry about, only the Gnus Cache stuff should worry about that. > Sharing seems to be a storage issue, so if a mailbox is > shared, it should be up to the backend to decide whether to share > marks as well, and which marks should be shared. The marks I want to prevent being stored in the backend are those that would break Gnus if they are changed at whim by the backend (by e.g. backup/restore). Consider the scenario where you mark an article as cached, take a backup of your nnml server, remove the article from the cache, restore the nnml server. Now Gnus will think the article is cached, but it isn't. The long term solution is of course that the scoring system, Gnus Cache and Gnus Agent keeps track of their internal states themselves, instead of pushing this into the backend which isn't safe. Fortunately, this doesn't seem impossible to achieve. Do you still disagree? ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-22 22:50 ` Simon Josefsson @ 2001-09-23 6:17 ` Paul Jarc 2001-09-23 10:13 ` Simon Josefsson 0 siblings, 1 reply; 39+ messages in thread From: Paul Jarc @ 2001-09-23 6:17 UTC (permalink / raw) Simon Josefsson <jas@extundo.com> wrote: > prj@po.cwru.edu (Paul Jarc) writes: >> I think everything stored in newsrc.eld should also be stored in the >> backend. > > Erm, even e.g. `cache', `download' and `unsend'? Um. Well, everything that would still be accurate if the group were moved to another Gnus. (I think this would include 'seen.) For other things, I wonder if it might make sense to make them not be marks. > I mean, if those marks are stored in the backend and the backend > somehow change these marks (e.g. by backup/restore) the marks will be > inaccurate and Gnus will break. Right. > If an article is cached or not is not something a backend should (or > even could!) worry about, only the Gnus Cache stuff should worry about > that. That's why I wonder if it might be good to, e.g., store all the cache information in one place, separate from marks. > The long term solution is of course that the scoring system, Gnus > Cache and Gnus Agent keeps track of their internal states themselves, > instead of pushing this into the backend which isn't safe. Ah - I guess you have the same idea. Convenient, that. :) paul ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-23 6:17 ` Paul Jarc @ 2001-09-23 10:13 ` Simon Josefsson 2001-09-23 15:13 ` Harry Putnam 2001-09-25 3:25 ` Paul Jarc 0 siblings, 2 replies; 39+ messages in thread From: Simon Josefsson @ 2001-09-23 10:13 UTC (permalink / raw) prj@po.cwru.edu (Paul Jarc) writes: > (I think this would include 'seen.) I don't think so, right now we have three seenness marks: `unread', `recent' and `seen'. Unread and Recent is managed by the backend. Seen is handled by Gnus. Making Seen handled by the backend would make it equivalent to Recent, which seems sort of useless. (And getting rid of Seen in favor of Recent, or vice versa, would exclude some possible uses of the marks.) > For other things, I wonder if it might make sense to make them not > be marks. Yup. ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-23 10:13 ` Simon Josefsson @ 2001-09-23 15:13 ` Harry Putnam 2001-09-23 16:56 ` Nevin Kapur ` (2 more replies) 2001-09-25 3:25 ` Paul Jarc 1 sibling, 3 replies; 39+ messages in thread From: Harry Putnam @ 2001-09-23 15:13 UTC (permalink / raw) Simon Josefsson <jas@extundo.com> writes: > prj@po.cwru.edu (Paul Jarc) writes: > >> (I think this would include 'seen.) > > I don't think so, right now we have three seenness marks: `unread', > `recent' and `seen'. > > Unread and Recent is managed by the backend. Seen is handled by Gnus. > Making Seen handled by the backend would make it equivalent to Recent, > which seems sort of useless. > > (And getting rid of Seen in favor of Recent, or vice versa, would > exclude some possible uses of the marks.) > >> For other things, I wonder if it might make sense to make them not >> be marks. > > Yup. It seems that marks, seen, recent, et all has introduced quite a few problems. For me it has been problems that seem to have solved themselves at least temporarily. Probably heresy but I'm wondering if the value of these things is so great as to merit not just dropping it as a nice idea but not that workable. My understanding is that this started as a way to allow for backing up readedness marks easily. For my usage that is not now nor has it ever been a big issue. When I've had occasion to lose readedness marks it has never been a major calamity. Usually I can reconstruct close enough without too much trouble, or just start over. How serious of a problem is it to simply start over with the marks? About the only ones that has immediate effects is the persistent, dormant and tick mark, which depending on setting might mean a bunch of stuff gets uncached and deleted or unmarked and deleted. And of course this could happen anyway, even with the new files in place. Like maybe a random rogue M-c in group buffer. I've used a backup system *outside* gnus for a long time. It does not backup marks but it does backup messages which is far more important. It seem to me that marks are really not a core need. That is, nothing really bad happens if you loose them. It seems further that the things that do represent significant effort to reconstruct are still no better than ever. That is, the stuff in `G p' settings still resides in .newsrc.eld and is therefore no more backupable than it ever was. (Running for the root cellar....) ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-23 15:13 ` Harry Putnam @ 2001-09-23 16:56 ` Nevin Kapur 2001-09-23 20:46 ` Simon Josefsson 2001-09-23 20:39 ` Simon Josefsson 2001-09-23 22:50 ` Kai Großjohann 2 siblings, 1 reply; 39+ messages in thread From: Nevin Kapur @ 2001-09-23 16:56 UTC (permalink / raw) Harry Putnam <reader@newsguy.com> writes: > Probably heresy but I'm wondering if the value of these things is so > great as to merit not just dropping it as a nice idea but not that > workable. As an end-user, I would like to weigh in in favor of the new marks code. I find the recently introduced Recent mark very useful. Also, the ability to rsync a directory and have one's previous state restored is very handy. Whatever is decided, I hope there is a user-configurable variable that allows one to use the current behavior. -- Nevin Kapur nevin@jhu.edu ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-23 16:56 ` Nevin Kapur @ 2001-09-23 20:46 ` Simon Josefsson 0 siblings, 0 replies; 39+ messages in thread From: Simon Josefsson @ 2001-09-23 20:46 UTC (permalink / raw) Cc: ding Nevin Kapur <nevin@jhu.edu> writes: >> Probably heresy but I'm wondering if the value of these things is so >> great as to merit not just dropping it as a nice idea but not that >> workable. > > As an end-user, I would like to weigh in in favor of the new marks > code. I find the recently introduced Recent mark very useful. Also, > the ability to rsync a directory and have one's previous state > restored is very handy. > > Whatever is decided, I hope there is a user-configurable variable that > allows one to use the current behavior. The `nnml-marks-is-evil' and `nnfolder-marks-is-evil' variables exists, and disabling the nnml/nnfolder marks stuff should be as easy as changing the default value of that variable. You can't disable the recent or seen marks AFAIK though. ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-23 15:13 ` Harry Putnam 2001-09-23 16:56 ` Nevin Kapur @ 2001-09-23 20:39 ` Simon Josefsson 2001-09-23 22:50 ` Kai Großjohann 2 siblings, 0 replies; 39+ messages in thread From: Simon Josefsson @ 2001-09-23 20:39 UTC (permalink / raw) Cc: ding Harry Putnam <reader@newsguy.com> writes: > It seems that marks, seen, recent, et all has introduced quite a few > problems. For me it has been problems that seem to have solved themselves at > least temporarily. > > Probably heresy but I'm wondering if the value of these things is so > great as to merit not just dropping it as a nice idea but not that > workable. If noone feel these changes serve any purpose, I guess we could back them out. But if someone finds them useful, maybe it is nice to keep it and isolate any possible problems. Most features have problems in the beginning, and we are talking about several new features here -- seen, recent, self-contained nnml and self-contained nnfolder are four separate features, with their own separate set of bugs. Thinking of them as one new feature is not fair, I think. > My understanding is that this started as a way to allow for backing up > readedness marks easily. For my usage that is not now nor has it ever > been a big issue. When I've had occasion to lose readedness marks it > has never been a major calamity. Usually I can reconstruct close > enough without too much trouble, or just start over. > > How serious of a problem is it to simply start over with the marks? > About the only ones that has immediate effects is the persistent, > dormant and tick mark, which depending on setting might mean a bunch > of stuff gets uncached and deleted or unmarked and deleted. And of > course this could happen anyway, even with the new files in place. > Like maybe a random rogue M-c in group buffer. > > I've used a backup system *outside* gnus for a long time. It does not > backup marks but it does backup messages which is far more important. > It seem to me that marks are really not a core need. That is, nothing > really bad happens if you loose them. Marks in nnml and nnfolder had this goal, yes. It sounds as if you don't need the feature, and modulo bugs the absence or presence of the feature shouldn't matter then. The seen and recent marks had different goals though. > It seems further that the things that do represent significant effort > to reconstruct are still no better than ever. That is, the stuff in > `G p' settings still resides in .newsrc.eld and is therefore no more > backupable than it ever was. Right, I think we could get to that as well, in time. ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-23 15:13 ` Harry Putnam 2001-09-23 16:56 ` Nevin Kapur 2001-09-23 20:39 ` Simon Josefsson @ 2001-09-23 22:50 ` Kai Großjohann 2 siblings, 0 replies; 39+ messages in thread From: Kai Großjohann @ 2001-09-23 22:50 UTC (permalink / raw) Cc: ding Harry Putnam <reader@newsguy.com> writes: > Probably heresy but I'm wondering if the value of these things is so > great as to merit not just dropping it as a nice idea but not that > workable. I think this marks stuff is important. For example, it would allow you to rsync your ~/Mail directory between two machines with no fuss, even if you otherwise use different servers. I would be happy if an effort was made to replace the .newsrc.eld file with per-server (or per-group, even) files. That would be way cool. kai -- Symbol's function definition is void: signature ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-23 10:13 ` Simon Josefsson 2001-09-23 15:13 ` Harry Putnam @ 2001-09-25 3:25 ` Paul Jarc 2001-09-25 10:04 ` Simon Josefsson 1 sibling, 1 reply; 39+ messages in thread From: Paul Jarc @ 2001-09-25 3:25 UTC (permalink / raw) Simon Josefsson <jas@extundo.com> wrote: > prj@po.cwru.edu (Paul Jarc) writes: >> (I think this would include 'seen.) > > I don't think so, right now we have three seenness marks: `unread', > `recent' and `seen'. > > Unread and Recent is managed by the backend. Seen is handled by Gnus. Currently, yes. I'm not convinced that it needs or ought to be that way. > Making Seen handled by the backend would make it equivalent to Recent, > which seems sort of useless. You're saying that Gnus is capable of storing Seen in a way that preserves the semantic difference from Recent, but backends are not? Maybe that's true for the current version of nnimap (either all marks are shared or all are private, right?), but it can't be generally true: if nothing else, backends could potentially store Seen the same way Gnus does, or nearly the same way. I wouldn't object to hiding Seen from nnimap if nnimap can't handle it properly, but such hiding should not be forced on all backends. The list of hidden marks should be configured at least per-backend, if not per-server or maybe even per-group. The backend interface should not assume limitations in backends' capabilities, and semantic issues should be kept orthogonal to storage issues as much as possible. paul ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-25 3:25 ` Paul Jarc @ 2001-09-25 10:04 ` Simon Josefsson 2001-09-25 15:45 ` Paul Jarc 0 siblings, 1 reply; 39+ messages in thread From: Simon Josefsson @ 2001-09-25 10:04 UTC (permalink / raw) Cc: ding On Mon, 24 Sep 2001, Paul Jarc wrote: > > Making Seen handled by the backend would make it equivalent to Recent, > > which seems sort of useless. > > You're saying that Gnus is capable of storing Seen in a way that > preserves the semantic difference from Recent, but backends are not? Well, sort of. I don't doubt that back ends are _capable_ of storing the mark, but I think back ends should not store marks that don't have any semantic meaning in the back end. Or maybe rather, back ends should not store marks that have a semantic meaning somewhere ELSE. Just like the Gnus Cache or Gnus Agent marks. This is because it prevents backup and stuff like that.. > Maybe that's true for the current version of nnimap (either all marks > are shared or all are private, right?) It depends on the server, I think. Some servers allows sharing marks, and some allows even fine-grained control over what marks are shared using IMAP ACLs, per group. Some doesn't allow non-standard marks at all. And some servers just forgets marks randomly.. > I wouldn't object to hiding Seen from nnimap if nnimap can't handle it > properly, but such hiding should not be forced on all backends. The > list of hidden marks should be configured at least per-backend, if not > per-server or maybe even per-group. The backend interface should not > assume limitations in backends' capabilities, and semantic issues > should be kept orthogonal to storage issues as much as possible. I think I agree -- but I view the Seen mark similar to the Cache or Agent marks, so I don't want the mark in the back end. (Hmm, this new "back end" feels a bit funny..) Ok, so are you suggesting that the Seen mark should be stored in the backend as well? In this case, what makes Seen different from Recent? I.e., why should we have both then? Maybe I'm just missing something. ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-25 10:04 ` Simon Josefsson @ 2001-09-25 15:45 ` Paul Jarc 2001-09-25 16:53 ` Simon Josefsson 0 siblings, 1 reply; 39+ messages in thread From: Paul Jarc @ 2001-09-25 15:45 UTC (permalink / raw) Simon Josefsson <jas@extundo.com> wrote: > I don't doubt that back ends are _capable_ of storing the mark, but > I think back ends should not store marks that don't have any > semantic meaning in the back end. I don't think *any* marks have semantic meaning in the backend. > Ok, so are you suggesting that the Seen mark should be stored in the > backend as well? In this case, what makes Seen different from Recent? I'm not sure I know what the semantic difference between them is. Tell me that, and I'll tell you how that difference can be preserved when storing them in the backend. paul ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-25 15:45 ` Paul Jarc @ 2001-09-25 16:53 ` Simon Josefsson 2001-09-25 17:08 ` Paul Jarc 0 siblings, 1 reply; 39+ messages in thread From: Simon Josefsson @ 2001-09-25 16:53 UTC (permalink / raw) prj@po.cwru.edu (Paul Jarc) writes: >> I don't doubt that back ends are _capable_ of storing the mark, but >> I think back ends should not store marks that don't have any >> semantic meaning in the back end. > > I don't think *any* marks have semantic meaning in the backend. Maybe I chosed a poor word, I meant the marks that the back end sanely can modify. Read, Tick, Answered etc marks all have a well defined meaning and can be modified, without causing harm in other parts of Gnus, by the backend. >> Ok, so are you suggesting that the Seen mark should be stored in the >> backend as well? In this case, what makes Seen different from Recent? > > I'm not sure I know what the semantic difference between them is. > Tell me that, and I'll tell you how that difference can be preserved > when storing them in the backend. As I think it is right now: Seen: Each Gnus instance keeps track of which articles you have seen and displays a `.' secondary mark on "unseen" articles. Recent: The back end keeps track of which articles you haven't seen, and Gnus displays a `N' secondary mark of "recent" articles. Both marks are different from the usual "unread" because the unread mark can be set and cleared by the user. ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-25 16:53 ` Simon Josefsson @ 2001-09-25 17:08 ` Paul Jarc 0 siblings, 0 replies; 39+ messages in thread From: Paul Jarc @ 2001-09-25 17:08 UTC (permalink / raw) Simon Josefsson <jas@extundo.com> wrote: > Seen: Each Gnus instance keeps track of which articles you have seen > and displays a `.' secondary mark on "unseen" articles. > > Recent: The back end keeps track of which articles you haven't seen, > and Gnus displays a `N' secondary mark of "recent" articles. So changes to Seen marks are initiated by Gnus, and changes to Recent marks are initiated by servers (not backends, I think - nnimap just relays the Recent marks it gets from the server, it doesn't track them internally, right?). I think backends just pass marks between servers and Gnus. They don't know where changes are initiated; they just keep everybody synchronized. If backends store Seen, nothing changes - it just makes it easier for users to manipulate those marks from outside Gnus. They wouldn't normally do this, but if they want to, why try to stop them? paul ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-22 16:53 ` Simon Josefsson 2001-09-22 21:19 ` Paul Jarc @ 2001-09-24 17:03 ` Dan Christensen 2001-09-24 17:40 ` Simon Josefsson 1 sibling, 1 reply; 39+ messages in thread From: Dan Christensen @ 2001-09-24 17:03 UTC (permalink / raw) Simon Josefsson <jas@extundo.com> writes: > Ok, here is the complete set of patches that fixes things, include > Paul's doc fix. Please test it and I'll commit it. It works for me. Thanks! > Are there any outstanding issues with the .mrk/.marks after this? Not that I can think of. > How is the performance for you? It seems as fast as usual now. Thanks. > Profiling results of > `nnfolder-request-update-info' and `nnfolder-request-set-mark' for > largish groups would be interesting. How do you want me to call those functions, and from what buffer? Should I elp-instrument-package all of gnus? Dan -- Dan Christensen jdc+news@uwo.ca ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-24 17:03 ` Dan Christensen @ 2001-09-24 17:40 ` Simon Josefsson 2001-09-25 2:28 ` Dan Christensen 0 siblings, 1 reply; 39+ messages in thread From: Simon Josefsson @ 2001-09-24 17:40 UTC (permalink / raw) Dan Christensen <jdc+news@uwo.ca> writes: >> Profiling results of >> `nnfolder-request-update-info' and `nnfolder-request-set-mark' for >> largish groups would be interesting. > > How do you want me to call those functions, and from what buffer? > Should I elp-instrument-package all of gnus? Yes, please elp-instrument-package "gnus" and "nn" and evaluate (setq nnfolder-marks-modtime (gnus-make-hashtable)) to clear the modtime cache. Then M-x elp-reset-all RET, enter a large nnfolder, and M-x elp-results RET. Then change one mark, M-x elp-reset-all RET, quit the group with `q' and M-x elp-results RET. I've committed the patch. ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-24 17:40 ` Simon Josefsson @ 2001-09-25 2:28 ` Dan Christensen 2001-09-25 17:19 ` Simon Josefsson 0 siblings, 1 reply; 39+ messages in thread From: Dan Christensen @ 2001-09-25 2:28 UTC (permalink / raw) [-- Attachment #1: Type: text/plain, Size: 743 bytes --] Simon Josefsson <jas@extundo.com> writes: > Profiling results of > `nnfolder-request-update-info' and `nnfolder-request-set-mark' for > largish groups would be interesting. Ok, attached are the results from a enter and the results from an exit, done as you suggested. One oddity: The first time I hit SPC after elp-reset-all, I get Retrieving newsgroup: nnfolder:geg... Key sequence A P uses invalid prefix characters The second time I hit SPC, Gnus enters the group without a problem. That's why the results show two calls to some functions. So take the elapsed time, not the average time, for these functions. If I do elp-results after the error message, then when I hit SPC a second time, emacs dies! Completely reproducible. [-- Attachment #2: enter.results --] [-- Type: application/octet-stream, Size: 15508 bytes --] Function Name Call Count Elapsed Time Average Time ====================================================== ========== ============ ============ gnus-topic-read-group 2 6.91517 3.457585 gnus-group-read-group 2 6.914893 3.4574465 gnus-summary-read-group 2 6.91401 3.457005 gnus-summary-read-group-1 2 6.913954 3.456977 gnus-summary-prepare 1 3.517201 3.517201 gnus-summary-prepare-threads 1 3.1134560000 3.1134560000 gnus-select-newsgroup 1 2.643356 2.643356 gnus-fetch-headers 1 2.1616239999 2.1616239999 gnus-retrieve-headers 2 1.9280249999 0.9640124999 gnus-get-newsgroup-headers-xover 1 1.194242 1.194242 gnus-summary-from-or-to-or-newsgroups 381 1.0954090000 0.0028750892 gnus-cache-retrieve-headers 1 0.964245 0.964245 nnfolder-retrieve-headers 1 0.963349 0.963349 nnfolder-possibly-change-group 3 0.94385 0.3146166666 nnfolder-read-folder 1 0.922397 0.922397 gnus-split-references 764 0.5518150000 0.0007222709 gnus-correct-substring 45 0.5452049999 0.0121156666 gnus-correct-length 512 0.528213 0.0010316660 nnheader-find-file-noselect 1 0.500325 0.500325 gnus-gather-threads-by-subject 1 0.346348 0.346348 gnus-adjust-marked-articles 1 0.342421 0.342421 gnus-uncompress-range 9 0.304108 0.0337897777 gnus-thread-total-score 199 0.294596 0.0014803819 gnus-user-format-function-t 127 0.2894059999 0.0022787874 gnus-thread-total-score-1 199 0.2866779999 0.0014405929 gnus-dd-mmm 127 0.2491480000 0.0019617952 gnus-simplify-subject-fuzzy 467 0.2024439999 0.0004334989 gnus-summary-limit-children 306 0.2016169999 0.0006588790 gnus-summary-initial-limit 1 0.112051 0.112051 gnus-article-marked-p 69 0.0892289999 0.0012931739 gnus-member-of-range 306 0.0890710000 0.0002910816 gnus-sort-threads-1 71 0.050934 0.0007173802 gnus-sort-threads 1 0.049246 0.049246 gnus-run-hooks 134 0.0483230000 0.0003606194 gnus-char-width 8341 0.0432159999 5.181...e-06 gnus-summary-highlight-line 126 0.040602 0.0003222380 nnfolder-retrieve-headers-with-nov 1 0.039379 0.039379 nnheader-insert-file-contents 1 0.037986 0.037986 gnus-set-work-buffer 485 0.0270190000 5.570...e-05 nnfolder-save-active 1 0.026995 0.026995 nnmail-save-active 1 0.026927 0.026927 nnmail-generate-active 1 0.025707 0.025707 gnus-point-at-eol 1155 0.0210250000 1.820...e-05 gnus-request-group 1 0.020602 0.020602 nnmail-activate 3 0.020492 0.0068306666 nnfolder-request-group 1 0.020346 0.020346 gnus-make-threads 3 0.019247 0.0064156666 nnfolder-request-list 1 0.01885 0.01885 nnmail-get-active 1 0.017745 0.017745 nnmail-parse-active 1 0.017681 0.017681 gnus-message 12 0.0144269999 0.00120225 gnus-put-text-property 509 0.013577 2.667...e-05 gnus-put-text-property-excluding-characters-with-faces 127 0.0127679999 0.0001005354 gnus-thread-sort-by-number 1148 0.0124979999 1.088...e-05 gnus-group-find-parameter 6 0.011593 0.0019321666 gnus-group-topic-parameters 6 0.0101129999 0.0016854999 gnus-parse-format 5 0.009327 0.0018654000 gnus-point-at-bol 631 0.0081069999 1.284...e-05 gnus-update-format-specifications 1 0.007939 0.007939 gnus-topic-hierarchical-parameters 6 0.0077969999 0.0012994999 gnus-parse-complex-format 3 0.0071330000 0.0023776666 gnus-complex-form-to-spec 6 0.006403 0.0010671666 gnus-topic-parent-topic 126 0.0063800000 5.063...e-05 gnus-current-topics 6 0.0052109999 0.0008684999 gnus-parse-simple-format 13 0.0050780000 0.0003906153 gnus-update-summary-mark-positions 1 0.004676 0.004676 gnus-summary-setup-default-charset 1 0.004659 0.004659 gnus-summary-insert-line 1 0.004307 0.004307 gnus-articles-to-read 1 0.004191 0.004191 gnus-thread-loop-p 267 0.0036759999 1.376...e-05 gnus-update-read-articles 1 0.003428 0.003428 gnus-id-to-thread 199 0.0032639999 1.640...e-05 gnus-set-format 2 0.003155 0.0015775 gnus-configure-frame 5 0.002849 0.0005698 gnus-parameter-charset 1 0.002456 0.002456 gnus-remove-from-range 2 0.00228 0.00114 gnus-parameter-ignored-charsets 1 0.002068 0.002068 gnus-group-auto-expirable-p 1 0.002002 0.002002 gnus-configure-windows 1 0.00197 0.00197 gnus-topic-parameters 18 0.0015940000 8.855...e-05 gnus-copy-sequence 31 0.0015260000 4.922...e-05 gnus-sorted-complement 2 0.0015119999 0.0007559999 gnus-topic-find-topology 36 0.001399 3.886...e-05 gnus-range-length 4 0.001287 0.00032175 gnus-sort-gathered-threads 1 0.001242 0.001242 gnus-killed-articles 1 0.001024 0.001024 nnmail-find-file 1 0.000909 0.000909 gnus-set-mode-line 1 0.000891 0.000891 gnus-continuum-version 1 0.000817 0.000817 gnus-set-sorted-intersection 1 0.000743 0.000743 gnus-apply-kill-file 1 0.000715 0.000715 nnfolder-group-pathname 6 0.0006840000 0.0001140000 gnus-group-goto-group 7 0.0005939999 8.485...e-05 nnheader-nov-delete-outside-range 1 0.000582 0.000582 gnus-sorted-intersection 1 0.000581 0.000581 nnheader-init-server-buffer 3 0.000568 0.0001893333 gnus-get-buffer-create 8 0.000562 7.025e-05 gnus-set-global-variables 3 0.000543 0.0001809999 gnus-summary-make-local-variables 2 0.000539 0.0002695 gnus-current-topic 6 0.000529 8.816...e-05 gnus-cache-articles-in-group 1 0.000502 0.000502 nnheader-find-nov-line 2 0.000487 0.0002435 gnus-newsgroup-kill-file 2 0.000411 0.0002055 gnus-summary-set-display-table 1 0.000392 0.000392 gnus-group-decoded-name 4 0.0003590000 8.975...e-05 gnus-find-method-for-group 10 0.000355 3.55e-05 gnus-group-topic-p 2 0.000332 0.000166 gnus-cache-file-name 1 0.000323 0.000323 nnfolder-group-nov-pathname 2 0.000315 0.0001575 gnus-list-of-unread-articles 1 0.000311 0.000311 nnfolder-server-opened 3 0.000292 9.733...e-05 gnus-last-element 1 0.000276 0.000276 gnus-short-group-name 1 0.00027 0.00027 gnus-get-unread-articles-in-group 1 0.000257 0.000257 gnus-group-topic-name 2 0.0002510000 0.0001255000 gnus-check-server 1 0.000234 0.000234 gnus-group-get-parameter 7 0.0002320000 3.314...e-05 gnus-make-hashtable 2 0.000228 0.000114 gnus-all-windows-visible-p 1 0.000205 0.000205 gnus-group-parameter-value 6 0.0002000000 3.333...e-05 gnus-server-opened 1 0.000197 0.000197 gnus-summary-setup-buffer 2 0.000197 9.85e-05 gnus-newsgroup-savable-name 3 0.000195 6.5e-05 nnoo-server-opened 3 0.000174 5.8e-05 nnmail-message-id 1 0.00016 0.00016 gnus-group-group-name 2 0.000146 7.3e-05 gnus-summary-position-point 1 0.000145 0.000145 gnus-summary-make-menu-bar 1 0.00013 0.00013 gnus-group-quit-config 1 0.000122 0.000122 gnus-file-newer-than 1 0.000116 0.000116 gnus-goto-colon 1 0.000109 0.000109 gnus-simplify-mode-line 2 9.2e-05 4.6e-05 gnus-virtual-group-p 1 8.9e-05 8.9e-05 gnus-turn-off-edit-menu 1 8.7e-05 8.7e-05 nnheader-report 1 7.4e-05 7.4e-05 gnus-set-difference 2 7.099...e-05 3.549...e-05 gnus-article-mark-to-type 6 6.8e-05 1.133...e-05 gnus-parameters-get-parameter 6 6.7e-05 1.116...e-05 gnus-mouse-face-function 3 6.400...e-05 2.133...e-05 gnus-undo-register 1 6.3e-05 6.3e-05 gnus-create-hash-size 2 5.9e-05 2.95e-05 gnus-use-long-file-name 5 5.699...e-05 1.14e-05 gnus-summary-buffer-name 2 5.200...e-05 2.600...e-05 gnus-visual-p 4 5.2e-05 1.3e-05 gnus-tilde-max-form 2 5.1e-05 2.55e-05 gnus-mode-line-buffer-identification 1 4.9e-05 4.9e-05 gnus-window-to-buffer-helper 5 4.8e-05 9.6e-06 nnheader-translate-file-chars 4 4.700...e-05 1.175...e-05 gnus-undo-boundary 4 4.4e-05 1.1e-05 gnus-correct-pad-form 2 3.6e-05 1.8e-05 nnoo-current-server 3 3.1e-05 1.033...e-05 gnus-summary-display-make-predicate 1 2.9e-05 2.9e-05 gnus-cut-threads 1 2.8e-05 2.8e-05 gnus-get-function 1 2.4e-05 2.4e-05 gnus-spec-tab 1 2.4e-05 2.4e-05 gnus-undo-register-1 1 2e-05 2e-05 gnus-make-sort-function 1 1.7e-05 1.7e-05 gnus-server-status 1 1.4e-05 1.4e-05 gnus-add-text-properties 1 1.3e-05 1.3e-05 gnus-windows-old-to-new 1 1.2e-05 1.2e-05 gnus-summary-make-tool-bar 1 1.2e-05 1.2e-05 gnus-server-to-method 1 1.2e-05 1.2e-05 gnus-make-thread-indent-array 1 1.1e-05 1.1e-05 gnus-set-default-directory 1 9e-06 9e-06 gnus-update-missing-marks 1 8e-06 8e-06 gnus-summary-mode 1 0.0 0.0 [-- Attachment #3: exit.results --] [-- Type: application/octet-stream, Size: 10613 bytes --] Function Name Call Count Elapsed Time Average Time ====================================================== ========== ============ ============ gnus-summary-exit 1 0.581116 0.581116 gnus-summary-update-info 1 0.5557259999 0.5557259999 gnus-update-marks 1 0.5105040000 0.5105040000 gnus-check-group 1 0.276002 0.276002 gnus-copy-sequence 961 0.0577880000 6.013...e-05 gnus-remove-from-range 21 0.044493 0.0021187142 gnus-compress-sequence 10 0.030644 0.0030644 gnus-mark-xrefs-as-read 1 0.026937 0.026937 gnus-create-xref-hashtb 1 0.026588 0.026588 gnus-request-set-mark 1 0.015017 0.015017 nnfolder-request-set-mark 1 0.014665 0.014665 gnus-group-update-group 1 0.014194 0.014194 gnus-topic-update-topic-line 3 0.0081170000 0.0027056666 nnfolder-open-marks 1 0.007009 0.007009 nnfolder-save-marks 1 0.006007 0.006007 gnus-prin1-to-string 1 0.004997 0.004997 gnus-topic-update-topics-containing-group 1 0.004803 0.004803 gnus-remassoc 39 0.004116 0.0001055384 gnus-update-read-articles 1 0.003412 0.003412 gnus-group-insert-group-line-info 1 0.002573 0.002573 gnus-add-to-range 1 0.002516 0.002516 gnus-group-insert-group-line 1 0.002429 0.002429 gnus-write-buffer 2 0.002343 0.0011715 gnus-configure-windows 1 0.002065 0.002065 gnus-close-group 1 0.001689 0.001689 nnheader-insert-file-contents 1 0.001559 0.001559 nnfolder-close-group 1 0.001449 0.001449 gnus-deaden-summary 1 0.001065 0.001065 gnus-topic-insert-topic-line 2 0.001058 0.000529 gnus-topic-parent-topic 21 0.001056 5.028...e-05 gnus-run-hooks 5 0.000965 0.000193 gnus-range-add 1 0.000941 0.000941 gnus-prin1 1 0.000874 0.000874 gnus-group-highlight-line 1 0.000836 0.000836 gnus-range-length 3 0.0008060000 0.0002686666 gnus-topic-find-groups 3 0.00076 0.0002533333 gnus-uncompress-range 3 0.0006889999 0.0002296666 gnus-find-method-for-group 21 0.000686 3.266...e-05 gnus-group-set-mode-line 2 0.000611 0.0003055 gnus-dribble-enter 1 0.0006 0.0006 gnus-make-directory 4 0.000496 0.000124 gnus-dead-summary-mode 1 0.000495 0.000495 gnus-group-get-parameter 5 0.0004840000 9.680...e-05 gnus-configure-frame 2 0.000459 0.0002295 gnus-group-jump-to-group 1 0.000455 0.000455 gnus-ephemeral-group-p 3 0.0004310000 0.0001436666 gnus-kill-save-kill-buffer 1 0.000424 0.000424 gnus-add-minor-mode 1 0.000419 0.000419 nnfolder-group-marks-pathname 2 0.000417 0.0002085 gnus-set-global-variables 2 0.000342 0.000171 gnus-group-next-unread-group 1 0.000328 0.000328 nnfolder-group-pathname 2 0.000319 0.0001595 gnus-group-position-point 2 0.000312 0.000156 gnus-cache-possibly-remove-articles 1 0.000282 0.000282 gnus-newsgroup-kill-file 1 0.000278 0.000278 gnus-group-search-forward 1 0.00027 0.00027 gnus-topic-goto-topic 4 0.000265 6.625e-05 gnus-topic-find-topology 6 0.00026 4.333...e-05 gnus-virtual-group-p 3 0.000255 8.5e-05 gnus-get-unread-articles-in-group 1 0.000255 0.000255 gnus-goto-colon 2 0.000241 0.0001205 gnus-group-topic-unread 5 0.000233 4.66e-05 gnus-server-opened 1 0.000226 0.000226 nnfolder-server-opened 2 0.000201 0.0001005 gnus-mode-line-buffer-identification 2 0.000194 9.7e-05 gnus-update-alist-soft 2 0.000166 8.3e-05 gnus-article-mark-to-type 14 0.000142 1.014...e-05 gnus-group-timestamp-delta 1 0.000139 0.000139 gnus-group-parameter-value 5 0.000138 2.76e-05 gnus-point-at-bol 13 0.000134 1.030...e-05 gnus-all-windows-visible-p 1 0.000131 0.000131 gnus-group-group-name 2 0.000129 6.45e-05 gnus-cache-possibly-remove-articles-1 1 0.00012 0.00012 gnus-group-quit-config 1 0.00012 0.00012 nnoo-server-opened 2 0.0001189999 5.949...e-05 gnus-add-text-properties 5 0.0001130000 2.26e-05 nnheader-set-temp-buffer 1 0.000112 0.000112 gnus-group-group-indentation 2 0.0001089999 5.449...e-05 gnus-article-unpropagatable-p 14 0.0001049999 7.499...e-06 gnus-put-text-property-excluding-characters-with-faces 1 9e-05 9e-05 gnus-topic-visible-p 2 8.5e-05 4.25e-05 gnus-group-topic-level 2 8.4e-05 4.2e-05 gnus-cache-fully-p 1 8e-05 8e-05 gnus-get-function 3 7.8e-05 2.6e-05 gnus-group-new-mail 1 7.8e-05 7.8e-05 gnus-newsgroup-savable-name 1 7.3e-05 7.3e-05 gnus-asynchronous-p 1 7.2e-05 7.2e-05 gnus-undo-register 1 6.3e-05 6.3e-05 nnfolder-save-buffer 1 6e-05 6e-05 gnus-put-text-property 3 5.9e-05 1.966...e-05 nnheader-translate-file-chars 6 5.8e-05 9.666...e-06 gnus-group-real-prefix 1 5.6e-05 5.6e-05 gnus-group-goto-group 1 5e-05 5e-05 gnus-server-to-method 3 4.7e-05 1.566...e-05 gnus-point-at-eol 2 4.6e-05 2.3e-05 gnus-delete-alist 2 4.200...e-05 2.100...e-05 gnus-make-hashtable 1 4.1e-05 4.1e-05 gnus-topic-update-unreads 2 4.1e-05 2.05e-05 gnus-window-to-buffer-helper 2 2.2e-05 1.1e-05 gnus-undo-register-1 1 2e-05 2e-05 nnoo-current-server 2 1.999...e-05 9.999...e-06 gnus-server-status 1 1.7e-05 1.7e-05 nnfolder-save-nov 1 1.6e-05 1.6e-05 gnus-get-buffer-create 1 1.4e-05 1.4e-05 gnus-windows-old-to-new 1 1.3e-05 1.3e-05 nnmail-purge-split-history 1 1.3e-05 1.3e-05 gnus-cache-write-active 1 1.2e-05 1.2e-05 gnus-undo-force-boundary 1 1.1e-05 1.1e-05 nnmail-new-mail-p 1 1.1e-05 1.1e-05 gnus-undo-boundary 1 9e-06 9e-06 gnus-use-long-file-name 1 8e-06 8e-06 gnus-extent-start-open 1 7e-06 7e-06 gnus-cache-save-buffers 1 7e-06 7e-06 gnus-group-remove-excess-properties 1 7e-06 7e-06 [-- Attachment #4: Type: text/plain, Size: 37 bytes --] -- Dan Christensen jdc+news@uwo.ca ^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: nnml/nnfolder marks faster 2001-09-25 2:28 ` Dan Christensen @ 2001-09-25 17:19 ` Simon Josefsson 0 siblings, 0 replies; 39+ messages in thread From: Simon Josefsson @ 2001-09-25 17:19 UTC (permalink / raw) Dan Christensen <jdc+news@uwo.ca> writes: >> Profiling results of >> `nnfolder-request-update-info' and `nnfolder-request-set-mark' for >> largish groups would be interesting. > > Ok, attached are the results from a enter and the results from an > exit, done as you suggested. Thanks! It seems the gnus-range functions are the major culprit. nnml/nnfolder-request-set-mark consumed 1-2 % of the time. nn*-request-update-info wasn't called though, maybe it is only called when doing M-g instead of entering the group? Maybe someone could try replacing the gnus-range-* functions with simple list manipulation functions to see how much the range functions really increase performance. They seem to consume quite some time.. Take a backup first, though. :) > One oddity: The first time I hit SPC after elp-reset-all, I get > > Retrieving newsgroup: nnfolder:geg... > Key sequence A P uses invalid prefix characters > > The second time I hit SPC, Gnus enters the group without a problem. Beats me. Anyone? > If I do elp-results after the error message, then when I hit SPC > a second time, emacs dies! Completely reproducible. Perhaps the emacs developers would like a bug report. Assuming the latest pretest exhibits the same problem. ^ permalink raw reply [flat|nested] 39+ messages in thread
end of thread, other threads:[~2001-12-31 11:20 UTC | newest] Thread overview: 39+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2001-09-08 19:38 nnml/nnfolder marks faster Simon Josefsson 2001-09-12 19:42 ` Dan Christensen 2001-09-14 13:43 ` Simon Josefsson 2001-09-14 14:51 ` Paul Jarc 2001-09-14 17:32 ` Simon Josefsson 2001-09-14 18:08 ` Paul Jarc 2001-12-30 3:52 ` Lars Magne Ingebrigtsen 2001-12-31 10:02 ` Paul Jarc 2001-12-31 10:07 ` Lars Magne Ingebrigtsen 2001-12-31 10:28 ` Paul Jarc 2001-12-31 10:34 ` Lars Magne Ingebrigtsen 2001-12-31 10:56 ` Paul Jarc 2001-12-31 10:59 ` Lars Magne Ingebrigtsen 2001-12-31 11:20 ` Paul Jarc 2001-09-14 18:11 ` Dan Christensen 2001-09-14 18:23 ` Dan Christensen 2001-09-20 18:38 ` Simon Josefsson 2001-09-21 15:00 ` Dan Christensen 2001-09-21 16:23 ` Simon Josefsson 2001-09-22 15:29 ` Dan Christensen 2001-09-22 16:53 ` Simon Josefsson 2001-09-22 21:19 ` Paul Jarc 2001-09-22 22:50 ` Simon Josefsson 2001-09-23 6:17 ` Paul Jarc 2001-09-23 10:13 ` Simon Josefsson 2001-09-23 15:13 ` Harry Putnam 2001-09-23 16:56 ` Nevin Kapur 2001-09-23 20:46 ` Simon Josefsson 2001-09-23 20:39 ` Simon Josefsson 2001-09-23 22:50 ` Kai Großjohann 2001-09-25 3:25 ` Paul Jarc 2001-09-25 10:04 ` Simon Josefsson 2001-09-25 15:45 ` Paul Jarc 2001-09-25 16:53 ` Simon Josefsson 2001-09-25 17:08 ` Paul Jarc 2001-09-24 17:03 ` Dan Christensen 2001-09-24 17:40 ` Simon Josefsson 2001-09-25 2:28 ` Dan Christensen 2001-09-25 17:19 ` Simon Josefsson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).