Re: nnml/nnfolder marks faster

[Simon Josefsson <jas@extundo.com>]

Dan Christensen <jdc+news@uwo.ca> writes:

> I have a few groups that have really large .mrk files, and this is
> causing slow performance.  I've noticed this more since the new
> .mrk code was introduced, and it is in particular noticeable when
> I *exit* a group with a large .mrk file.  But I don't know if the
> new .mrk code is causing any problems, or whether it just makes
> them more visible.

I guess the latter, I think the problem you mention existed earlier
(because the backend marks code hasn't touched that code).

> As an example, my nnfolder:debian.big group has about 1000 articles
> in it, and the .mrk file is somehow using about 840 bytes per article,
> which seems really excessive:

Yes, too much.

> Here are a few excerpts from the .mrk file which look suspicious:
>
> There are screenfuls and screenfuls of "seen" marks that look like:
>
>   (184755 . 184758) (184759 . 184762)
>
> which should be merged into a single long list I would think.

That is probably better yes, but I don't know the seen mark, it seem
to be handled specially somehow.  Lars?

> Also, the score caching has stuff like:
>
>   (score (185776 . -1) (185820 . -1) (185962 . -1) (185831 . -1)
>   (185991 . -1)
>
> which isn't even stored in numerical order (how is lookup done??).  
> I also wonder if this could be stored in a collapsed form using ranges
>
>   (score ((12345 . 15000) . -1)    
>
> etc. (in cases where there are consecutive articles with the same
> score.)
>
> Hmm, looking further, it is the score entries that are taking up 99%
> of the .mrk file, and most of them don't correspond to articles that
> exist in the group!  Shouldn't score cache entries be removed when
> the article is deleted from the group (e.g. by the expiry mechanism)?
> I smell a bug.

I'm not sure score marks should be in the file at all.  Isn't the
scores calculated each time you enter the group?  Maybe the score is
cached in .newsrc/.mark?

Likewise, `downloaded', `cache' and `unsent' should probably never be
propagated to the backend.