Re: Some Gnus Registry questions

Re: Some Gnus Registry questions

[Ted Zlatanov]

On Thu, 11 Apr 2013 08:51:39 +0200 Tassilo Horn <tsdh@gnu.org> wrote: 

TH> lately I was wondering why emacs increased from taking up 1.4% of my
TH> memory to nearly 5% just immediately after starting Gnus.  By using
TH> Stefan Monnier's excellent memory-usage.el package (especially
TH> `memory-usage-find-large-variables') I was quickly able to identify that
TH> `gnus-registry-db' was the culprit.

TH> So now I've set `gnus-registry-max-entries' to 10000 (from nil), which
TH> has shrunken its size about a factor 5.  Thereby, some questions
TH> wrt. the registry came to my mind.

TH> 1) What's `gnus-registry-max-pruned-entries' good for?  Of course I've
TH>    read its docs, but still I have no idea.  What is a pruned entry?

We prune entries that are disposable, meaning they don't have any keys
considered "precious."

TH> 2) The default value of `gnus-registry-track-extra' is (subject sender
TH>    recipient).  When looking at the gnus registry eioio file, I can see
TH>    that especially the subject tracking is responsible for a very large
TH>    portion.  Now the question is: do I actually need that?

Probably no, if you never search by subject.  If you split by subject
with the registry, then yes.

TH>    I don't use client-side splitting, so I think the only position where
TH>    I use the registry is for referring articles (^) and gathering
TH>    threads (A T), right?  And since I also use

TH>      (setq gnus-summary-thread-gathering-function
TH>            'gnus-gather-threads-by-references)

TH>    I probably don't need the registry tracking subjects, right?
   
Correct.  I wonder if it could figure this out automatically.

TH>    In the same sense, do I actually need sender and recipient tracking?
TH>    For article referring and thread gathering, I think only the
TH>    Message-Id is needed, right?

Correct.

Ted

Re: Some Gnus Registry questions

[Tassilo Horn]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> TH> 2) The default value of `gnus-registry-track-extra' is (subject sender
> TH>    recipient).  When looking at the gnus registry eioio file, I can see
> TH>    that especially the subject tracking is responsible for a very large
> TH>    portion.  Now the question is: do I actually need that?
>
> Probably no, if you never search by subject.

I sometimes search by subject, but then it's with nnir on nnimap
groups.

> If you split by subject with the registry, then yes.
>
> TH>    I don't use client-side splitting, so I think the only position where
> TH>    I use the registry is for referring articles (^) and gathering
> TH>    threads (A T), right?  And since I also use
>
> TH>      (setq gnus-summary-thread-gathering-function
> TH>            'gnus-gather-threads-by-references)
>
> TH>    I probably don't need the registry tracking subjects, right?
>    
> Correct.  I wonder if it could figure this out automatically.
>
> TH>    In the same sense, do I actually need sender and recipient tracking?
> TH>    For article referring and thread gathering, I think only the
> TH>    Message-Id is needed, right?
>
> Correct.

Thanks for the confirmation!

Bye,
Tassilo

Re: Some Gnus Registry questions

[Tassilo Horn]

Tassilo Horn <tsdh@gnu.org> writes:

>> TH>    I don't use client-side splitting, so I think the only position where
>> TH>    I use the registry is for referring articles (^) and gathering
>> TH>    threads (A T), right?  And since I also use
>>
>> TH>      (setq gnus-summary-thread-gathering-function
>> TH>            'gnus-gather-threads-by-references)
>>
>> TH>    I probably don't need the registry tracking subjects, right?
>>    
>> Correct.  I wonder if it could figure this out automatically.
>>
>> TH>    In the same sense, do I actually need sender and recipient tracking?
>> TH>    For article referring and thread gathering, I think only the
>> TH>    Message-Id is needed, right?
>>
>> Correct.
>
> Thanks for the confirmation!

Ok, so now I've set gnus-registry-track-extra to nil in order to make
the registry only track message ids.  That works fine for new articles,
but is there a way to remove the sender, recipients, and subjects from
old articles that are already contained in the registry?

`gnus-registry-remake-db' looks like it will erase all information, so
I've not tried that so far.

Bye,
Tassilo

Re: Some Gnus Registry questions

[Ted Zlatanov]

On Thu, 13 Jun 2013 08:08:40 +0200 Tassilo Horn <tsdh@gnu.org> wrote: 

TH> Ok, so now I've set gnus-registry-track-extra to nil in order to make
TH> the registry only track message ids.  That works fine for new articles,
TH> but is there a way to remove the sender, recipients, and subjects from
TH> old articles that are already contained in the registry?

TH> `gnus-registry-remake-db' looks like it will erase all information, so
TH> I've not tried that so far.

I have definitely not needed this so far :)

You can probably write it starting with `gnus-registry-remove-ignored'
for the iteration across all articles and for each article, do something
like this (untested):

(defun gnus-registry-remove-id-key (id key)
  (let ((db gnus-registry-db)
        (entry (gnus-registry-get-or-make-entry id)))
    (registry-delete db (list id) nil)
    (setq entry (assq-delete-all key entry))
    (gnus-registry-insert db id entry)
    entry))

where `key' is 'subject for example.  If that works for you, I can add
both a more generic iteration and the remove function above.

Ted

Re: Some Gnus Registry questions

[Tassilo Horn]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 13 Jun 2013 08:08:40 +0200 Tassilo Horn <tsdh@gnu.org> wrote: 
>
> TH> Ok, so now I've set gnus-registry-track-extra to nil in order to make
> TH> the registry only track message ids.  That works fine for new articles,
> TH> but is there a way to remove the sender, recipients, and subjects from
> TH> old articles that are already contained in the registry?
>
> TH> `gnus-registry-remake-db' looks like it will erase all information, so
> TH> I've not tried that so far.
>
> I have definitely not needed this so far :)
>
> You can probably write it starting with `gnus-registry-remove-ignored'
> for the iteration across all articles and for each article, do something
> like this (untested):
>
> (defun gnus-registry-remove-id-key (id key)
>   (let ((db gnus-registry-db)
>         (entry (gnus-registry-get-or-make-entry id)))
>     (registry-delete db (list id) nil)
>     (setq entry (assq-delete-all key entry))
>     (gnus-registry-insert db id entry)
>     entry))
>
> where `key' is 'subject for example.  If that works for you, I can add
> both a more generic iteration and the remove function above.

Thanks.  I've come up with this, and it seems to work:

--8<---------------cut here---------------start------------->8---
(defun gnus-registry-remove-extra-data (extra)
  "Remove tracked EXTRA data from the gnus registry.
EXTRA is a list of symbols.  Valid symbols are those contained in
the docs of `gnus-registry-track-extra'.  This command is useful
if you stop tracking some extra data and now want to purge it
from your existing entries."
  (interactive (list (mapcar 'intern
			     (completing-read-multiple
			      "Extra data: "
			      '("subject" "sender" "recipient")))))
  (when extra
    (let ((db gnus-registry-db))
      (registry-reindex db)
      (loop for k being the hash-keys of (oref db :data)
	    using (hash-value v)
	    do (let ((newv (cl-remove-if #'(lambda (entry)
					     (member (car entry) extra))
					 v)))
		 (registry-delete db (list k) nil)
		 (gnus-registry-insert db k newv)))
      (registry-reindex db))))
--8<---------------cut here---------------end--------------->8---

However, it didn't shrink the size of the registry that much (reduced
the size by about one fourth).

Anyway, I think it's useful.  Should I add it to gnus-registry.el?  (I
guess I'll have to remove the `cl-remove-if' to stay compatible with
older emacsen...)

Bye,
Tassilo

Re: Some Gnus Registry questions

[Ted Zlatanov]

On Wed, 03 Jul 2013 10:07:09 +0200 Tassilo Horn <tsdh@gnu.org> wrote: 

TH> Thanks.  I've come up with this, and it seems to work:
TH> (defun gnus-registry-remove-extra-data (extra)
TH>   "Remove tracked EXTRA data from the gnus registry.
TH> EXTRA is a list of symbols.  Valid symbols are those contained in
TH> the docs of `gnus-registry-track-extra'.  This command is useful
TH> if you stop tracking some extra data and now want to purge it
TH> from your existing entries."
TH>   (interactive (list (mapcar 'intern
TH> 			     (completing-read-multiple
TH> 			      "Extra data: "
TH> 			      '("subject" "sender" "recipient")))))
TH>   (when extra
TH>     (let ((db gnus-registry-db))
TH>       (registry-reindex db)
TH>       (loop for k being the hash-keys of (oref db :data)
TH> 	    using (hash-value v)
TH> 	    do (let ((newv (cl-remove-if #'(lambda (entry)
TH> 					     (member (car entry) extra))
TH> 					 v)))
TH> 		 (registry-delete db (list k) nil)
TH> 		 (gnus-registry-insert db k newv)))
TH>       (registry-reindex db))))

TH> However, it didn't shrink the size of the registry that much (reduced
TH> the size by about one fourth).

TH> Anyway, I think it's useful.  Should I add it to gnus-registry.el?  (I
TH> guess I'll have to remove the `cl-remove-if' to stay compatible with
TH> older emacsen...)

Yes, please, and do what you can for older Emacsen... we can rewrite
that code altogether in a Gnus helper function if necessary.

Ted

Some Gnus Registry questions

[Tassilo Horn]

Hi all,

lately I was wondering why emacs increased from taking up 1.4% of my
memory to nearly 5% just immediately after starting Gnus.  By using
Stefan Monnier's excellent memory-usage.el package (especially
`memory-usage-find-large-variables') I was quickly able to identify that
`gnus-registry-db' was the culprit.

So now I've set `gnus-registry-max-entries' to 10000 (from nil), which
has shrunken its size about a factor 5.  Thereby, some questions
wrt. the registry came to my mind.

1) What's `gnus-registry-max-pruned-entries' good for?  Of course I've
   read its docs, but still I have no idea.  What is a pruned entry?

2) The default value of `gnus-registry-track-extra' is (subject sender
   recipient).  When looking at the gnus registry eioio file, I can see
   that especially the subject tracking is responsible for a very large
   portion.  Now the question is: do I actually need that?

   I don't use client-side splitting, so I think the only position where
   I use the registry is for referring articles (^) and gathering
   threads (A T), right?  And since I also use

     (setq gnus-summary-thread-gathering-function
           'gnus-gather-threads-by-references)

   I probably don't need the registry tracking subjects, right?
   
   In the same sense, do I actually need sender and recipient tracking?
   For article referring and thread gathering, I think only the
   Message-Id is needed, right?

Bye,
Tassilo