Trying to prune the registry because it's full; apply: registry max-size limit reached

Trying to prune the registry because it's full; apply: registry max-size limit reached

[Tassilo Horn]

Hi all,

just now when I wanted to enter this group, I got the error message

  Trying to prune the registry because it's full
  apply: registry max-size limit reached

with this backtrace:

--8<---------------cut here---------------start------------->8---
Debugger entered: ((cl-assertion-failed (not (registry-full db)) "registry max-size limit reached"))
  cl--assertion-failed((not (registry-full db)) "registry max-size limit reached" nil nil)
  registry-insert([eieio-class-tag--registry-db "~/.gnus.d/.gnus.registry.eieio" 0.2 10000 0.1...])
  gnus-registry-insert( "<59afc63a-24c7-4b7b-89d8-4b99ba1f3b5b@googlegroups.com>" ((creation-time (21784 58735 12080 293000)) (group) (sender) (subject)))
  gnus-registry-get-or-make-entry("<59afc63a-24c7-4b7b-89d8-4b99ba1f3b5b@googlegroups.com>")
  gnus-registry-get-id-key("<59afc63a-24c7-4b7b-89d8-4b99ba1f3b5b@googlegroups.com>" group)
  gnus-registry-register-message-ids()
  run-hooks(gnus-summary-prepare-hook)
  apply(run-hooks gnus-summary-prepare-hook)
  gnus-run-hooks(gnus-summary-prepare-hook)
  gnus-summary-prepare()
  gnus-summary-read-group-1("nnimap+Fastmail:INBOX.mailinglists.clojure" nil t nil nil nil)
  gnus-summary-read-group("nnimap+Fastmail:INBOX.mailinglists.clojure" nil t nil nil nil nil)
  gnus-group-read-group(nil t)
  gnus-group-select-group(nil)
  gnus-topic-select-group(nil)
  funcall-interactively(gnus-topic-select-group nil)
  call-interactively(gnus-topic-select-group nil nil)
  command-execute(gnus-topic-select-group)
--8<---------------cut here---------------end--------------->8---

My registry-related settings are just

--8<---------------cut here---------------start------------->8---
(setq gnus-registry-ignored-groups '(("^nntp" t) ("^nndraft" t)
                                     ("^nnir" t)
                                     ("training" t) ("Junk" t)
                                     ("Trash" t) ("Spam" t))
      ;; Don't track anything except for the message ids.
      gnus-registry-track-extra nil
      gnus-registry-max-entries 10000)

(gnus-registry-initialize)
--8<---------------cut here---------------end--------------->8---

I had thought that with a maximum number of entries, as soon as I hit
it, the registry would prune the oldest entries to regain some space.
But according to the messages, it seems that pruning didn't succeed.

For the time being, I have increased `gnus-registry-max-entries'.  Is
that a bug?  I mean, given that `gnus-registry-prune-factor' is on its
standard value of 0.1, pruning should have left me with a registry of
9000 entries and then insertion should have worked again, according to
the docs:

,----[ (info "(gnus)Gnus Registry Setup") ]
|  -- Variable: gnus-registry-max-entries
|      The number (an integer or ‘nil’ for unlimited) of entries the
|      registry will keep.  If the registry has reached or exceeded this
|      size, it will reject insertion of new entries.
| 
|  -- Variable: gnus-registry-prune-factor
|      This option (a float between 0 and 1) controls how much the
|      registry is cut back during pruning.  In order to prevent constant
|      pruning, the registry will be pruned back to less than
|      ‘gnus-registry-max-entries’.  This option controls exactly how much
|      less: the target is calculated as the maximum number of entries
|      minus the maximum number times this factor.  The default is 0.1:
|      i.e., if your registry is limited to 50000 entries, pruning will
|      try to cut back to 45000 entries.  Entries with keys marked as
|      precious will not be pruned.
`----

Hm, actually the docs don't really mention that pruning will take place
automatically in case I'm hitting the limit but according to the
message I got before the error, I think it does (and everything else
wouldn't make sense anyway).

Bye,
Tassilo

Re: Trying to prune the registry because it's full; apply: registry max-size limit reached

[Eric Abrahamsen]

Tassilo Horn <tsdh@gnu.org> writes:

> Hi all,
>
> just now when I wanted to enter this group, I got the error message
>
>   Trying to prune the registry because it's full
>   apply: registry max-size limit reached
>
> with this backtrace:
>
> Debugger entered: ((cl-assertion-failed (not (registry-full db)) "registry max-size limit reached"))
>   cl--assertion-failed((not (registry-full db)) "registry max-size limit reached" nil nil)
>   registry-insert([eieio-class-tag--registry-db "~/.gnus.d/.gnus.registry.eieio" 0.2 10000 0.1...])
>   gnus-registry-insert( "<59afc63a-24c7-4b7b-89d8-4b99ba1f3b5b@googlegroups.com>" ((creation-time (21784 58735 12080 293000)) (group) (sender) (subject)))
>   gnus-registry-get-or-make-entry("<59afc63a-24c7-4b7b-89d8-4b99ba1f3b5b@googlegroups.com>")
>   gnus-registry-get-id-key("<59afc63a-24c7-4b7b-89d8-4b99ba1f3b5b@googlegroups.com>" group)
>   gnus-registry-register-message-ids()
>   run-hooks(gnus-summary-prepare-hook)
>   apply(run-hooks gnus-summary-prepare-hook)
>   gnus-run-hooks(gnus-summary-prepare-hook)
>   gnus-summary-prepare()
>   gnus-summary-read-group-1("nnimap+Fastmail:INBOX.mailinglists.clojure" nil t nil nil nil)
>   gnus-summary-read-group("nnimap+Fastmail:INBOX.mailinglists.clojure" nil t nil nil nil nil)
>   gnus-group-read-group(nil t)
>   gnus-group-select-group(nil)
>   gnus-topic-select-group(nil)
>   funcall-interactively(gnus-topic-select-group nil)
>   call-interactively(gnus-topic-select-group nil nil)
>   command-execute(gnus-topic-select-group)
>
>
> My registry-related settings are just
>
> (setq gnus-registry-ignored-groups '(("^nntp" t) ("^nndraft" t)
>                                      ("^nnir" t)
>                                      ("training" t) ("Junk" t)
>                                      ("Trash" t) ("Spam" t))
>       ;; Don't track anything except for the message ids.
>       gnus-registry-track-extra nil
>       gnus-registry-max-entries 10000)
>
> (gnus-registry-initialize)
>
> I had thought that with a maximum number of entries, as soon as I hit
> it, the registry would prune the oldest entries to regain some space.
> But according to the messages, it seems that pruning didn't succeed.

This is related to a change I made recently. What used to happen was --
assuming max entries of 10000 and a prune factor of 0.1 -- every time
the registry exceeded 9000 (not 10000), it would prune. That wasn't what
was supposed to happen: it was supposed to hit 10000 and then prune back
to 9000.

A few days ago I changed it so that it really did reach its max size
before pruning. Obviously something isn't quite aligned right. I'm
assuming some sort of off-by-one error: we're using #'< where we should
be using #'<=, or something like that.

Hang on a second, and I'll test this again...

Eric

Re: Trying to prune the registry because it's full; apply: registry max-size limit reached

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 3059 bytes --]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Tassilo Horn <tsdh@gnu.org> writes:
>
>> Hi all,
>>
>> just now when I wanted to enter this group, I got the error message
>>
>>   Trying to prune the registry because it's full
>>   apply: registry max-size limit reached
>>
>> with this backtrace:
>>
>> Debugger entered: ((cl-assertion-failed (not (registry-full db)) "registry max-size limit reached"))
>>   cl--assertion-failed((not (registry-full db)) "registry max-size limit reached" nil nil)
>>   registry-insert([eieio-class-tag--registry-db "~/.gnus.d/.gnus.registry.eieio" 0.2 10000 0.1...])
>>   gnus-registry-insert( "<59afc63a-24c7-4b7b-89d8-4b99ba1f3b5b@googlegroups.com>" ((creation-time (21784 58735 12080 293000)) (group) (sender) (subject)))
>>   gnus-registry-get-or-make-entry("<59afc63a-24c7-4b7b-89d8-4b99ba1f3b5b@googlegroups.com>")
>>   gnus-registry-get-id-key("<59afc63a-24c7-4b7b-89d8-4b99ba1f3b5b@googlegroups.com>" group)
>>   gnus-registry-register-message-ids()
>>   run-hooks(gnus-summary-prepare-hook)
>>   apply(run-hooks gnus-summary-prepare-hook)
>>   gnus-run-hooks(gnus-summary-prepare-hook)
>>   gnus-summary-prepare()
>>   gnus-summary-read-group-1("nnimap+Fastmail:INBOX.mailinglists.clojure" nil t nil nil nil)
>>   gnus-summary-read-group("nnimap+Fastmail:INBOX.mailinglists.clojure" nil t nil nil nil nil)
>>   gnus-group-read-group(nil t)
>>   gnus-group-select-group(nil)
>>   gnus-topic-select-group(nil)
>>   funcall-interactively(gnus-topic-select-group nil)
>>   call-interactively(gnus-topic-select-group nil nil)
>>   command-execute(gnus-topic-select-group)
>>
>>
>> My registry-related settings are just
>>
>> (setq gnus-registry-ignored-groups '(("^nntp" t) ("^nndraft" t)
>>                                      ("^nnir" t)
>>                                      ("training" t) ("Junk" t)
>>                                      ("Trash" t) ("Spam" t))
>>       ;; Don't track anything except for the message ids.
>>       gnus-registry-track-extra nil
>>       gnus-registry-max-entries 10000)
>>
>> (gnus-registry-initialize)
>>
>> I had thought that with a maximum number of entries, as soon as I hit
>> it, the registry would prune the oldest entries to regain some space.
>> But according to the messages, it seems that pruning didn't succeed.
>
> This is related to a change I made recently. What used to happen was --
> assuming max entries of 10000 and a prune factor of 0.1 -- every time
> the registry exceeded 9000 (not 10000), it would prune. That wasn't what
> was supposed to happen: it was supposed to hit 10000 and then prune back
> to 9000.
>
> A few days ago I changed it so that it really did reach its max size
> before pruning. Obviously something isn't quite aligned right. I'm
> assuming some sort of off-by-one error: we're using #'< where we should
> be using #'<=, or something like that.

Embarrassingly, that is indeed what was happening. Would you mind giving
this patch a whirl? If it works correctly, I'll push it (plus a
ChangeLog notice).

Sorry about that,

Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-registry-pruning-routine.patch --]
[-- Type: text/x-diff, Size: 1098 bytes --]

From 5e9f920ec706393fd9b3a0d260abd13259b26a0d Mon Sep 17 00:00:00 2001
From: Eric Abrahamsen <eric@ericabrahamsen.net>
Date: Mon, 30 Mar 2015 15:12:40 +0800
Subject: [PATCH] Fix registry pruning routine

* lisp/registry.el (registry-prune): Re-use `registry-full' in
  `registry-prune'.  It's a bit of redundant work, but safer.  Also
  ensure that target-size is an integer.
---
 lisp/registry.el | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/lisp/registry.el b/lisp/registry.el
index c8d7d85..551cee8 100644
--- a/lisp/registry.el
+++ b/lisp/registry.el
@@ -361,11 +361,12 @@ from the front of the list are deleted first.
 
 Returns the number of deleted entries."
   (let ((size (registry-size db))
-	(target-size (- (oref db max-size)
-			(* (oref db max-size)
-			   (oref db prune-factor))))
+	(target-size
+	 (round (- (oref db max-size)
+		   (* (oref db max-size)
+		      (oref db prune-factor)))))
 	candidates)
-    (if (> size (oref db max-size))
+    (if (registry-full db)
 	(progn
 	  (setq candidates
 		(registry-collect-prune-candidates
-- 
2.3.4

Re: Trying to prune the registry because it's full; apply: registry max-size limit reached

[Tassilo Horn]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

Hi Eric,

>> A few days ago I changed it so that it really did reach its max size
>> before pruning. Obviously something isn't quite aligned right. I'm
>> assuming some sort of off-by-one error: we're using #'< where we
>> should be using #'<=, or something like that.
>
> Embarrassingly, that is indeed what was happening. Would you mind
> giving this patch a whirl? If it works correctly, I'll push it (plus a
> ChangeLog notice).

I've switched back to 10000 max entries, and now

  (registry-full gnus-registry-db)

returns t.  I can enter groups and read messages without an error but I
don't get any messages about pruning.  But after reading a bunch of new
messages now, (registry-full gnus-registry-db) gives nil and
(registry-size gnus-registry-db) says 9007, so apparently it has pruned
anyhow.

Bye,
Tassilo

Re: Trying to prune the registry because it's full; apply: registry max-size limit reached

[Eric Abrahamsen]

Tassilo Horn <tsdh@gnu.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
> Hi Eric,
>
>>> A few days ago I changed it so that it really did reach its max size
>>> before pruning. Obviously something isn't quite aligned right. I'm
>>> assuming some sort of off-by-one error: we're using #'< where we
>>> should be using #'<=, or something like that.
>>
>> Embarrassingly, that is indeed what was happening. Would you mind
>> giving this patch a whirl? If it works correctly, I'll push it (plus a
>> ChangeLog notice).
>
> I've switched back to 10000 max entries, and now
>
>   (registry-full gnus-registry-db)
>
> returns t.  I can enter groups and read messages without an error but I
> don't get any messages about pruning.  But after reading a bunch of new
> messages now, (registry-full gnus-registry-db) gives nil and
> (registry-size gnus-registry-db) says 9007, so apparently it has pruned
> anyhow.

Okay, thanks. My guess is that the "Trying to prune the registry because
it's full" message is getting buried in other messages -- I don't see
how it could not be called. If you're really idle at the moment, could
you look in *Messages* and confirm that? But if you cut back to 10000
max entries and now have 9007, then things are almost certainly working
correctly.

Later on I'll make a new defvar (something like
`gnus-registry-pruning-functions') where third-party packages can
register pruning functions. That way, if a package registers a
"precious" key that cause entries to hang around, it can also register a
function for cleaning up after itself.

Thanks,
Eric

Re: Trying to prune the registry because it's full; apply: registry max-size limit reached

[Tassilo Horn]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>>> Embarrassingly, that is indeed what was happening. Would you mind
>>> giving this patch a whirl? If it works correctly, I'll push it (plus
>>> a ChangeLog notice).
>>
>> I've switched back to 10000 max entries, and now
>>
>>   (registry-full gnus-registry-db)
>>
>> returns t.  I can enter groups and read messages without an error but I
>> don't get any messages about pruning.  But after reading a bunch of new
>> messages now, (registry-full gnus-registry-db) gives nil and
>> (registry-size gnus-registry-db) says 9007, so apparently it has pruned
>> anyhow.
>
> Okay, thanks. My guess is that the "Trying to prune the registry because
> it's full" message is getting buried in other messages -- I don't see
> how it could not be called.  If you're really idle at the moment, could
> you look in *Messages* and confirm that?

I've been there and searched for "prune" without any matches.

> But if you cut back to 10000 max entries and now have 9007, then
> things are almost certainly working correctly.

Yes, seems so.

> Later on I'll make a new defvar (something like
> `gnus-registry-pruning-functions') where third-party packages can
> register pruning functions. That way, if a package registers a
> "precious" key that cause entries to hang around, it can also register
> a function for cleaning up after itself.

Sounds good.

Bye,
Tassilo

Re: Trying to prune the registry because it's full; apply: registry max-size limit reached

[Ted Zlatanov]

On Mon, 30 Mar 2015 17:45:47 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 

EA> Later on I'll make a new defvar (something like
EA> `gnus-registry-pruning-functions') where third-party packages can
EA> register pruning functions. That way, if a package registers a
EA> "precious" key that cause entries to hang around, it can also register a
EA> function for cleaning up after itself.

Eric, thank you so much for dealing with the pruning stuff and working
on the Gnus registry.

I am not sure if such a facility will be necessary.  What packages would
use it?  Do we have more than one such package?

Ted

Re: Trying to prune the registry because it's full; apply: registry max-size limit reached

[Eric Abrahamsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Mon, 30 Mar 2015 17:45:47 +0800 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 
>
> EA> Later on I'll make a new defvar (something like
> EA> `gnus-registry-pruning-functions') where third-party packages can
> EA> register pruning functions. That way, if a package registers a
> EA> "precious" key that cause entries to hang around, it can also register a
> EA> function for cleaning up after itself.
>
> Eric, thank you so much for dealing with the pruning stuff and working
> on the Gnus registry.
>
> I am not sure if such a facility will be necessary.  What packages would
> use it?  Do we have more than one such package?

Probably not :) It's mostly theoretical at this point. Though it is
entirely possible that precious keys could fill up the registry and
prevent opening of Summary buffers. This could be saved for a (very)
rainy day, or left until someone actually complains about it.

E