Re: [PATCH] Two issues with the gnus-registry

[Ted Zlatanov <tzz@lifelogs.com>]

On Fri, 24 Oct 2014 12:04:25 -0700 Eric Abrahamsen <eric@ericabrahamsen.net> wrote: 

EA> I'm using the gnus-registry in Gnorb to keep track of correspondences
EA> between Gnus messages and Org headings, using a key on the registry
EA> entries called 'gnorb-ids. This is meant to be a "precious" key, ie
EA> entries with this key are not pruned. So far so good:

EA> (oref gnus-registry-db :precious) => (gnorb-ids mark)

EA> But the entries still do get pruned! I've just had tracked entries start
EA> disappearing on me, and realized that they're getting pruned from the
EA> registry when I save it. There appear to be two problems here:

EA> 1. The pruning doesn't start with the oldest entries first. The
EA> docstring of `registry-prune' says it will, but looking over the code in
EA> both registry.el and gnus-registry.el, I don't see how that would
EA> happen. The entries aren't sorted as they're entered, nor does
EA> `gnus-registry-save' pass a sortfun in to `registry-save'. In fact, the
EA> entries I'm losing are the most recently-created ones.

Huh.  I could've sworn this worked.  It must be my mistake.

EA> 2. The bigger problem is that "precious" entries still seem to get
EA> pruned. This is really difficult to edebug, because of the enormous
EA> loops involved, and because any time the cursor passes the "db" variable
EA> and tries to pretty-print it, Emacs runs out of memory (my registry has
EA> about 18,000 entries in it). But something is still wrong here!

Yeah, keeping the whole database in a big data structure is pretty bad.
Plus I wrote that code in my "loop-happy" period.  I'd like to use
something better to store these records.

EA> I added a sortfun to the `gnus-registry-save' call, but obviously it
EA> slows the save process *way* the heck down. I'm not sure what else to
EA> do, though, since (as far as I know) hash tables aren't guaranteed to
EA> keep their sort order. Is that correct?

Right.  But you don't need to sort the whole database, only the
non-precious entries.  So it should be: (prune (sort (select-non-precious)))
which in theory should be a fairly constant number (as we keep pruning it).

EA> And I really don't know why the precious entries are getting pruned.
EA> In a day or so, when I have more time, I'll test with a dummy registry,
EA> with max-entries set to 10 and 'creation-time added to the precious
EA> entries -- should make debugging easier.

The registry has ERT tests, which I thought covered this case.  Can you
look at `tests/gnustest-registry.el'?  As a first step, can you try
making tests to demonstrate the problems?

EA> Given the commentary in registry.el, I wonder if the whole pruning
EA> arrangement actually never quite got finished:

EA> ;; The user decides which fields are "precious", F2 for example.  At
EA> ;; PRUNE TIME (when the :prune-function is called), the registry will
EA> ;; trim any entries without the F2 field until the size is :max-soft
EA> ;; or less.  No entries with the F2 field will be removed at PRUNE
EA> ;; TIME.

EA> It looks like there's supposed to be a :prune-function slot on registry
EA> objects, and presumably the Gnus registry would have a prune function
EA> that worked more like what the comments above outline.

EA> I'd be happy to help work on this, with a little direction...

Assume it's broken.  And feel free to propose or make bigger changes as
you see fit; there's little in Emacs to support this kind of database so
I wrote a lot of it from scratch.  Sorry for the trouble.

Ted