From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/85187 Path: news.gmane.org!not-for-mail From: Eric Abrahamsen Newsgroups: gmane.emacs.gnus.general Subject: Re: [PATCH] Two issues with the gnus-registry Date: Mon, 27 Oct 2014 12:15:24 -0700 Message-ID: <87zjchxr1v.fsf@ericabrahamsen.net> References: <87egtx70hy.fsf@ericabrahamsen.net> <87wq7lsggo.fsf@lifelogs.com> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: ger.gmane.org 1414437092 6357 80.91.229.3 (27 Oct 2014 19:11:32 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 27 Oct 2014 19:11:32 +0000 (UTC) To: ding@gnus.org Original-X-From: ding-owner+M33431@lists.math.uh.edu Mon Oct 27 20:11:24 2014 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1XiphL-0000pv-FL for ding-account@gmane.org; Mon, 27 Oct 2014 20:11:23 +0100 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1Xipgd-00033P-65; Mon, 27 Oct 2014 14:10:39 -0500 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1Xipgb-00033B-F9 for ding@lists.math.uh.edu; Mon, 27 Oct 2014 14:10:37 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtps (TLSv1:AES128-SHA:128) (Exim 4.76) (envelope-from ) id 1XipgZ-0001KJ-Hn for ding@lists.math.uh.edu; Mon, 27 Oct 2014 14:10:36 -0500 Original-Received: from plane.gmane.org ([80.91.229.3]) by quimby.gnus.org with esmtp (Exim 4.80) (envelope-from ) id 1XipgX-0001yh-Du for ding@gnus.org; Mon, 27 Oct 2014 20:10:33 +0100 Original-Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1XipgR-0000Fu-EE for ding@gnus.org; Mon, 27 Oct 2014 20:10:27 +0100 Original-Received: from c-76-28-195-250.hsd1.wa.comcast.net ([76.28.195.250]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 27 Oct 2014 20:10:27 +0100 Original-Received: from eric by c-76-28-195-250.hsd1.wa.comcast.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 27 Oct 2014 20:10:27 +0100 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 119 Original-X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: c-76-28-195-250.hsd1.wa.comcast.net User-Agent: Gnus/5.130012 (Ma Gnus v0.12) Emacs/25.0.50 (gnu/linux) Cancel-Lock: sha1:jy4/uhvTkmCL0nkRalrbB2FlZiU= X-Spam-Score: -3.5 (---) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:85187 Archived-At: Ted Zlatanov writes: > On Fri, 24 Oct 2014 12:04:25 -0700 Eric Abrahamsen wrote: > > EA> I'm using the gnus-registry in Gnorb to keep track of correspondences > EA> between Gnus messages and Org headings, using a key on the registry > EA> entries called 'gnorb-ids. This is meant to be a "precious" key, ie > EA> entries with this key are not pruned. So far so good: > > EA> (oref gnus-registry-db :precious) => (gnorb-ids mark) > > EA> But the entries still do get pruned! I've just had tracked entries start > EA> disappearing on me, and realized that they're getting pruned from the > EA> registry when I save it. There appear to be two problems here: > > EA> 1. The pruning doesn't start with the oldest entries first. The > EA> docstring of `registry-prune' says it will, but looking over the code in > EA> both registry.el and gnus-registry.el, I don't see how that would > EA> happen. The entries aren't sorted as they're entered, nor does > EA> `gnus-registry-save' pass a sortfun in to `registry-save'. In fact, the > EA> entries I'm losing are the most recently-created ones. > > Huh. I could've sworn this worked. It must be my mistake. > > EA> 2. The bigger problem is that "precious" entries still seem to get > EA> pruned. This is really difficult to edebug, because of the enormous > EA> loops involved, and because any time the cursor passes the "db" variable > EA> and tries to pretty-print it, Emacs runs out of memory (my registry has > EA> about 18,000 entries in it). But something is still wrong here! > > Yeah, keeping the whole database in a big data structure is pretty bad. > Plus I wrote that code in my "loop-happy" period. I'd like to use > something better to store these records. Better than a hash table? Dunno what that would be. But I suppose precious entries could go in one table, and non-precious in another... I don't think the "loop" macro itself is such a problem, but stepping through the main prune or save functions does get cumbersome with large data sets. Eieio is supposed to make edebug use the object-print method when displaying objects, but the line that would do that (eieio.el:895) is currently commented out, I don't know why. If that were working, registries could have their own object-print method, would make edebug much more usable. > EA> I added a sortfun to the `gnus-registry-save' call, but obviously it > EA> slows the save process *way* the heck down. I'm not sure what else to > EA> do, though, since (as far as I know) hash tables aren't guaranteed to > EA> keep their sort order. Is that correct? > > Right. But you don't need to sort the whole database, only the > non-precious entries. So it should be: (prune (sort (select-non-precious))) > which in theory should be a fairly constant number (as we keep pruning it). Right, makes sense. I started on a patch for the pruning process, but realized I was a bit confused by the interaction between max-entries/:max-hard and max-pruned-entries/:max-soft. Just to make sure I get it: The :max-hard limit should *only* come into play when adding new entries: they'll be rejected if the registry is already full. The :max-soft limit should *only* come into play when pruning: we do our best to prune down to :max-soft by deleting non-precious entries. If all entries are precious, we accept a too-large registry. Right now, the :prune-factor is used in conjunction with :max-hard. If the above is correct, that's backwards: it should be used when pruning with :max-soft. Is all that correct? > EA> And I really don't know why the precious entries are getting pruned. > EA> In a day or so, when I have more time, I'll test with a dummy registry, > EA> with max-entries set to 10 and 'creation-time added to the precious > EA> entries -- should make debugging easier. > > The registry has ERT tests, which I thought covered this case. Can you > look at `tests/gnustest-registry.el'? As a first step, can you try > making tests to demonstrate the problems? Will do. > EA> Given the commentary in registry.el, I wonder if the whole pruning > EA> arrangement actually never quite got finished: > > EA> ;; The user decides which fields are "precious", F2 for example. At > EA> ;; PRUNE TIME (when the :prune-function is called), the registry will > EA> ;; trim any entries without the F2 field until the size is :max-soft > EA> ;; or less. No entries with the F2 field will be removed at PRUNE > EA> ;; TIME. > > EA> It looks like there's supposed to be a :prune-function slot on registry > EA> objects, and presumably the Gnus registry would have a prune function > EA> that worked more like what the comments above outline. > > EA> I'd be happy to help work on this, with a little direction... > > Assume it's broken. And feel free to propose or make bigger changes as > you see fit; there's little in Emacs to support this kind of database so > I wrote a lot of it from scratch. Sorry for the trouble. Not at all! It's worked great so far, and is impressive for something done from scratch. My guess is not many people have had a need for the precious functionality, so that hasn't been an issue. It's ideal for use in Gnorb (tracking correspondences between messages and Org headings), I just *really* need precious entries to hang around. I also looked into subclassing the basic registry, but so much necessary functionality (tracking message movement and summary-buffer scanning) is only in gnus-registry, so that wasn't very realistic. At some point (later) it might be worth considering making the gnus-registry an actual subclass of the basic registry, and turning the various action functions, etc, into methods. That's getting ahead of things -- tests first. Thanks, Eric