From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/85186 Path: news.gmane.org!not-for-mail From: Ted Zlatanov Newsgroups: gmane.emacs.gnus.general Subject: Re: [PATCH] Two issues with the gnus-registry Date: Mon, 27 Oct 2014 11:03:03 -0400 Organization: =?utf-8?B?0KLQtdC+0LTQvtGAINCX0LvQsNGC0LDQvdC+0LI=?= @ Cienfuegos Message-ID: <87wq7lsggo.fsf@lifelogs.com> References: <87egtx70hy.fsf@ericabrahamsen.net> Reply-To: ding@gnus.org NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: ger.gmane.org 1414422276 30970 80.91.229.3 (27 Oct 2014 15:04:36 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 27 Oct 2014 15:04:36 +0000 (UTC) To: ding@gnus.org Original-X-From: ding-owner+M33430@lists.math.uh.edu Mon Oct 27 16:04:28 2014 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1XilqN-0003dT-Tp for ding-account@gmane.org; Mon, 27 Oct 2014 16:04:28 +0100 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1XilpF-0001uq-Sw; Mon, 27 Oct 2014 10:03:17 -0500 Original-Received: from mx2.math.uh.edu ([129.7.128.33]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1XilpD-0001ud-1Y for ding@lists.math.uh.edu; Mon, 27 Oct 2014 10:03:15 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx2.math.uh.edu with esmtps (TLSv1:AES128-SHA:128) (Exim 4.76) (envelope-from ) id 1XilpA-0000ls-Va for ding@lists.math.uh.edu; Mon, 27 Oct 2014 10:03:14 -0500 Original-Received: from plane.gmane.org ([80.91.229.3]) by quimby.gnus.org with esmtp (Exim 4.80) (envelope-from ) id 1Xilp9-0004jB-16 for ding@gnus.org; Mon, 27 Oct 2014 16:03:11 +0100 Original-Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1Xilp8-0002xU-9E for ding@gnus.org; Mon, 27 Oct 2014 16:03:10 +0100 Original-Received: from c-98-229-61-72.hsd1.ma.comcast.net ([98.229.61.72]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 27 Oct 2014 16:03:10 +0100 Original-Received: from tzz by c-98-229-61-72.hsd1.ma.comcast.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 27 Oct 2014 16:03:10 +0100 X-Injected-Via-Gmane: http://gmane.org/ Mail-Followup-To: ding@gnus.org Original-Lines: 70 Original-X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: c-98-229-61-72.hsd1.ma.comcast.net X-Face: bd.DQ~'29fIs`T_%O%C\g%6jW)yi[zuz6;d4V0`@y-~$#3P_Ng{@m+e4o<4P'#(_GJQ%TT= D}[Ep*b!\e,fBZ'j_+#"Ps?s2!4H2-Y"sx" Mail-Copies-To: never User-Agent: Gnus/5.130012 (Ma Gnus v0.12) Emacs/25.0.50 (gnu/linux) Cancel-Lock: sha1:YKLAvFOxFE/fKSRCCcsqkU9d1LM= X-Spam-Score: -3.5 (---) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:85186 Archived-At: On Fri, 24 Oct 2014 12:04:25 -0700 Eric Abrahamsen wrote: EA> I'm using the gnus-registry in Gnorb to keep track of correspondences EA> between Gnus messages and Org headings, using a key on the registry EA> entries called 'gnorb-ids. This is meant to be a "precious" key, ie EA> entries with this key are not pruned. So far so good: EA> (oref gnus-registry-db :precious) => (gnorb-ids mark) EA> But the entries still do get pruned! I've just had tracked entries start EA> disappearing on me, and realized that they're getting pruned from the EA> registry when I save it. There appear to be two problems here: EA> 1. The pruning doesn't start with the oldest entries first. The EA> docstring of `registry-prune' says it will, but looking over the code in EA> both registry.el and gnus-registry.el, I don't see how that would EA> happen. The entries aren't sorted as they're entered, nor does EA> `gnus-registry-save' pass a sortfun in to `registry-save'. In fact, the EA> entries I'm losing are the most recently-created ones. Huh. I could've sworn this worked. It must be my mistake. EA> 2. The bigger problem is that "precious" entries still seem to get EA> pruned. This is really difficult to edebug, because of the enormous EA> loops involved, and because any time the cursor passes the "db" variable EA> and tries to pretty-print it, Emacs runs out of memory (my registry has EA> about 18,000 entries in it). But something is still wrong here! Yeah, keeping the whole database in a big data structure is pretty bad. Plus I wrote that code in my "loop-happy" period. I'd like to use something better to store these records. EA> I added a sortfun to the `gnus-registry-save' call, but obviously it EA> slows the save process *way* the heck down. I'm not sure what else to EA> do, though, since (as far as I know) hash tables aren't guaranteed to EA> keep their sort order. Is that correct? Right. But you don't need to sort the whole database, only the non-precious entries. So it should be: (prune (sort (select-non-precious))) which in theory should be a fairly constant number (as we keep pruning it). EA> And I really don't know why the precious entries are getting pruned. EA> In a day or so, when I have more time, I'll test with a dummy registry, EA> with max-entries set to 10 and 'creation-time added to the precious EA> entries -- should make debugging easier. The registry has ERT tests, which I thought covered this case. Can you look at `tests/gnustest-registry.el'? As a first step, can you try making tests to demonstrate the problems? EA> Given the commentary in registry.el, I wonder if the whole pruning EA> arrangement actually never quite got finished: EA> ;; The user decides which fields are "precious", F2 for example. At EA> ;; PRUNE TIME (when the :prune-function is called), the registry will EA> ;; trim any entries without the F2 field until the size is :max-soft EA> ;; or less. No entries with the F2 field will be removed at PRUNE EA> ;; TIME. EA> It looks like there's supposed to be a :prune-function slot on registry EA> objects, and presumably the Gnus registry would have a prune function EA> that worked more like what the comments above outline. EA> I'd be happy to help work on this, with a little direction... Assume it's broken. And feel free to propose or make bigger changes as you see fit; there's little in Emacs to support this kind of database so I wrote a lot of it from scratch. Sorry for the trouble. Ted