gnus-registry rewrite

gnus-registry rewrite

[Ted Zlatanov]

There is a new gnus-registry.el in the master branch.  The paint is
barely dry.  Here's what's new:

- fewer variables per square inch (many obsoleted)

- all the data management is in registry.el

- EIEIO-based and very fast to save/read the file; now it's in
  ~/.gnus.registry.eieio so you can use your old registry from an older
  Gnus without a problem.

- secondary indices handle everything, including the group for an
  article, for very fast lookups (no more list traversals)

- ERT tests: run "emacs -batch -L . -l ert.el -l gnus-registry.el -f ert-run-tests-batch-and-exit

What's missing: documentation.  So try it out and see what works and
what doesn't; if there are no changes or outstanding bugs in a few days
I'll write the documentation.

Ted

Re: gnus-registry rewrite

[Ted Zlatanov]

On Tue, 05 Apr 2011 11:28:39 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> There is a new gnus-registry.el in the master branch.  The paint is
TZ> barely dry.  Here's what's new:
...
TZ> What's missing: documentation.  So try it out and see what works and
TZ> what doesn't; if there are no changes or outstanding bugs in a few days
TZ> I'll write the documentation.

I can also write a converter from the old registry format to the new if
anyone needs it.

Ted

Re: gnus-registry rewrite

[Kan-Ru Chen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 05 Apr 2011 11:28:39 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 
>
> TZ> There is a new gnus-registry.el in the master branch.  The paint is
> TZ> barely dry.  Here's what's new:
> ...
> TZ> What's missing: documentation.  So try it out and see what works and
> TZ> what doesn't; if there are no changes or outstanding bugs in a few days
> TZ> I'll write the documentation.
>
> I can also write a converter from the old registry format to the new if
> anyone needs it.

It would be great if the old .eld registry can automatically converted
to new eieio format.

Kanru

Re: gnus-registry rewrite

[Ted Zlatanov]

On Wed, 06 Apr 2011 16:10:35 +0800 Kan-Ru Chen <kanru@kanru.info> wrote: 

KC> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I can also write a converter from the old registry format to the new if
>> anyone needs it.

KC> It would be great if the old .eld registry can automatically converted
KC> to new eieio format.

I don't think automatic conversion is needed since the registry data is
mostly volatile.  I added the interactive `gnus-registry-import-eld'
function, go ahead and try it and then `s'ave from Gnus (but back up
your current .eieio registry first :)

Let me know if there are problems with it.  It Works For Me.
Ted

Re: gnus-registry rewrite

[Kan-Ru Chen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 06 Apr 2011 16:10:35 +0800 Kan-Ru Chen <kanru@kanru.info> wrote: 
>
> KC> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> I can also write a converter from the old registry format to the new if
>>> anyone needs it.
>
> KC> It would be great if the old .eld registry can automatically converted
> KC> to new eieio format.
>
> I don't think automatic conversion is needed since the registry data is
> mostly volatile.  I added the interactive `gnus-registry-import-eld'
> function, go ahead and try it and then `s'ave from Gnus (but back up
> your current .eieio registry first :)

65731 records imported. Works great :)

> Let me know if there are problems with it.  It Works For Me.
> Ted

Hum.. eieio is a lot faster than eld when I `s'ave from Gnus. (I
occasionally hit `s' in the Group buffer)

Kanru

Re: gnus-registry rewrite

[Ted Zlatanov]

On Thu, 07 Apr 2011 01:29:41 +0800 Kan-Ru Chen <kanru@kanru.info> wrote: 

KC> Hum.. eieio is a lot faster than eld when I `s'ave from Gnus. (I
KC> occasionally hit `s' in the Group buffer)

Yeah, native hashtables are dumped at the C level.  The old code did a
conversion from hashtable to alist and then reformatted the output so
it's readable, which was painful.

Ted

Re: gnus-registry rewrite

[Ted Zlatanov]

On Sun, 17 Apr 2011 09:22:29 -0400 Andrew Cohen <cohen@andy.bu.edu> wrote: 

AC> I am really liking the speed of the registry rewrite. But I think that I
AC> am doing something wrong, or there is a bug? Two questions:

AC> 2. There used to be a variable that would prevent messages in certain
AC>    groups from ever being entered into the registry. Seems to be gone in
AC>    the new version?

It was on my TODO list but somehow I dropped it.  I just re-added it,
can you try it out?

Ted

Re: gnus-registry rewrite

[Ted Zlatanov]

On Mon, 18 Apr 2011 14:17:04 -0400 Andrew Cohen <cohen@andy.bu.edu> wrote: 

AC> I think I understand the logic in
AC> gnus-registry--split-fancy-with-parent-internal but I'm not 100%
AC> sure. The following changes seem to fix my problem that unfollowed
AC> groups are still followed. Ted, does this look right?

Yes, and thank you for the fixes.  Committed.

Ted

Re: gnus-registry rewrite

[Andrew Cohen]

I think I understand the logic in
gnus-registry--split-fancy-with-parent-internal but I'm not 100%
sure. The following changes seem to fix my problem that unfollowed
groups are still followed. Ted, does this look right?

Best,
Andy

diff --git a/lisp/gnus-registry.el b/lisp/gnus-registry.el
index eab4403..ac06be1 100644
--- a/lisp/gnus-registry.el
+++ b/lisp/gnus-registry.el
@@ -453,7 +453,7 @@ See the Info node `(gnus)Fancy Mail Splitting' for more details."
         (loop for group in (gnus-registry-get-id-key reference 'group)
               when (gnus-registry-follow-group-p group)
               do (gnus-message 7 "%s traced %s to %s" log-agent reference group)
-              do (push group found)))
+               (push group found)))
       ;; filter the found groups and return them
       ;; the found groups are the full groups
       (setq found (gnus-registry-post-process-groups
@@ -478,7 +478,7 @@ See the Info node `(gnus)Fancy Mail Splitting' for more details."
                          (if gnus-registry-track-extra 7 9)
                          "%s (extra tracking) traced subject '%s' to %s"
                          log-agent subject group)
-                     collect group))
+                    and collect group))
          ;; filter the found groups and return them
          ;; the found groups are NOT the full groups
          (setq found (gnus-registry-post-process-groups
@@ -505,7 +505,7 @@ See the Info node `(gnus)Fancy Mail Splitting' for more details."
                          (if gnus-registry-track-extra 7 9)
                          "%s (extra tracking) traced sender '%s' to %s"
                          log-agent sender group)
-                     collect group)))
+                     and collect group)))
 
        ;; filter the found groups and return them
        ;; the found groups are NOT the full groups
@@ -535,7 +535,7 @@ See the Info node `(gnus)Fancy Mail Splitting' for more details."
                              (if gnus-registry-track-extra 7 9)
                              "%s (extra tracking) traced recipient '%s' to %s"
                              log-agent recp group)
-                         collect group)))))
+                        and collect group)))))
 
        ;; filter the found groups and return them
        ;; the found groups are NOT the full groups

gnus-registry rewrite

[Andrew Cohen]

I am really liking the speed of the registry rewrite. But I think that I
am doing something wrong, or there is a bug? Two questions:

1. I have gnus-registry-unfollowed-groups set to a list '("spam"
   "train") yet messages are still being split into these groups. (I
   also have them in nnmail-split-fancy-with-parent-ignore-groups, but
   this doesn't change things). Are there some parens missing around the
   calls to gnus-registry-follow-group-p in various places?

2. There used to be a variable that would prevent messages in certain
   groups from ever being entered into the registry. Seems to be gone in
   the new version?

Best,
Andy