Gnus registry upgrade didn't import old registry

Gnus registry upgrade didn't import old registry

[Dan Christensen]

When I started a recent Gnus, it created a new registry, but didn't
slurp in my old registry.  Was it supposed to?

Dan

Re: Gnus registry upgrade didn't import old registry

[Ted Zlatanov]

On Tue, 13 Sep 2011 10:59:29 -0400 Dan Christensen <jdc@uwo.ca> wrote: 

DC> When I started a recent Gnus, it created a new registry, but didn't
DC> slurp in my old registry.  Was it supposed to?

Sorry for the inconvenience.  You have to use `gnus-registry-import-eld'
but it's not called automatically.  The reason is that it doesn't remove
the old file so it could get imported twice.  I'm not sure how to best
deal with this issue...

The good news is, importing won't wipe your current registry.  But
please back up beforehand anyway.

Ted

Re: Gnus registry upgrade didn't import old registry

[Dan Christensen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 13 Sep 2011 10:59:29 -0400 Dan Christensen <jdc@uwo.ca> wrote: 
>
> DC> When I started a recent Gnus, it created a new registry, but didn't
> DC> slurp in my old registry.  Was it supposed to?
>
> Sorry for the inconvenience.  You have to use `gnus-registry-import-eld'
> but it's not called automatically.  The reason is that it doesn't remove
> the old file so it could get imported twice.  I'm not sure how to best
> deal with this issue...

Ah, thanks.

> The good news is, importing won't wipe your current registry.  But
> please back up beforehand anyway.

Ok, it seems to have worked, although it said

  importing: 7700 of 10000 (77.00%)
  Trying to prune the registry because it's full
  registry max-hard size limit reached

part way through and quit.  Do you think it got the most recent part?

Dan

Re: Gnus registry upgrade didn't import old registry

[Dan Christensen]

Oh, is it normal that the .eieio file is a few times bigger than the
.eld file?  For 10000 entries, my .eld file is 1MB and my .eieio file
is 3MB.

Dan

Re: Gnus registry upgrade didn't import old registry

[Dan Christensen]

Hmm, now slurping in new mail doesn't seem to be working:

  Reading incoming mail from file... [8 times]
  Gnus registry: article <CA+SKv+r-ssdb2p06eb4NKKdyeZkO-=NMSDhfqv2TDhUpnSajzw@mail.gmail.com> spooled to nnfolder:picasa-linux
  Trying to prune the registry because it's full
  Mail source (file :path ~/Mail/lists) failed: (error Cannot get new mail)

Here are my settings:

  (setq gnus-registry-max-entries 10000
        gnus-registry-use-long-group-names t
        gnus-registry-ignored-groups '(("nntp" t)
                                       ("nnrss" t)
                                       ("spam" t)
                                       ("train" t))
  )
  
  (gnus-registry-initialize)

I tried increasing max-entries to 12000, but it didn't help.  I'll
restore my registry backup, which I expect will fix things.

Dan

Re: Gnus registry upgrade didn't import old registry

[Ted Zlatanov]

On Mon, 26 Sep 2011 16:41:08 -0400 Dan Christensen <jdc@uwo.ca> wrote: 

DC> Oh, is it normal that the .eieio file is a few times bigger than the
DC> .eld file?  For 10000 entries, my .eld file is 1MB and my .eieio file
DC> is 3MB.

It's all done by EIEIO so I haven't worried about the size.  We store
more information, perhaps that's why.  Is the size the same between the
old and the new registry?

Ted

Re: Gnus registry upgrade didn't import old registry

[Ted Zlatanov]

On Mon, 26 Sep 2011 16:50:01 -0400 Dan Christensen <jdc@uwo.ca> wrote: 

DC> Hmm, now slurping in new mail doesn't seem to be working:
DC>   Reading incoming mail from file... [8 times]
DC>   Gnus registry: article <CA+SKv+r-ssdb2p06eb4NKKdyeZkO-=NMSDhfqv2TDhUpnSajzw@mail.gmail.com> spooled to nnfolder:picasa-linux
DC>   Trying to prune the registry because it's full
DC>   Mail source (file :path ~/Mail/lists) failed: (error Cannot get new mail)

DC> Here are my settings:

DC>   (setq gnus-registry-max-entries 10000
DC>         gnus-registry-use-long-group-names t
DC>         gnus-registry-ignored-groups '(("nntp" t)
DC>                                        ("nnrss" t)
DC>                                        ("spam" t)
DC>                                        ("train" t))
DC>   )
  
DC>   (gnus-registry-initialize)

DC> I tried increasing max-entries to 12000, but it didn't help.  I'll
DC> restore my registry backup, which I expect will fix things.

I haven't run into this issue.  What's the (registry-size
gnus-registry-db)?  Can you e-mail me a copy of the .eieio and .eld
files if they don't have private information?

I have max-entries at 100000, could you try that?  I run up to the limit
often and the prune-factor of 0.1 gets applied.

Ted

Re: Gnus registry upgrade didn't import old registry

[Dan Christensen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Mon, 26 Sep 2011 16:41:08 -0400 Dan Christensen <jdc@uwo.ca> wrote: 
>
> DC> Oh, is it normal that the .eieio file is a few times bigger than the
> DC> .eld file?  For 10000 entries, my .eld file is 1MB and my .eieio file
> DC> is 3MB.
>
> It's all done by EIEIO so I haven't worried about the size.  We store
> more information, perhaps that's why.  Is the size the same between the
> old and the new registry?

I believe I used 10000 in both cases, and that's reinforced by the
message gnus-registry-import-eld gave when reading the old one.

I don't mind that it's bigger, since it saves more quickly.

Dan

Re: Gnus registry upgrade didn't import old registry

[Dan Christensen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I haven't run into this issue.  What's the (registry-size
> gnus-registry-db)?  

(registry-size gnus-registry-db)
10000

> Can you e-mail me a copy of the .eieio and .eld
> files if they don't have private information?

Sure.  Will do in a minute.

> I have max-entries at 100000, could you try that?  

I did

(setq gnus-registry-max-entries 100000)

and still I get the error when trying to import my old .eld file
with 10000 entries.  And still

(registry-size gnus-registry-db)
10000

(Incidentally, with max-entries that high, how big is your .eieie
file??)

Dan

Re: Gnus registry upgrade didn't import old registry

[Dan Christensen]

Dan Christensen <jdc@uwo.ca> writes:

>> I have max-entries at 100000, could you try that?  
>
> I did
>
> (setq gnus-registry-max-entries 100000)
>
> and still I get the error when trying to import my old .eld file
> with 10000 entries.  And still
>
> (registry-size gnus-registry-db)
> 10000

I restarted emacs with the increased value in my .gnus.el, and
then it worked.  But I fear this is just delaying the problem for a
decade or two.  

Here's my guess at the bug:

In gnus-registry.el:

(defun gnus-registry-fixup-registry (db)
  (when db
    (let ((old (oref db :tracked)))
...
      (oset db :max-hard
            (or gnus-registry-max-entries
                most-positive-fixnum))
...
      (oset db :max-soft
            (or gnus-registry-max-pruned-entries
                most-positive-fixnum))

If the user sets gnus-registry-max-entries but not
gnus-registry-max-pruned-entries, then the soft limit might be (much)
higher than the hard limit.  Maybe the last line should be changed to
replace `most-positive-fixnum' with something like `(oget db :max-hard)'
(just guessing at how to read a stored value).

Dan

Re: Gnus registry upgrade didn't import old registry

[Ted Zlatanov]

On Mon, 26 Sep 2011 22:12:23 -0400 Dan Christensen <jdc@uwo.ca> wrote: 

DC> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I have max-entries at 100000, could you try that?  

DC> (Incidentally, with max-entries that high, how big is your .eieie
DC> file??)

54 MB currently.

Ted

Re: Gnus registry upgrade didn't import old registry

[Ted Zlatanov]

On Mon, 26 Sep 2011 22:35:25 -0400 Dan Christensen <jdc@uwo.ca> wrote: 

DC> Here's my guess at the bug:

DC> In gnus-registry.el:

DC> (defun gnus-registry-fixup-registry (db)
DC>   (when db
DC>     (let ((old (oref db :tracked)))
DC> ...
DC>       (oset db :max-hard
DC>             (or gnus-registry-max-entries
DC>                 most-positive-fixnum))
DC> ...
DC>       (oset db :max-soft
DC>             (or gnus-registry-max-pruned-entries
DC>                 most-positive-fixnum))

DC> If the user sets gnus-registry-max-entries but not
DC> gnus-registry-max-pruned-entries, then the soft limit might be (much)
DC> higher than the hard limit.  Maybe the last line should be changed to
DC> replace `most-positive-fixnum' with something like `(oget db :max-hard)'
DC> (just guessing at how to read a stored value).

I have (in set-custom-variables)

'(gnus-registry-ignored-groups (quote (("nntp" t) ("nnrss" t) ("spam" t) ("train" t) ("cron" t))))
'(gnus-registry-install t)
'(gnus-registry-max-entries 100000)
'(gnus-registry-split-strategy (quote majority))
'(gnus-registry-track-extra (quote (sender subject)))
'(gnus-registry-unfollowed-groups (quote ("train$" "delayed$" "drafts$" "queue$" "INBOX$")))

and the pruning works fine for me.  Could you look at what the following
produces?  This should simulate the candidates that `registry-prune'
gets:

  (let* ((db gnus-registry-db)
         (hs (oref db :max-hard))
         (ss (oref db :max-soft)))
    (oset db :max-hard 10000)
    (oset db :max-soft most-positive-fixnum)
    (prog1 (list (registry-prune-soft-candidates db)
                 (registry-prune-hard-candidates db))
           (oset db :max-hard hs)
           (oset db :max-soft ss)))

Thanks
Ted

Re: Gnus registry upgrade didn't import old registry

[Dan Christensen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Could you look at what the following produces?  This should simulate
> the candidates that `registry-prune' gets:
>
>   (let* ((db gnus-registry-db)
>          (hs (oref db :max-hard))
>          (ss (oref db :max-soft)))
>     (oset db :max-hard 10000)
>     (oset db :max-soft most-positive-fixnum)
>     (prog1 (list (registry-prune-soft-candidates db)
>                  (registry-prune-hard-candidates db))
>            (oset db :max-hard hs)
>            (oset db :max-soft ss)))

The output is long and is shown with ...:

((536870911 (nil "<20110913120207.7B6336142@lwahl>" "fake+none+nndraft:drafts+1" "fake+none+nndraft:drafts+2" "<73b0a166382dc.4e6e5c38@uwo.ca>" "<1315880651.2311.22.camel@mike-desktop>" "<4E6F239E.8030506@gmail.com>" "<4E6EAF23.1070406@uwo.ca>" "<LYRIS-60085979-25089418-2011.09.13-02.55.25--jdc#uwo.ca@go.netatlantic.com>" "<2139667233.835036.1315899248272.JavaMail.app@ela4-bed46.prod>" "<4E6F60C1.7090809@wagnerrp.com>" "<p062408a2ca951193efe9@simon.thehobsons.co.uk>" ...)) (8997.2 nil))

But there's a lot in that sub-list starting with nil:

(length (cadar result))
10028

Could it be that during the import-eld process, pruning doesn't take
place?  But why then was I still getting errors when new mail was
arriving?  (And could those be reduced to warnings, so that one can
still receive mail when a problem occurs?)

Dan

Re: Gnus registry upgrade didn't import old registry

[Ted Zlatanov]

On Tue, 27 Sep 2011 22:43:36 -0400 Dan Christensen <jdc@uwo.ca> wrote: 

DC> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Could you look at what the following produces?  This should simulate
>> the candidates that `registry-prune' gets:
>> 
>> (let* ((db gnus-registry-db)
>> (hs (oref db :max-hard))
>> (ss (oref db :max-soft)))
>> (oset db :max-hard 10000)
>> (oset db :max-soft most-positive-fixnum)
>> (prog1 (list (registry-prune-soft-candidates db)
>> (registry-prune-hard-candidates db))
>> (oset db :max-hard hs)
>> (oset db :max-soft ss)))

DC> The output is long and is shown with ...:

DC> ((536870911 (nil "<20110913120207.7B6336142@lwahl>"
DC> "fake+none+nndraft:drafts+1" "fake+none+nndraft:drafts+2"
DC> "<73b0a166382dc.4e6e5c38@uwo.ca>"
DC> "<1315880651.2311.22.camel@mike-desktop>"
DC> "<4E6F239E.8030506@gmail.com>" "<4E6EAF23.1070406@uwo.ca>"
DC> "<LYRIS-60085979-25089418-2011.09.13-02.55.25--jdc#uwo.ca@go.netatlantic.com>"
DC> "<2139667233.835036.1315899248272.JavaMail.app@ela4-bed46.prod>"
DC> "<4E6F60C1.7090809@wagnerrp.com>"
DC> "<p062408a2ca951193efe9@simon.thehobsons.co.uk>" ...)) (8997.2 nil))

DC> But there's a lot in that sub-list starting with nil:

DC> (length (cadar result))
DC> 10028

Hmmmm.  I'll think about it, nothing obvious so far.  Sorry.

DC> Could it be that during the import-eld process, pruning doesn't take
DC> place?

Right.

DC> But why then was I still getting errors when new mail was arriving?
DC> (And could those be reduced to warnings, so that one can still
DC> receive mail when a problem occurs?)

I think the hard limit was not adjusted somehow (maybe because of the
abort).  Could you please look at

 (let* ((db gnus-registry-db)
  (list (oref db :max-hard)) (oref db :max-soft)))

and see if it's 100K or 10K for the hard limit (and what's the soft
limit).

I haven't seen your data files yet, perhaps my spam filter ate them.
I'll look again.  They should help me duplicate the problem.

Thanks!
Ted