nnir + namazu: always <= 999 matches ?

nnir + namazu: always <= 999 matches ?

[George McNinch]

Hi,

Probably I'm doing something bone-headed, but I can't work out what.

I have some nnml groups, and I've configured nnir for use with the
namazu engine. For any search, the ephemeral groups produced by nnir
seem to contain no more than 999 matches even when namazu gets more hits
than 999, and I'm not sure how to override that max.

Here are more details.  My config files contain:

,----
| (setq gnus-secondary-select-methods
|       '(...
| 	  (nnml "mail"
| 	    (nnir-search-engine namazu)
| 	    (nnml-directory "/home/george/Mail/"))))
`----

and also              
              
,----
| (setq nnir-namazu-index-directory "/home/george/Config/namazu/")
| (setq nnir-namazu-remove-prefix "/home/george/Mail/")
`----

I've created the index via mknmz.

If I run the following (in an *eshell*, or what have you) to search for
mails containing (say) the phrase "linear":

,----
| namazu -c -q -a -s linear /home/george/Config/namazu/
`----

I get the answer => 1403

(the "-c" flag means: "print the count of the number of hits").

But, if I "GG" on one of the groups associated with this nnml server,
and type "linear", I'm prompted

,----
| How many articles from nnir-82y50pqfrq.fsf (available 999, default 200):
`----

To confirm this number, note that if I eval (length nnir-artlist) in a
scratch buffer, I get 999.

So where did the remaining 1403-999 hits go?

In fact, as far as I can tell (i.e. varying the search term), *all*
namazu searches result in an ephemeral buffer containing <= 999
articles. I didn't immediately see a setting to over-ride that size
limit (sorry in advance if I'm missed something obvious...), because I
can't seem to see how/where in nnir.el it is imposed.

Note that the *Messages* buffer confirms that namazu is being called
with "the same" args [except the -c] as above; here is the line from the
*Messages* buffer:

,----
| namazu args: -q -a -s linear /home/george/Config/namazu/
`----

And subsequent to running the search ["GG linear" in the group buffer]
the " *nnir*" buffer indeed contains 1403 "entries".

Looking at the *nnir* buffer in more detail, the first 821 hits have
score >=3, and entries 822 through 1403 have score 2.  So there is no
"obvious" (to me) reason for there to be only 999 hits.

For what it is worth, after changing back to find-grep:

,----
| (setq gnus-secondary-select-methods
|       '(...
| 	  (nnml "mail"
| ;	    (nnir-search-engine namazu)
|           (nnir-search-engine find-grep)
| 	    (nnml-directory "/home/george/Mail/"))))
`----

restarting emacs, starting gnus, marking all groups in the nnml+mail
server via #, and searching with "GG linear", I get approx. 1600 hits,
and (upon request) an ephemeral buffer containing all (~1600) matching
articles.

So the issue seems to be related to use of namazu in nnir.

(Note that I'm *not* complaining here about the difference between ~1600
find-grep hits and the ~1400 namazu hits ... )

I should have tried also swish-e for comparison, but it was slow in
building the index and I got impatient...

Thanks for any help!

All the best,
gm



-- 
   ^^^^^   George McNinch <gmcninch (at) gmail.com>

Re: nnir + namazu: always <= 999 matches ? ANS: commas!

[George McNinch]

[-- Attachment #1: Type: text/plain, Size: 1161 bytes --]

Earlier, I wrote:

    gm> I have some nnml groups, and I've configured nnir for use with
    gm> the namazu engine. For any search, the ephemeral groups produced
    gm> by nnir seem to contain no more than 999 matches even when
    gm> namazu gets more hits than 999, and I'm not sure how to override
    gm> that max.

Aha, I finally noticed that the *nnir* buffer had entries like:

,----
| 1,402. Re: Your message of Mon, 3 Nov 1997 15:44:21 -0500 (score: 2)
| /home/george/Mail/mathematics/135 (2,723 bytes)
`----

Note the *comma* in 1,402.

The regexp matching these entries did not allow for this comma, which
explains hit-counts being always <= 999. 

The "hit number" that has the comma in it isn't actually used anywhere,
so there seems to be no harm to just include the possible comma in the
regexp.

Maybe there is some way to tell namazu not to emit commas when counting
past 1000, but I didn't see how in the man page.

Here is a patch for nnir.el fixing the regexp in nnir-run-namazu, though
I wasn't quite sure how I ought to "make the patch correctly" so that
someone can easily put the fix in the git archive. It is anyhow a simple
change.


[-- Attachment #2: nnir.el-diff --]
[-- Type: text/plain, Size: 923 bytes --]

diff -c /home/george/Config/dot-emacs/contrib/old/nnir.el /home/george/Config/dot-emacs/contrib/new/nnir.el
*** /home/george/Config/dot-emacs/contrib/old/nnir.el	2014-03-06 11:46:24.029862148 -0500
--- /home/george/Config/dot-emacs/contrib/new/nnir.el	2014-03-06 11:52:55.873872182 -0500
***************
*** 1483,1489 ****
  
        (goto-char (point-min))
        (while (re-search-forward
!               "^\\([0-9]+\\.\\).*\\((score: \\([0-9]+\\)\\))\n\\([^ ]+\\)"
                nil t)
          (setq score (match-string 3)
                group (file-name-directory (match-string 4))
--- 1483,1489 ----
  
        (goto-char (point-min))
        (while (re-search-forward
!               "^\\([0-9,]+\\.\\).*\\((score: \\([0-9]+\\)\\))\n\\([^ ]+\\)"
                nil t)
          (setq score (match-string 3)
                group (file-name-directory (match-string 4))

Diff finished.  Thu Mar  6 11:53:21 2014

[-- Attachment #3: Type: text/plain, Size: 101 bytes --]




-- 
   -===-   George McNinch <gmcninch@gmail.com>
   -===-   http://gmcninch.math.tufts.edu     

Re: nnir + namazu: always <= 999 matches ? ANS: commas!

[Andreas Schwab]

George McNinch <gmcninch@gmail.com> writes:

> Maybe there is some way to tell namazu not to emit commas when counting
> past 1000, but I didn't see how in the man page.

Look to be hardcoded.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: nnir + namazu: always <= 999 matches ? ANS: commas!

[Lars Ingebrigtsen]

George McNinch <gmcninch@gmail.com> writes:

> Here is a patch for nnir.el fixing the regexp in nnir-run-namazu, though
> I wasn't quite sure how I ought to "make the patch correctly" so that
> someone can easily put the fix in the git archive. It is anyhow a simple
> change.

Thanks; applied to bzr Emacs (and soon Ma Gnus).

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/