nnmaildir support for nnir.el

nnmaildir support for nnir.el

[Justus-bulk]

[Repost from my newly-subscribed address]

Hi,

I've hacked up some initial nnmaildir support for nnir.el (gnus
contrib), for the Swish++ and Namazu backends.

During the process, I rolled some redundant code shared by these
backends into a new function, which may be useful for other backends
as well.

Please review and test, and send me your comments. I hope this (or an
improved version) can be committed to CVS.

See below a diff with respect to the latest CVS version (7.22) for
convenient reviewing. For convenient testing, you can download the
full file at http://www.montefiore.ulg.ac.be/~piater/test/nnir.el .

Thanks,
Justus


diff -c /local/home/piater/share/emacs/lisp/nnir-orig.el /local/home/piater/share/emacs/lisp/nnir.el
*** /local/home/piater/share/emacs/lisp/nnir-orig.el	2008-01-04 15:10:46.200002809 +0100
--- /local/home/piater/share/emacs/lisp/nnir.el	2008-02-02 21:05:29.663526101 +0100
***************
*** 661,667 ****
             (if (string-match "^\\([^+]+\\)\\+\\(.+\\)$" gname)
                 (format "%s:%s" (match-string 1 gname) (match-string 2 gname))
               (concat gname ":")))
!        "native")))
  
  ;; Summary mode commands.
  
--- 663,669 ----
             (if (string-match "^\\([^+]+\\)\\+\\(.+\\)$" gname)
                 (format "%s:%s" (match-string 1 gname) (match-string 2 gname))
               (concat gname ":")))
!        (format "%s:%s" (car gnus-select-method) (cadr gnus-select-method)))))
  
  ;; Summary mode commands.
  
***************
*** 830,835 ****
--- 832,865 ----
  
  (nnoo-define-skeleton nnir)
  
+ 
+ ;; Helper function currently used by the Swish++ and Namazu backends;
+ ;; perhaps useful for other backends as well
+ (defun nnir-compose-result (dirnam article score prefix server)
+   "Extract the group from dirnam, and create a result vector
+ ready to be added to the list of search results."
+   
+   ;; remove nnir-*-remove-prefix from beginning of dirnam filename
+   (when (string-match (concat "^" prefix) dirnam)
+     (setq dirnam (replace-match "" t t dirnam)))
+ 
+   ;; remove trailing slash and, for nnmaildir, cur/new/tmp
+   (setq dirnam (substring dirnam 0 (if (string= server "nnmaildir:") -5 -1)))
+ 
+   ;; eliminate all ".", "/", "\" from beginning. Always matches.
+   (string-match "^[./\\]*\\(.*\\)$" dirnam)
+   (setq group (substitute ?. ?/ (match-string 1 dirnam))) ;; "/" -> "."
+   (setq group (substitute ?. ?\\ group)) ;; "\\" -> "."
+ 
+   (vector (nnir-group-full-name group server)
+ 	  (if (string= server "nnmaildir:")
+ 	      (nnmaildir-base-name-to-article-number
+ 	       (substring article 0 (string-match ":" article))
+ 	       group nil)
+ 	    (string-to-int article))
+ 	  (string-to-int score)))
+ 
+ 
  ;;; Search Engine Interfaces:
  
  ;; freeWAIS-sf interface.
***************
*** 939,944 ****
--- 969,977 ----
             (groupspec (cdr (assq 'group query)))
             (prefix (nnir-read-server-parm 'nnir-swish++-remove-prefix server))
             (artlist nil)
+ 	   (article-pattern (if (string= server "nnmaildir:")
+ 				":[0-9]+"
+ 			      "^[0-9]+\\(\\.[a-z0-9]+\\)?$"))
             (score nil) (artno nil) (dirnam nil) (group nil) )
  
        (when (equal "" qstring)
***************
*** 986,992 ****
                dirnam (file-name-directory (match-string 2)))
  
          ;; don't match directories
!         (when (string-match "^[0-9]+\\(\\.[a-z0-9]+\\)?$" artno)
            ;; nnml-use-compressed-files might be any string, but probably this
            ;; is sufficient.  Note that we can't only use the value of
            ;; nnml-use-compressed-files because old articles might have been
--- 1019,1025 ----
                dirnam (file-name-directory (match-string 2)))
  
          ;; don't match directories
!         (when (string-match article-pattern artno)
            ;; nnml-use-compressed-files might be any string, but probably this
            ;; is sufficient.  Note that we can't only use the value of
            ;; nnml-use-compressed-files because old articles might have been
***************
*** 996,1019 ****
              ;; maybe limit results to matching groups.
              (when (or (not groupspec)
                        (string-match groupspec dirnam))
! 
!               ;; remove nnir-swish++-remove-prefix from beginning of dirname
!               (when (string-match (concat "^" prefix)
!                                   dirnam)
!                 (setq dirnam (replace-match "" t t dirnam)))
! 
!               (setq dirnam (substring dirnam 0 -1))
!               ;; eliminate all ".", "/", "\" from beginning. Always matches.
!               (string-match "^[./\\]*\\(.*\\)$" dirnam)
!               ;; "/" -> "."
!               (setq group (substitute ?. ?/ (match-string 1 dirnam)))
!               ;; "\\" -> "."
!               (setq group (substitute ?. ?\\ group))
! 
!               (push (vector (nnir-group-full-name group server)
!                             (string-to-int artno)
!                             (string-to-int score))
!                     artlist)))))
  
        (message "Massaging swish++ output...done")
  
--- 1029,1036 ----
              ;; maybe limit results to matching groups.
              (when (or (not groupspec)
                        (string-match groupspec dirnam))
! 	      (push (nnir-compose-result dirnam artno score prefix server)
! 		    artlist)))))
  
        (message "Massaging swish++ output...done")
  
***************
*** 1207,1212 ****
--- 1224,1232 ----
      (error "The Namazu backend cannot search specific groups"))
    (save-excursion
      (let (
+ 	  (article-pattern (if (string= server "nnmaildir:")
+ 				":[0-9]+"
+ 			      "^[0-9]+$"))
            (artlist nil)
            (qstring (cdr (assq 'query query)))
            (prefix (nnir-read-server-parm 'nnir-namazu-remove-prefix server))
***************
*** 1256,1273 ****
                article (file-name-nondirectory (match-string 4)))
  
          ;; make sure article and group is sane
!         (when (and (string-match "^[0-9]+$" article)
                     (not (null group)))
!           (when (string-match (concat "^" prefix) group)
!             (setq group (replace-match "" t t group)))
! 
!           ;; remove trailing slash from groupname
!           (setq group (substring group 0 -1))
! 
!           ;; stuff results into artlist vector
!           (push (vector (nnir-group-full-name (substitute ?. ?/ group) server)
!                         (string-to-int article)
!                         (string-to-int score)) artlist)))
  
        ;; sort artlist by score
        (apply 'vector
--- 1276,1285 ----
                article (file-name-nondirectory (match-string 4)))
  
          ;; make sure article and group is sane
!         (when (and (string-match article-pattern article)
                     (not (null group)))
! 	  (push (nnir-compose-result group article score prefix server)
! 		artlist)))
  
        ;; sort artlist by score
        (apply 'vector

Diff finished.  Sat Feb  2 21:20:46 2008

Re: nnmaildir support for nnir.el

[Reiner Steib]

On Sun, Feb 03 2008, Justus-bulk@Piater.name wrote:

> I've hacked up some initial nnmaildir support for nnir.el (gnus
> contrib), for the Swish++ and Namazu backends.
>
> During the process, I rolled some redundant code shared by these
> backends into a new function, which may be useful for other backends
> as well.

Could you please provide ChangeLog entries?

> Please review and test, and send me your comments. I hope this (or an
> improved version) can be committed to CVS.

AFAICS, the patch is too large (< 15 lines of new code) to be
installed without legal papers.  So we need an assignment, either for
Gnus or for Emacs (which covers Gnus as well).  If you are willing to
sign, I'll send you the form off-list.

For files in contrib/ we usually don't need an assignment, but as we
want to integrate nnir.el in Gnus, we don't want to add un-assigned
code there.

> See below a diff with respect to the latest CVS version (7.22) for
> convenient reviewing. For convenient testing, you can download the
> full file at http://www.montefiore.ulg.ac.be/~piater/test/nnir.el .

When I diff this file with the version in the Gnus trunk, there are
lots of whitespace changes.  I can probably deal with this by applying
the output `diff -u -w nnir.el your-version-nnir.el' (or the patch
from your message), but in the future, please avoid gratuitous
whitespace changes.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnmaildir support for nnir.el

[Justus-bulk]

Reiner Steib <reinersteib+gmane@imap.cc> wrote on Sun, 03 Feb 2008
15:16:33 +0100:

> Could you please provide ChangeLog entries?

(nnir-group-server): For native groups, obtain the server name from
gnus-select-method.

(nnir-run-swish++) and (nnir-run-namazu): Added support for nnmaildir.

(nnir-compose-result): New function unifying some code formerly
duplicated between nnir-run-swish++ and nnir-run-namazu.

> AFAICS, the patch is too large (< 15 lines of new code) to be
> installed without legal papers.  So we need an assignment, either for
> Gnus or for Emacs (which covers Gnus as well).  If you are willing to
> sign, I'll send you the form off-list.

This is about assigning the copyright to the FSF, right? No problem,
I'll (as)sign.

OTOH, my changes are really minimal; most of the "new" code
(nnir-compose-result) was just rearranged from elsewhere.

> When I diff this file with the version in the Gnus trunk, there are
> lots of whitespace changes.

Sorry about that. The reason is that the "download" option of
http://quimby.gnus.org/cgi-bin/cvsweb.cgi/ is broken (could somebody
fix this?). So when I just quickly wanted to try out nnir a few weeks
ago, the quickest was to simply cut & paste from the browser
window. And then I started hacking on the code without rembering to
retrieve a clean checkout first.

Anyway, I've replaced the file on my server
(http://www.montefiore.ulg.ac.be/~piater/test/nnir.el) by a clean
version.  Note that I did some superficial spambot protection by
changing @ to <at>; you may want to change them back before checking
in.

Cheers,
Justus

Re: nnmaildir support for nnir.el

[Justus-bulk]

Reiner Steib <reinersteib+gmane@imap.cc> wrote on Sat, 16 Feb 2008
22:40:49 +0100:

> Please come back to us on <ding@gnus.org> after receiving the
> confirmation from the FSF so that we can install your changes.

... which I hereby do.

See my message of Mon, 04 Feb 2008 18:28:31 +0100 for changelog
entries for my first set of changes, my message of Sun, 02 Mar 2008
17:36:12 +0100 for my latest change, and below for one more changelog
entry for this latest change
(http://www.montefiore.ulg.ac.be/~piater/test/nnir.el).

Thanks,
Justus


(nnir-run-swish++): Exclude unreadable files from search results.

Re: nnmaildir support for nnir.el

[Reiner Steib]

On Mon, Feb 04 2008, Justus-bulk@Piater.name wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> wrote:
>
>> Could you please provide ChangeLog entries?
>
> (nnir-group-server): For native groups, obtain the server name from
> gnus-select-method.
>
> (nnir-run-swish++) and (nnir-run-namazu): Added support for nnmaildir.
>
> (nnir-compose-result): New function unifying some code formerly
> duplicated between nnir-run-swish++ and nnir-run-namazu.

Installed.  Thanks for your contribution.

> Anyway, I've replaced the file on my server
> (http://www.montefiore.ulg.ac.be/~piater/test/nnir.el) by a clean
> version.  Note that I did some superficial spambot protection by
> changing @ to <at>; you may want to change them back before checking
> in.

Is it okay for you to use your address unprotected and without the
"-bulk" suffix in the file and in the ChangeLogs?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnmaildir support for nnir.el

[Justus-bulk]

Reiner Steib <reinersteib+gmane@imap.cc> wrote on Sun, 13 Apr 2008
15:48:13 +0200:

> Is it okay for you to use your address unprotected and without the
> "-bulk" suffix in the file and in the ChangeLogs?

Certainly without the "-bulk" suffix. OK also for unprotected address,
except for places where it is easily accessible to spambots.

Thanks,
Justus

Re: nnmaildir support for nnir.el

[Reiner Steib]

On Sun, Apr 13 2008, Justus-bulk@Piater.name wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> wrote on Sun, 13 Apr 2008
> 15:48:13 +0200:
>
>> Is it okay for you to use your address unprotected and without the
>> "-bulk" suffix in the file and in the ChangeLogs?
>
> Certainly without the "-bulk" suffix. OK also for unprotected address,
> except for places where it is easily accessible to spambots.

As the lisp and ChangeLog files are accessible via CVS web frontends
and the commit list also have web-enabled archives, I can't promise
this.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnmaildir support for nnir.el

[Christoph Conrad]

Hi Justus,

> Please review and test,

I only tested whether nnml swish++ does still work - seems to work. In
the moment i have no time and energy to do anything else with your code,
i hope some other person can do so. I also have no commit rights for
Gnus CVS.

With kind regards,
Christoph

Re: nnmaildir support for nnir.el

[Reiner Steib]

On Sat, Feb 02 2008, Christoph Conrad wrote:

> Hi Justus,
>
>> Please review and test,
>
> I only tested whether nnml swish++ does still work - seems to work. In
> the moment i have no time and energy to do anything else with your code,
> i hope some other person can do so. I also have no commit rights for
> Gnus CVS.

I can't fetch the Justus' message.  So if you want us to install,
please re-post the patch/code.

If it's more than a "tiny change", it would be good to have an
assignment, because we want to install nnir.el properly in Gnus and
Emacs.

I don't remember what the latest status was, but I could find it in
the archives.  Here it is...
<http://thread.gmane.org/gmane.emacs.gnus.general/62292/focus=63474>
(WRT to Adrian's assignment, the was some delay because of the
employer disclaimer.  I'll ask him about the status.)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnmaildir support for nnir.el

[Christoph Conrad]

Hi Reiner,

> I can't fetch the Justus' message. So if you want us to install,
> please re-post the patch/code.

I send Justus code to your GMX address.

> If it's more than a "tiny change", it would be good to have an
> assignment, because we want to install nnir.el properly in Gnus and
> Emacs.

I am not sure if its "tiny", please have a look at it and judge by
yourself.

> (WRT to Adrian's assignment, the was some delay because of the
> employer disclaimer. I'll ask him about the status.)

Ok.

With kind regards,
Christoph