nnir.el and absent files

nnir.el and absent files

[Justus-bulk]

[-- Attachment #1: Type: text/plain, Size: 794 bytes --]

Hi,

How is nnir.el supposed to behave if the search engine returns files
that no longer exist (because the mails in question have been deleted
or moved elsewhere since the last index generation)?

My nnmaildir-enabled nnir.el bombs out in this case; see the stack
backtrace in the attachment.

Before I go bug hunting, maybe somebody can explain to me how this is
supposed to work (or not). So:

- Did I break this when I added nnmaildir support?

- Is it handled by nnir.el (How?), but I failed to take care of this
  case when adding nnmaildir support?

- Is this an open issue with nnir.el and has nothing to do with my
  nnmaildir changes? For example, does nnir.el simply assume that the
  search engine never returns inexistent files? (Swish++ does not seem
  to care.)

Thanks,
Justus


[-- Attachment #2: Backtrace --]
[-- Type: text/plain, Size: 4427 bytes --]

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
  byte-code("..." [nlist2 x high throw iterate-loop nil] 4)
  nnmaildir--nlist-iterate((...) (nil) #[(article) "..." [nnmaildir--cur-server group article nov cl-struct-nnmaildir--art-tags num nnmaildir--update-nov nnmaildir--cache-nov 0 error "nnmaildir--art-num accessing a non-nnmaildir--art" 3 princ "	" "nnmaildir--art-msgid accessing a non-nnmaildir--art" 4 1 "	Xref: nnmaildir " ":" 2 "\n" nntp-server-buffer gname] 10])
  byte-code("..." [...] 5)
  nnmaildir-retrieve-headers((nil) "lists.computer" "" nil)
  gnus-retrieve-headers((nil) "lists.computer" nil)
  (setq foo (gnus-retrieve-headers (list artno) artfullgroup nil))
  (let ((--cl-var-- ...)) (cond (... ... ... ...) (... ... ... ...) (t ...)))
  (case (setq foo (gnus-retrieve-headers ... artfullgroup nil)) (nov (goto-char ...) (setq novitem ...) (unless novitem ... ...)) (headers (goto-char ...) (setq novitem ...) (unless novitem ... ...)) (t (error "Unknown header type %s while requesting article %s of group %s" foo artno artfullgroup)))
  (let ((gnus-override-method ...)) (case (setq foo ...) (nov ... ... ...) (headers ... ... ...) (t ...)))
  (while (not (null artlist)) (setq art (car artlist)) (or (numberp art) (nnheader-report ... "nnir-retrieve-headers doesn't grok message ids: %s" art)) (setq artitem (nnir-artlist-article nnir-artlist art)) (setq artrsv (nnir-artitem-rsv artitem)) (setq artfullgroup (nnir-artitem-group artitem)) (setq artno (nnir-artitem-number artitem)) (setq artgroup (gnus-group-real-name artfullgroup)) (setq server (nnir-group-server artfullgroup)) (set-buffer nntp-server-buffer) (nnir-possibly-change-server server) (let (...) (case ... ... ... ...)) (mail-header-set-number novitem art) (mail-header-set-from novitem (mail-header-from novitem)) (mail-header-set-subject novitem (format "[%d: %s/%d] %s" artrsv artgroup artno ...)) (push novitem novdata) (setq artlist (cdr artlist)))
  (let ((artlist ...) (art nil) (artitem nil) (artgroup nil) (artno nil) (artrsv nil) (artfullgroup nil) (novitem nil) (novdata nil) (foo nil) server) (while (not ...) (setq art ...) (or ... ...) (setq artitem ...) (setq artrsv ...) (setq artfullgroup ...) (setq artno ...) (setq artgroup ...) (setq server ...) (set-buffer nntp-server-buffer) (nnir-possibly-change-server server) (let ... ...) (mail-header-set-number novitem art) (mail-header-set-from novitem ...) (mail-header-set-subject novitem ...) (push novitem novdata) (setq artlist ...)) (setq novdata (nreverse novdata)) (set-buffer nntp-server-buffer) (erase-buffer) (mapcar (quote nnheader-insert-nov) novdata) (quote nov))
  (save-excursion (let (... ... ... ... ... ... ... ... ... ... server) (while ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) (setq novdata ...) (set-buffer nntp-server-buffer) (erase-buffer) (mapcar ... novdata) (quote nov)))
  nnir-retrieve-headers((1 2 3 4 5 6 7 8) "((query . \"steib reiner\") (unique-id . \"x8tfxvcnffh.fsf\"))" "-ephemeral" nil)
  gnus-retrieve-headers((1 2 3 4 5 6 7 8) "nnir:((query . \"steib reiner\") (unique-id . \"x8tfxvcnffh.fsf\"))" nil)
  byte-code("..." [...] 6)
  gnus-cache-retrieve-headers((1 2 3 4 5 6 7 8) "nnir:((query . \"steib reiner\") (unique-id . \"x8tfxvcnffh.fsf\"))" nil)
  gnus-retrieve-headers((1 2 3 4 5 6 7 8) "nnir:((query . \"steib reiner\") (unique-id . \"x8tfxvcnffh.fsf\"))" nil)
  gnus-fetch-headers((1 2 3 4 5 6 7 8))
  gnus-select-newsgroup("nnir:((query . \"steib reiner\") (unique-id . \"x8tfxvcnffh.fsf\"))" t nil)
  gnus-summary-read-group-1("nnir:((query . \"steib reiner\") (unique-id . \"x8tfxvcnffh.fsf\"))" t t nil nil nil)
  gnus-summary-read-group("nnir:((query . \"steib reiner\") (unique-id . \"x8tfxvcnffh.fsf\"))" t t nil nil nil nil)
  byte-code("..." [] 8)
  gnus-group-read-group(t t "nnir:((query . \"steib reiner\") (unique-id . \"x8tfxvcnffh.fsf\"))" nil)
  byte-code("..." [...] 5)
  gnus-group-read-ephemeral-group("nnir:((query . \"steib reiner\") (unique-id . \"x8tfxvcnffh.fsf\"))" (nnir "") t (#<buffer *Group*> . group) nil)
  (let ((parms nil)) (if extra-parms (setq parms ...) (setq parms ...)) (add-to-list (quote parms) (cons ... ...) t) (gnus-group-read-ephemeral-group (concat "nnir:" ...) (quote ...) t (cons ... gnus-current-window-configuration) nil))
  gnus-group-make-nnir-group(nil "steib reiner")
  call-interactively(gnus-group-make-nnir-group)

Re: nnir.el and absent files

[Justus-bulk]

Hi -

Replying to my own posting:

I've made a trivial change to fix this for the swish++ backend by
simply testing each file for readability. See below for a diff with
respect to my previous version of nnir.el with nnmaildir support at
http://www.montefiore.ulg.ac.be/~piater/test/nnir.el ; I've now
replaced this file by the new one including the absent-file fix.

I consider this a workaround; I think desktop search engines should
offer the option of not returning files that have been removed since
the last index generation.

Justus


diff -c /local/home/piater/WWW/test/nnir.el /local/home/piater/share/emacs/lisp/gnus/contrib/nnir.el
*** /local/home/piater/WWW/test/nnir.el	2008-02-04 18:16:27.401299301 +0100
--- /local/home/piater/share/emacs/lisp/gnus/contrib/nnir.el	2008-03-02 17:21:07.995871209 +0100
***************
*** 1017,1027 ****
        (while (re-search-forward
                "\\(^[0-9]+\\) \\([^ ]+\\) [0-9]+ \\(.*\\)$" nil t)
          (setq score (match-string 1)
!               artno (file-name-nondirectory (match-string 2))
!               dirnam (file-name-directory (match-string 2)))
! 
!         ;; don't match directories
!         (when (string-match article-pattern artno)
  	  ;; nnml-use-compressed-files might be any string, but probably this
  	  ;; is sufficient.  Note that we can't only use the value of
  	  ;; nnml-use-compressed-files because old articles might have been
--- 1017,1029 ----
        (while (re-search-forward
                "\\(^[0-9]+\\) \\([^ ]+\\) [0-9]+ \\(.*\\)$" nil t)
          (setq score (match-string 1)
! 	      filenam (match-string 2)
!               artno (file-name-nondirectory filenam)
!               dirnam (file-name-directory filenam))
! 
!         ;; don't match directories or inexistent/unreadable files
!         (when (and (string-match article-pattern artno)
! 		   (file-readable-p filenam))
  	  ;; nnml-use-compressed-files might be any string, but probably this
  	  ;; is sufficient.  Note that we can't only use the value of
  	  ;; nnml-use-compressed-files because old articles might have been

Re: nnir.el and absent files

[Reiner Steib]

On Sun, Mar 02 2008, Justus-bulk@Piater.name wrote:

> I've made a trivial change to fix this for the swish++ backend by
> simply testing each file for readability. See below for a diff [...]

Installed.  Thanks for your contribution.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/