From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/53803 Path: main.gmane.org!not-for-mail From: kai.grossjohann@gmx.net (=?iso-8859-1?q?Kai_Gro=DFjohann?=) Newsgroups: gmane.emacs.gnus.general Subject: Re: Fledgling project nnsearch.el Date: Sun, 24 Aug 2003 16:57:07 +0200 Organization: University of Duisburg, Germany Sender: ding-owner@lists.math.uh.edu Message-ID: <848ypj5cjg.fsf@slowfox.is.informatik.uni-duisburg.de> References: <841xvbb03m.fsf@slowfox.is.informatik.uni-duisburg.de> NNTP-Posting-Host: deer.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: sea.gmane.org 1061737069 2338 80.91.224.253 (24 Aug 2003 14:57:49 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Sun, 24 Aug 2003 14:57:49 +0000 (UTC) Original-X-From: ding-owner+M2343@lists.math.uh.edu Sun Aug 24 16:57:46 2003 Return-path: Original-Received: from malifon.math.uh.edu ([129.7.128.13]) by deer.gmane.org with esmtp (Exim 3.35 #1 (Debian)) id 19qwJO-0001qX-00 for ; Sun, 24 Aug 2003 16:57:46 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by malifon.math.uh.edu with smtp (Exim 3.20 #1) id 19qwIv-0002fK-00; Sun, 24 Aug 2003 09:57:17 -0500 Original-Received: from sclp3.sclp.com ([64.157.176.121]) by malifon.math.uh.edu with smtp (Exim 3.20 #1) id 19qwIo-0002fF-00 for ding@lists.math.uh.edu; Sun, 24 Aug 2003 09:57:10 -0500 Original-Received: (qmail 70920 invoked by alias); 24 Aug 2003 14:57:10 -0000 Original-Received: (qmail 70915 invoked from network); 24 Aug 2003 14:57:10 -0000 Original-Received: from quimby.gnus.org (80.91.224.244) by sclp3.sclp.com with SMTP; 24 Aug 2003 14:57:10 -0000 Original-Received: from news by quimby.gnus.org with local (Exim 3.12 #1 (Debian)) id 19qwNu-0005tJ-00 for ; Sun, 24 Aug 2003 17:02:26 +0200 Original-To: ding@gnus.org Original-Path: not-for-mail Original-Newsgroups: gnus.ding Original-Lines: 431 Original-NNTP-Posting-Host: pd951f940.dip.t-dialin.net Original-X-Trace: quimby.gnus.org 1061737346 22086 217.81.249.64 (24 Aug 2003 15:02:26 GMT) Original-X-Complaints-To: usenet@quimby.gnus.org Original-NNTP-Posting-Date: 24 Aug 2003 15:02:26 GMT Mail-Copies-To: never User-Agent: Gnus/5.1003 (Gnus v5.10.3) Emacs/21.3.50 (gnu/linux) Cancel-Lock: sha1:BD29d3TcrZ63lLCxkCzcz3s/EM4= Precedence: bulk Xref: main.gmane.org gmane.emacs.gnus.general:53803 X-Report-Spam: http://spam.gmane.org/gmane.emacs.gnus.general:53803 --=-=-= Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit kai.grossjohann@gmx.net (Kai Großjohann) writes: > What does nnsearch currently do? > > * It can use Namazu to query your nnml backend, if it is named > "nnml:". > > How to set it up? > > * Let's say you have this server spec: > (nnml "" ...other.parms...) > Change it to read like so: > (nnml "" > (nnsearch-root-directory "~/Mail") > (nnsearch-namazu-index-directory "~/.namazu") > ...other.parms...) Now you can use it on multiple nnml backends, regardless of their name. The logic to choose the search engine is in place. (Though only Namazu is currently supported.) Now the server spec should read like this: (nnml "" (nnsearch-function nnsearch-namazu-query) (nnsearch-root-directory "~/Mail") (nnsearch-namazu-index-directory "~/.namazu") ...other.parms...) What's the scoop on the current feature-freeze? Does this mean nnsearch.el can't go in at the moment? (It can't go in until it gets much better anyway.) -- Two cafe au lait please, but without milk. --=-=-= Content-Type: application/emacs-lisp Content-Disposition: attachment; filename=nnsearch.el Content-Transfer-Encoding: 8bit ;;; nnsearch.el --- Search engine integration ;; Copyright (C) 2003 Free Software Foundation, Inc. ;; Author: Kai Großjohann ;; Keywords: mail, news, processes ;; This file is free software; you can redistribute it and/or modify ;; it under the terms of the GNU General Public License as published by ;; the Free Software Foundation; either version 2, or (at your option) ;; any later version. ;; This file is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; You should have received a copy of the GNU General Public License ;; along with GNU Emacs; see the file COPYING. If not, write to ;; the Free Software Foundation, Inc., 59 Temple Place - Suite 330, ;; Boston, MA 02111-1307, USA. ;;; Commentary: ;; This file provides search engine integration with Gnus. (nnoo-declare nnsearch) (nnoo-define-basics nnsearch) (gnus-declare-backend "nnsearch" 'mail) ;;; Variables: (defvar nnsearch-parameter-alist nil "Alist of (SERVER-NAME . PARAMETERS) pairs. PARAMETERS is again an alist.") (defvar nnsearch-work-buffer "*nnsearch work*" "Name of temporary buffer.") (defvar nnsearch-query-cache-timeout 10 "Number of seconds the query cache is valid.") (defvar nnsearch-namazu-configuration "\ Index \"@INDEX@\" Template \"@INDEX@\" Scoring \"tfidf\" EmphasisTags \"\" \"\" " "Template for Namazu config file contents. This string will be munged as shown below and the result will be used as the Namazu config file. Munging replaces @INDEX@ with the name of the index directory, also see the variable `nnsearch-namazu-index-directory'.") (defvar nnsearch-namazu-result-template "\ result=${namazu::counter} uri=${uri} score=${namazu::score} endresult " "This string will be used as the NMZ.result.normal file.") ;; These four are for caching query results. Due to the structure of ;; Gnus, queries are wont to be issued multiple times. (defvar nnsearch-last-server nil "Name of server used for the most recent query.") (defvar nnsearch-last-query nil "Last query issued.") (defvar nnsearch-last-result nil "Last query result.") (defvar nnsearch-last-time nil "Time when last query was issued.") ;;; Code: ;;; Gnus glue: (defun gnus-group-search (query) "Search in all groups. Asks for query." (interactive "sQuery: ") (gnus-group-read-ephemeral-group (concat "nnsearch:" query) '(nnsearch "") t (cons (current-buffer) gnus-current-window-configuration) nil)) ;;; Server interface: (deffoo nnsearch-open-server (server &optional definitions) ;; Just set the server variables appropriately. (nnoo-change-server 'nnsearch server definitions)) (deffoo nnsearch-request-group (group &optional server fast) "GROUP is the query string." (nnsearch-possibly-change-server server) (unless (and (string= group nnsearch-last-query) (gnus-server-equal server nnsearch-last-server) nnsearch-last-time (< (- (float-time) nnsearch-last-time) nnsearch-query-cache-timeout)) (setq nnsearch-last-result (nnsearch-query group) nnsearch-last-query group nnsearch-last-server server nnsearch-last-time (float-time))) (save-excursion (set-buffer nntp-server-buffer) (nnheader-insert "211 %d %d %d %s\n" (length nnsearch-last-result) ;total # 1 ;first # (length nnsearch-last-result) ;last # group))) (deffoo nnsearch-retrieve-headers (articles &optional group server fetch-old) (save-excursion (let ((artlist (copy-sequence articles)) (idx 1) (art nil) (artitem nil) (artgroup nil) (artno nil) (artscore nil) (artfullgroup nil) (novitem nil) (novdata nil) (foo nil) server) (while (not (null artlist)) (setq art (car artlist)) (or (numberp art) (nnheader-report 'nnsearch "nnsearch-retrieve-headers doesn't grok message ids: %s" art)) (setq artitem (nnsearch-artlist-article nnsearch-last-result art)) (setq artscore (nnsearch-artitem-score artitem)) (setq artfullgroup (nnsearch-artitem-group artitem)) (setq artno (nnsearch-artitem-number artitem)) (setq artgroup (gnus-group-real-name artfullgroup)) (setq server (gnus-find-method-for-group artfullgroup)) ;; retrieve NOV or HEAD data for this article, transform into ;; NOV data and prepend to `novdata' (set-buffer nntp-server-buffer) (nnsearch-possibly-change-server server) (case (setq foo (gnus-retrieve-headers (list artno) artfullgroup nil)) (nov (goto-char (point-min)) (setq novitem (nnheader-parse-nov)) (unless novitem (pop-to-buffer nntp-server-buffer) (error "nnheader-parse-nov returned nil for article %s in group %s" artno artfullgroup))) (headers (goto-char (point-min)) (setq novitem (nnheader-parse-head)) (unless novitem (pop-to-buffer nntp-server-buffer) (error "nnheader-parse-head returned nil for article %s in group %s" artno artfullgroup))) (t (nnheader-report 'nnsearch "Don't support header type %s." foo))) ;; replace article number in original group with article number ;; in nnsearch group (mail-header-set-number novitem idx) (mail-header-set-from novitem (mail-header-from novitem)) (mail-header-set-subject novitem (format "[%d: %s/%d] %s" artscore artgroup artno (mail-header-subject novitem))) ;;-(mail-header-set-extra novitem nil) (push novitem novdata) (setq artlist (cdr artlist)) (setq idx (1+ idx))) (setq novdata (nreverse novdata)) (set-buffer nntp-server-buffer) (erase-buffer) (mapcar 'nnheader-insert-nov novdata) 'nov))) (deffoo nnsearch-request-article (article &optional group server to-buffer) (save-excursion (let* ((artitem (nnsearch-artlist-article nnsearch-last-result article)) (artfullgroup (nnsearch-artitem-group artitem)) (artno (nnsearch-artitem-number artitem)) ;; Bug? ;; Why must we bind nntp-server-buffer here? It won't ;; work if `buf' is used, say. (Of course, the set-buffer ;; line below must then be updated, too.) (nntp-server-buffer (or to-buffer nntp-server-buffer))) (set-buffer nntp-server-buffer) (erase-buffer) (message "Requesting article %d from group %s" artno artfullgroup) (gnus-request-article artno artfullgroup nntp-server-buffer) (cons artfullgroup artno)))) (nnoo-define-skeleton nnsearch) ;;; Util functions: (defun nnsearch-possibly-change-server (server) (unless (and server (nnsearch-server-opened server)) (nnsearch-open-server server))) ;;; Main querying logic: (defun nnsearch-query (query) "Execute given query on all servers that support searching." (mapcan (lambda (x) (nnsearch-query-1 (car x) query)) gnus-opened-servers)) (defun nnsearch-query-1 (server query) "If the server supports querying, issue query." (let ((func (nnsearch-get-parameter 'nnsearch-function server t))) (when func (funcall func server query)))) ;;; Server parameters: (defun nnsearch-get-parameter (param server &optional optional) "Get the value of the given parameter in the given server. This looks it up in the server spec first, and then in `nnsearch-parameter-alist' if not found there. Unless OPTIONAL is non-nil, barf if parameter cannot be found." (when (stringp server) (setq server (gnus-server-to-method server))) (let ((entry (assq param server))) (if entry (second entry) ;; Look it up in nnsearch-parameter-alist. (setq server (gnus-method-to-full-server-name server)) (setq server (replace-regexp-in-string "\\+" ":" server)) (setq entry (assq server nnsearch-parameter-alist)) (unless (or entry optional) (error "Can't find entry for server %s in nnsearch-parameter-alist" server)) (setq entry (cdr entry)) (setq entry (assq param entry)) (unless (or entry optional) (error "Can't find entry for parameter %s in nnsearch-parameter-alist" param)) (when entry (second entry))))) ;;; Interfaces to search engines: ;; Namazu: (defun nnsearch-namazu-query (server query) "Run Namazu and return vector of triples [GROUP ARTICLE SCORE]. The argument QUERY is a string in Namazu query format. The second argument ROOT-DIRECTORY gives the root of the nnml hierarchy." (nnsearch-namazu-setup) (save-excursion (set-buffer (get-buffer-create nnsearch-work-buffer)) (erase-buffer) (call-process "namazu" nil t nil "-f" (expand-file-name "rc" (nnsearch-get-parameter 'nnsearch-namazu-index-directory server)) "-a" query) (append (nnsearch-namazu-parse-result server query)))) (defun nnsearch-namazu-parse-result (server query) "Parse Namazu result in current buffer. Return list of triples [GROUP ARTICLE SCORE]." (let ((root-directory (nnsearch-get-parameter 'nnsearch-root-directory server))) (setq root-directory (file-truename (expand-file-name root-directory))) (when (stringp server) (setq server (gnus-server-to-method server))) (setq server (gnus-method-to-server-name server)) (goto-char (point-min)) (let ((numresults 0) (currentres 0) result file score group article) ;; Find out number of results. (search-forward " Total ") (setq numresults (read (current-buffer))) (forward-line 1) ;; Make result vector. (setq result (make-vector numresults nil)) ;; Skip header. (while (and (not (looking-at "result=")) (not (eobp))) (forward-line 1)) ;; Read results. Each iteration assumes that point is at beginning ;; of "result=" line. (while (not (eobp)) ;; uri= (forward-line 1) (unless (looking-at "uri=") (error "Wrong result format from Namazu, expected uri")) (search-forward "=") (setq file (buffer-substring (point) (progn (end-of-line) (point)))) (setq file (file-truename file)) (setq file (file-relative-name file root-directory)) ;; Split file name into group and article. (setq article (string-to-int (file-name-nondirectory file))) (setq group (file-name-directory file)) (when (string-match "/\\'" group) (setq group (substring group 0 -1))) (setq group (replace-regexp-in-string "/" "." group t t)) ;; score= (forward-line 1) (unless (looking-at "score=") (error "Wrong result format from Namazu, expected score")) (search-forward "=") (setq score (read (current-buffer))) ;; endresult (forward-line 1) (unless (looking-at "endresult") (error "Wrong result format from Namazu, expected endresult")) ;; Add result to vector. (aset result currentres (vector (format "%s:%s" server group) article score)) ;; Find next result. (while (and (not (eobp)) (not (looking-at "result="))) (forward-line 1)) (setq currentres (1+ currentres))) result))) (defun nnsearch-namazu-setup () "Initial setup before running Namazu." (save-excursion (find-file (expand-file-name "NMZ.result.normal" nnsearch-namazu-index-directory)) (erase-buffer) (insert nnsearch-namazu-result-template) (save-buffer) (kill-buffer (current-buffer)) (find-file (expand-file-name "rc" nnsearch-namazu-index-directory)) (erase-buffer) (insert nnsearch-namazu-configuration) (goto-char (point-min)) (while (search-forward "@INDEX@" nil t) (replace-match nnsearch-namazu-index-directory t t)) (save-buffer) (kill-buffer (current-buffer)))) ;;; Data type article list. (defun nnsearch-artlist-length (artlist) "Returns number of articles in artlist." (length artlist)) (defun nnsearch-artlist-article (artlist n) "Returns from ARTLIST the Nth artitem (counting starting at 1)." (elt artlist (1- n))) (defun nnsearch-artitem-group (artitem) "Returns the group from the ARTITEM." (elt artitem 0)) (defun nnsearch-artlist-artitem-group (artlist n) "Returns from ARTLIST the group of the Nth artitem (counting from 1)." (nnsearch-artitem-group (nnsearch-artlist-article artlist n))) (defun nnsearch-artitem-number (artitem) "Returns the number from the ARTITEM." (elt artitem 1)) (defun nnsearch-artlist-artitem-number (artlist n) "Returns from ARTLIST the number of the Nth artitem (counting from 1)." (nnsearch-artitem-number (nnsearch-artlist-article artlist n))) (defun nnsearch-artitem-score (artitem) "Returns the score (Retrieval Status Value, RSV) from the ARTITEM." (elt artitem 2)) (defun nnsearch-artlist-artitem-score (artlist n) "Returns from ARTLIST the score of the Nth artitem (counting from 1)." (nnsearch-artitem-score (nnsearch-artlist-article artlist n))) (provide 'nnsearch) ;;; nnsearch.el ends here --=-=-=--