gnus-glimpse.el -- search a newsgroup using glimpse

gnus-glimpse.el -- search a newsgroup using glimpse

[Kai Grossjohann]

Hi,

I hacked up the following to search the current newsgroup using
glimpse.  Beware: it only works with nnml groups!

Situation:

  - You are indexing your home dir or your ~/Mail using glimpseindex
    every night.
  - You search for old mails using glimpse.
  - You would like to get a summary buffer of all the mails that
    glimpse found.

Then, gnus-glimpse.el is for you!

I bound it to "G G" in the summary buffer, so you can type "1 RET" to
enter some huge group quickly.  Then, you typ "G G" to have glimpse
search for some articles.

This piece is a quick and dirty hack and very slow, but performance
will surely improve as soon as Lars implements the extra entry point
that creates a summary buffer given a list of article numbers.  I was
too lazy to do this myself :-/

I'm posting this even though it doesn't really work, to get your
opinions.  Be aware that you are using something that has been tested
on all of 1 (one) test case so far...

Things to do:
  - Test it.
  - Make it faster.
  - Fetch old articles.
  - Better interface to glimpse (ideas, anyone?).

kai
-- 
Life is hard and then you die.

;; gnus-glimpse.el -- search current group with glimpse
;; University of Dortmund, Information Retrieval Group, CS Dept.
;; RCS Status      : $Id$
;; Author          : Kai Grossjohann
;; Created On      : Tue May 20 19:08:10 1997
;; Last Modified By: 
;; Last Modified On: 
;; Update Count    : 0
;; Status          : Unknown, Use with caution!

(require 'gnus-sum)

(defvar gnus-glimpse-program
  "/usr/local/bin/glimpse"
  "Path to glimpse program to run.")

(defvar gnus-glimpse-options "-W -y -l"
  "Default options to pass to glimpse.")

(defun gnus-glimpse-group-to-directory (group)
  "Return directory name for this group."
  (string-match "^nnml:\\(.*\\)$" group)
  (expand-file-name
   (concat nnml-directory
           (gnus-newsgroup-directory-form
            (substring group (match-beginning 1) (match-end 1))))))

(defun gnus-glimpse-group-option ()
  "Return the string used as the group option for glimpse."
  (let ((s (gnus-glimpse-group-to-directory gnus-newsgroup-name)))
    (string-match (expand-file-name "~/") s)
    (concat "-F '" (replace-match "" nil nil s 0)
            "/[0-9]'")))

(defun gnus-summary-glimpse (glimpse-opt)
  (interactive "sGlimpse options: ")
  (let ((cmd (concat gnus-glimpse-program " "
             gnus-glimpse-options " "
             (gnus-glimpse-group-option) " "
             glimpse-opt))
        (gnus-glimpse-articles nil)
        (num-articles nil)
        (cur-article 1))
    (save-excursion
      (set-buffer (get-buffer-create " *glimpse output*"))
      (erase-buffer)
      (message "Running %s..." cmd) (sit-for 0)
      (shell-command cmd t)
      (message "Running %s...done" cmd) (sit-for 0)
      (goto-char (point-min))
      (while (re-search-forward
              (concat "^" (gnus-glimpse-group-to-directory gnus-newsgroup-name)
                      "/\\([0-9]*\\)$")
              nil t nil)
        (replace-match "\\1" nil nil))
      (sort-numeric-fields 1 (point-min) (point-max))
      (goto-char (point-min))
      (insert "(setq gnus-glimpse-articles '(\n")
      (goto-char (point-max))
      (insert "))")
      (eval-current-buffer)
      )
    (setq num-articles (length gnus-glimpse-articles))
    (message "Found %d articles." num-articles) (sit-for 0)
    (while gnus-glimpse-articles 
      (setq cur-article (1+ cur-article))
      (gnus-summary-goto-article (pop gnus-glimpse-articles) nil t))))

(define-key gnus-summary-goto-map "G" 'gnus-summary-glimpse)

(provide 'gnus-glimpse)

;; gnus-glimpse.el ends here
                           

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Mark Eichin]

This is pretty cool!  I had some problems with the code that filters
on gnus-glimpse-group-to-directory, but after I upgraded gnus and
tweaked some other things, the problem went away.

The use of gnus-summary-goto-article is an interesting cheat :-)  It's
also a little confusing when there *are* other articles visible.
Perhaps something like the / restrict code uses might be another
approach?

I'd also be interested in an "nnglimpse" backend of some sort, when I
want to look at things regardless of category.  That sounds a little
harder (maybe not much, given some of the other backends...)

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Kai Grossjohann]

>>>>> Mark Eichin writes:

  Mark> The use of gnus-summary-goto-article is an interesting cheat
  Mark> :-) It's also a little confusing when there *are* other
  Mark> articles visible.  Perhaps something like the / restrict code
  Mark> uses might be another approach?

Well, that might be best.  Hm.  I'll have to look at the code that
does this, maybe I can come up with something.

  Mark> I'd also be interested in an "nnglimpse" backend of some sort,
  Mark> when I want to look at things regardless of category.  That
  Mark> sounds a little harder (maybe not much, given some of the
  Mark> other backends...)

I used nnweb the other day (for the first time!) and I quite liked
it.  That might be a place to start, at least for the interface.

kai
-- 
Life is hard and then you die.

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Greg Stark]

Interestingly i was just thinking the other day about how to implement
searching through a group for text in an article in a way that would support
protocol features to do it on the server side. It might be worth thinking of
some abstraction to handle this though i'm not sure of exactly what it should
look like.

For my purposes it would be enough to provide optional backend methods to
perform the various searches the / keymap supports. Then Gnus would implement
them itself the way it does currently if the backend doesn't provide them.
This is a bit limitting for something like gnus-glimpse where you might want
to drop in different search engines for the same backend, or support multiple
backends in the same search engine. But maybe with nnoo it wouldn't be hard
for gnus-glimpse to provide useful generic searching functions that the
backends could easily use to implement searching without too much code
duplication. Then you could set nnmail-search-engine or to 'glimpse and the
backend functions would try to use glimpse if available, and if not let Gnus
implement the searching functions itself. 

I would like to see something like this, it would let us do some funky things
like have / do a dejanews query but use the results to access the normal news
server, which would let you tick and cache articles on your normal news
server. It would also let me take advantage of native searching features in
the protocol for which i wrote a backend.

greg

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Mark Eichin]

> This is pretty cool!  I had some problems with the code that filters
> on gnus-glimpse-group-to-directory, but after I upgraded gnus and

Found it again -- to set you at ease, it's totally my own config
problem ($HOME and /etc/passwd didn't agree, though they pointed the
same place.  My problem, not the code's problem.)

This is pretty cool for retroactively starting new folders -- just GG
the list name, # and B m everything.  (Then get annoyed about B m
fetching mail :-)

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Lars Magne Ingebrigtsen]

Greg Stark <gsstark@mit.edu> writes:

> Interestingly i was just thinking the other day about how to
> implement searching through a group for text in an article in a way
> that would support protocol features to do it on the server side. It
> might be worth thinking of some abstraction to handle this though
> i'm not sure of exactly what it should look like.

Yes; server-side searches is a quite interesting area.  It's been on
the todo list for a while, but I haven't done anything concrete.

The search languages the different backends provide would be quite
varied, and I don't presently have any clear idea of how Gnus should
specify the searches...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Kai Grossjohann]

>>>>> Greg Stark writes:

  Greg> Interestingly i was just thinking the other day about how to
  Greg> implement searching through a group for text in an article in
  Greg> a way that would support protocol features to do it on the
  Greg> server side. It might be worth thinking of some abstraction to
  Greg> handle this though i'm not sure of exactly what it should look
  Greg> like.

Hm.  What would you all think if each backend would (optionally)
provide an nnchoke-search function expecting a string as argument and
returning a list of article numbers (maybe message-id's)?

Gnus could then have some function that queried the user for a search
string, would pass it to the nnchoke-search-function and display the
articles.

There could be different .el files for different search engines or
methods (put a require in .gnus.el to select one), or maybe one would
specify (setq nnml-search-type 'glimpse) which would lead Gnus to
invoke nnml-search-with-glimpse.

The problem of the different query string formats would still remain,
of course.  As a first approximation one would require the user to
enter the right string format.

In addition, it might also be nice if Gnus had a function that, given
a list of ("groupname" . articlenumber) pairs (or something), would
display these articles from the groups.  But that can be done later.

What do you think?
kai
-- 
Life is hard and then you die.

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Kai Grossjohann]

>>>>> Lars Magne Ingebrigtsen writes:

  Lars> The search languages the different backends provide would be
  Lars> quite varied, and I don't presently have any clear idea of how
  Lars> Gnus should specify the searches...

Specifying a syntax for queries that

  - the average user can understand,
  - is expressive enough to cover interesting cases, and
  - doesn't produce unmanageably long queries

is certainly not simple to think up.  Hm.  Infoseek uses a format that
might be workable.  An example:

    widgets +subject:foo from:john -subject:re

This query would search for "widgets" anywhere in a mail, "foo" in the
subject header, "john" in the from header, and "re" in the subject
header.  The search is on a best-match basis, but the "+" indicates
that "foo" *must* be in the subject and the "-" indicates that "re"
*may not* be in the subject.  For the other elements of the query, a
message that contains the word would tend to be ranked higher that a
message that doesn't.

The nnchoke-search functions I suggested before would have to accept a
string in the above format and do something meaningful with it.  For
some search engines, a lot of the semantics would have to be left
out.  For instance, I don't think field searches would be feasible
with glimpse, so there a query "subject:foo" would have to be
equivalent to just "foo".

Hm.  I see that I have assumed that the search engine does ranking.
(Documents that better match the query are ranked higher; to be exact,
documents that have a higher probability of being relevant to the
user's information need are ranked higher.)  Maybe we have to leave
this one out in the first version so there would be no difference when
specifying "+".

kai
-- 
Life is hard and then you die.

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Kai Grossjohann]

On searching newsgroups and on search engines with ranking.

It seems to me that there are two possibilities of presenting the
result of the query to a user:

(1) Messages that are ranked higher appear in the beginning of the
    summary buffer, lower ranked messages are put near the end.  This
    is what search engines have traditionally done.

(2) Use the scoring mechanism of Gnus to score highly relevant
    messages higher.  This gives the user a less intuitive feedback on
    which mails are relevant and how relevant they are.  But it
    interacts better with threading.  (After all, the probability is
    high that a user wants to read more messages from the same thread
    when one message of the thread has been found.)

The two alternatives conflict.  I don't see how to resolve this
conflict.

Any ideas?
kai
-- 
Life is hard and then you die.

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Norman Walsh]

/ Kai Grossjohann <grossjohann@charly.cs.uni-dortmund.de> was heard to say:
| I hacked up the following to search the current newsgroup using
| glimpse.  Beware: it only works with nnml groups!

Ok, so I finally got around to installing glimpse and giving this a
try.  I like the concept, but I'm a little confused by the display. Is
there any way to tell, in the list, which ones matched?

Looking through the list archives, I gather part of the problem is
that other messages in the group are not suppressed.  Has anyone done
any more work on this?

BTW, here's a crude hack to suppress bbdb auto notification:

*** gnus-glimpse.el~	Thu Nov  9 08:11:49 2000
--- gnus-glimpse.el	Thu Nov  9 11:04:51 2000
***************
*** 41,45 ****
          (gnus-glimpse-articles nil)
          (num-articles nil)
!         (cur-article 1))
      (save-excursion
        (set-buffer (get-buffer-create " *glimpse output*"))
--- 41,49 ----
          (gnus-glimpse-articles nil)
          (num-articles nil)
!         (cur-article 1)
! 	(save-bbdb-notice-hook
! 	 (if (boundp 'bbdb-notice-hook)
! 	     (symbol-value 'bbdb-notice-hook)
! 	   nil)))
      (save-excursion
        (set-buffer (get-buffer-create " *glimpse output*"))
***************
*** 65,69 ****
      (while gnus-glimpse-articles 
        (setq cur-article (1+ cur-article))
!       (gnus-summary-goto-article (pop gnus-glimpse-articles) nil t))))
  
  (define-key gnus-summary-goto-map "G" 'gnus-summary-glimpse)
--- 69,75 ----
      (while gnus-glimpse-articles 
        (setq cur-article (1+ cur-article))
!       (gnus-summary-goto-article (pop gnus-glimpse-articles) nil t))
!     (if (boundp 'bbdb-notice-hook)
! 	(setq bbdb-notice-hook save-bbdb-notice-hook))))
  
  (define-key gnus-summary-goto-map "G" 'gnus-summary-glimpse)

                                        Be seeing you,
                                          norm

-- 
Norman Walsh <ndw@nwalsh.com> | We dance around in a ring and suppose,
http://nwalsh.com/            | but the Secret sits in the middle and
                              | knows.--Robert Frost

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Kai Großjohann]

You might wish to try nnir.el which supports other search engines in
addition to gnus-glimpse.el.  But thanks for that patch, I think I
need to apply it to nnir.el, too.
kai
-- 
The arms should be held in a natural and unaffected way and never
be conspicuous. -- Revised Technique of Latin American Dancing

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Norman Walsh]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 580 bytes --]

/ Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) was heard to say:
| You might wish to try nnir.el which supports other search engines in

I've obviously set my expire time too small for ding. Where might I
find nnir?

                                        Be seeing you,
                                          norm

-- 
Norman Walsh <ndw@nwalsh.com> | The stone fell on the pitcher? Woe to
http://nwalsh.com/            | the pitcher. The pitcher fell on the
                              | stone? Woe to the pitcher.--Rabbinic
                              | Saying

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Francisco Solsona]

Norman Walsh <ndw@nwalsh.com> writes:

> / Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) was heard to say:
> | You might wish to try nnir.el which supports other search engines in
> 
> I've obviously set my expire time too small for ding. Where might I
> find nnir?

>From nnir.el:

,--------------------
| ;; The most recent version of this can always be fetched from the
| ;; following FTP site:                      
| ;; ls6-ftp.cs.uni-dortmund.de:/pub/src/emacs
`--------------------

--Francisco

Re: gnus-glimpse.el -- search a newsgroup using glimpse

[Norman Walsh]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 579 bytes --]

/ Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) was heard to say:
| You might wish to try nnir.el which supports other search engines in

Tres cool!

                                        Be seeing you,
                                          norm

-- 
Norman Walsh <ndw@nwalsh.com> | There has never been a perfect
http://nwalsh.com/            | government, because men have passions;
                              | and if they did not have passions,
                              | there would be no need for
                              | government.--Voltaire