nnir and the new nnimap

nnir and the new nnimap

[Knut Anders Hatlen]

Hi,

It looks like nnir has some dependencies on the old nnimap, and has
stopped working.

In ~/.gnus:

(require 'nnir)

(setq gnus-select-method
      '(nnimap "server"
               (nnimap-stream shell)
               (nnir-search-engine imap)))

In the group buffer:

G-G some query string RET
==> nnir-run-imap: Symbol's value as variable is void: nnimap-server-buffer


-- 
Knut Anders

Re: nnir and the new nnimap

[Lars Magne Ingebrigtsen]

Knut Anders Hatlen <kahatlen@gmail.com> writes:

> It looks like nnir has some dependencies on the old nnimap, and has
> stopped working.

I've never used nnir myself, but looking at it, it looks like it has
deep hooks into nnimap internals.  So it's not going to work any time
soon unless somebody whips it into shape again.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnir and the new nnimap

[Vegard Vesterheim]

On Tue, 21 Sep 2010 23:37:14 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:

> Knut Anders Hatlen <kahatlen@gmail.com> writes:
>
>> It looks like nnir has some dependencies on the old nnimap, and has
>> stopped working.
>
> I've never used nnir myself, 

I think nnir is great for IMAP searching, I use it all the
time. Dovecot seems to do a decent job in indexing mails, so
server-side search is reasonably fast.

How do you guys search in old emails?

> but looking at it, it looks like it has
> deep hooks into nnimap internals.  So it's not going to work any time
> soon unless somebody whips it into shape again.

Ouch, that means no improved imap speed for me any time soon then...

 - Vegard V -

Re: nnir and the new nnimap

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Knut Anders Hatlen <kahatlen@gmail.com> writes:
>> It looks like nnir has some dependencies on the old nnimap, and has
>> stopped working.

> I've never used nnir myself, but looking at it, it looks like it has
> deep hooks into nnimap internals.  So it's not going to work any time
> soon unless somebody whips it into shape again.

For those who would like to try:
 - Here is the command you need to make nnimap feed the server
	http://tools.ietf.org/html/rfc3501#section-6.4.4
 - The response is a list of sequence numbers, 
    http://tools.ietf.org/html/rfc3501#section-2.3.1.2
   which will possibly need to be transformed into UIDs
    http://tools.ietf.org/html/rfc3501#section-2.3.1.1
   for nnimap to make use of it...? 

For the one that chooses to reimplement it, two humble requests:
 - Make it possible to span more than one group (e.g. have a group
   parameter on the main group, that refers to the archive group(s) it
   expires its matches to)
 - Make it possible to warp from the match and into the nnimap group it
   is in, so that a found article can be ticked, and so that group
   parameters and gnus-posting-styles match the group

Re: nnir and the new nnimap

[Lars Magne Ingebrigtsen]

Vegard Vesterheim <vegard.vesterheim@uninett.no> writes:

> I think nnir is great for IMAP searching, I use it all the
> time. Dovecot seems to do a decent job in indexing mails, so
> server-side search is reasonably fast.
>
> How do you guys search in old emails?

Search is too newfangled for me.

>> but looking at it, it looks like it has
>> deep hooks into nnimap internals.  So it's not going to work any time
>> soon unless somebody whips it into shape again.
>
> Ouch, that means no improved imap speed for me any time soon then...

I'll take a look at it if nobody else volunteers...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnir and the new nnimap

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> For the one that chooses to reimplement it, two humble requests:
>  - Make it possible to span more than one group (e.g. have a group
>    parameter on the main group, that refers to the archive group(s) it
>    expires its matches to)

IMAP SEARCH is only done on one group at a time, so searching N groups
will takes N times as much time as searching 1 group.

>  - Make it possible to warp from the match and into the nnimap group it
>    is in, so that a found article can be ticked, and so that group
>    parameters and gnus-posting-styles match the group

Hm...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnir and the new nnimap

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> IMAP SEARCH is only done on one group at a time, so searching N groups
> will takes N times as much time as searching 1 group.

In my case N == 2, and the largest number of articles are in group
number 2, while group number 1 is the logical group, with the expected
group parameters and gnus-posting-styles.

The reason for this pattern is that once upon a time, nnimap had speed
problems with a large number of \Seen articles, interspersed with the
odd ticked articles.

If I could move most of the articles to an archive group I could access
them if I absolutely _had_ to, and have much improved speed for the
newest and most relevant articles.

Re: nnir and the new nnimap

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Search is too newfangled for me.

Really?  So you still keep an up-to-date bookmarks list, and doesn't
bother with that newfangled google stuff?

Re: nnir and the new nnimap

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Really?  So you still keep an up-to-date bookmarks list, and doesn't
> bother with that newfangled google stuff?

Get off my lawn, you young whippersnappers!

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnir and the new nnimap

[Lars Magne Ingebrigtsen]

I had a peek at nnir.  It seems kinda... under-documented.

What does an nnir method using nnimap look like?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnir and the new nnimap

[Knut Anders Hatlen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I had a peek at nnir.  It seems kinda... under-documented.
>
> What does an nnir method using nnimap look like?

It's just an ordinary nnimap method that includes (nnir-search-engine
imap). For example:

       (nnimap "my-server"
               (nnimap-stream shell)
               (nnir-search-engine imap))

Then G-G in the group buffer should do what you want. Well, at least it
used to... :)

-- 
Knut Anders

Re: nnir and the new nnimap

[Lars Magne Ingebrigtsen]

Knut Anders Hatlen <kahatlen@gmail.com> writes:

> It's just an ordinary nnimap method that includes (nnir-search-engine
> imap). For example:
>
>        (nnimap "my-server"
>                (nnimap-stream shell)
>                (nnir-search-engine imap))
>
> Then G-G in the group buffer should do what you want. Well, at least it
> used to... :)

`G G' does nothing in the group buffer for me.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnir and the new nnimap

[Knut Anders Hatlen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Knut Anders Hatlen <kahatlen@gmail.com> writes:
>
>> It's just an ordinary nnimap method that includes (nnir-search-engine
>> imap). For example:
>>
>>        (nnimap "my-server"
>>                (nnimap-stream shell)
>>                (nnir-search-engine imap))
>>
>> Then G-G in the group buffer should do what you want. Well, at least it
>> used to... :)
>
> `G G' does nothing in the group buffer for me.

You may also need (require 'nnir). And restart Gnus, I think.

-- 
Knut Anders

Re: nnir and the new nnimap

[Lars Magne Ingebrigtsen]

Knut Anders Hatlen <kahatlen@gmail.com> writes:

> You may also need (require 'nnir). And restart Gnus, I think.

Thanks.  Kinda irregular interface, then...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnir and the new nnimap

[Andrew Cohen]

[-- Attachment #1: Type: text/plain, Size: 466 bytes --]

I rely on nnir so I thought I would just try to get it working with
minimal rewriting.  I haven't tested this much, but it worked on a few
simple searches. The main change was to nnir-run-imap, but I also had to
make one change to nnimap-retrieve-headers that worries me. Currently
this function returns 't, and nnir wants it to return 'headers. I made
the change and nothing exploded. Is this OK?

Of course all this needs to be rewritten, but in the meantime...



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnir.el patch for new imap --]
[-- Type: text/x-diff, Size: 1728 bytes --]

--- nnir.el	2010-09-22 08:02:59.000000000 -0400
+++ nnir.new.el	2010-09-23 08:35:07.000000000 -0400
@@ -967,23 +967,27 @@
 	  (defs (caddr (gnus-server-to-method srv)))
 	  (criteria (or (cdr (assq 'criteria query))
 			nnir-imap-search-field))
-	  artlist buf)
+	  (gnus-inhibit-demon t)
+	  artlist)
       (message "Opening server %s" server)
       (condition-case ()
-	  (when (nnimap-open-server server defs) ;; xxx
-	    (setq buf nnimap-server-buffer) ;; xxx
-	    (message "Searching %s..." group)
-            (let ((arts 0)
-                  (mbx (gnus-group-real-name group)))
-              (when (imap-mailbox-select mbx nil buf)
-                (mapc
-                 (lambda (artnum)
-                   (push (vector group artnum 1) artlist)
-                   (setq arts (1+ arts)))
-                 (imap-search (nnir-imap-make-query criteria qstring) buf))
-                (message "Searching %s... %d matches" mbx arts)))
-            (message "Searching %s...done" group))
-        (quit nil))
+	  (when (nnimap-possibly-change-group (gnus-group-short-name group) server)
+	    (with-current-buffer (nnimap-buffer)
+	      (message "Searching %s..." group)
+	      (let ((arts 0)
+		    (result
+		     (nnimap-command "UID SEARCH  %s" 
+				     (nnir-imap-make-query criteria qstring))))
+		(mapc
+		 (lambda (artnum)
+		   (push (vector group artnum 1) artlist)
+		   (setq arts (1+ arts)))
+		 (and (car result)
+		      (delete 0 (mapcar #'string-to-number
+					(cdr (assoc "SEARCH" (cdr result)))))))
+		(message "Searching %s... %d matches" group arts)))
+	    (message "Searching %s...done" group))
+	(quit nil))
       (reverse artlist))))
 
 (defun nnir-imap-make-query (criteria qstring)

Re: nnir and the new nnimap

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> I rely on nnir so I thought I would just try to get it working with
> minimal rewriting.  I haven't tested this much, but it worked on a few
> simple searches. The main change was to nnir-run-imap, but I also had to
> make one change to nnimap-retrieve-headers that worries me. Currently
> this function returns 't, and nnir wants it to return 'headers. I made
> the change and nothing exploded. Is this OK?

Looks OK to me.  I've applied it.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnir and the new nnimap

[Andrew Cohen]

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

    Lars> Andrew Cohen <cohen@andy.bu.edu> writes:
    >> I rely on nnir so I thought I would just try to get it working
    >> with minimal rewriting.  I haven't tested this much, but it
    >> worked on a few simple searches. The main change was to
    >> nnir-run-imap, but I also had to make one change to
    >> nnimap-retrieve-headers that worries me. Currently this function
    >> returns 't, and nnir wants it to return 'headers. I made the
    >> change and nothing exploded. Is this OK?

    Lars> Looks OK to me.  I've applied it.

and the change to nnimap-retrieve-headers didn't get applied, probably
because I forgot to include it in my patch :)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnimap-retrieve-headers change --]
[-- Type: text/x-diff, Size: 332 bytes --]

--- nnimap.el	2010-09-23 16:58:05.000000000 -0400
+++ nnimap.new.el	2010-09-23 17:07:28.000000000 -0400
@@ -135,7 +135,7 @@
 	(nnimap-transform-headers))
       (insert-buffer-substring
        (nnimap-find-process-buffer (current-buffer))))
-    t))
+    'headers))
 
 (defun nnimap-transform-headers ()
   (goto-char (point-min))

Re: nnir and the new nnimap

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> and the change to nnimap-retrieve-headers didn't get applied, probably
> because I forgot to include it in my patch :)

:-)  Applied and pushed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnir and the new nnimap

[Vegard Vesterheim]

On Wed, 22 Sep 2010 18:39:58 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:

> Steinar Bang <sb@dod.no> writes:
>
>> For the one that chooses to reimplement it, two humble requests:
>>  - Make it possible to span more than one group (e.g. have a group
>>    parameter on the main group, that refers to the archive group(s) it
>>    expires its matches to)
>
> IMAP SEARCH is only done on one group at a time, so searching N groups
> will takes N times as much time as searching 1 group.

I'm happy to report that IMAP search now seem to work for me, even
across groups. Great!

I have, however, done a simple modification to the actual search
function (nnir-run-imap).  I like to express my search criteria in IMAP
syntax directly as IMAP itself supports a query language of sorts. I am
not sure I see the need to invent another, more limited,
mini-query-language implemented in elisp, at least not until it can be
supported with a nice GUI for building complex queries.

The native IMAP search method allows both simple and complex queries, like.

     "from lars"
     "from lars since 20-Sep-2010"
     "from lars since 20-Sep-2010 subject imap"
     "(or from lars from vegard) since 20-Sep-2010 subject imap"
     "(not (or from lars from vegard)) since 20-Sep-2010 subject imap"
     "subject imap body query"

>>  - Make it possible to warp from the match and into the nnimap group it
>>    is in, so that a found article can be ticked, and so that group
>>    parameters and gnus-posting-styles match the group
>
> Hm...

Yes, this would be *very* useful. The function bound to 'G T'
(gnus-summary-nnir-goto-thread) is supposed to do that, but it doesn't
work for me.

 - Vegard V -

Re: nnir and the new nnimap

[Steinar Bang]

>>>>> Vegard Vesterheim <vegard.vesterheim@uninett.no>:

>>> - Make it possible to warp from the match and into the nnimap group it
>>> is in, so that a found article can be ticked, and so that group
>>> parameters and gnus-posting-styles match the group
>> 
>> Hm...

> Yes, this would be *very* useful. The function bound to 'G T'
> (gnus-summary-nnir-goto-thread) is supposed to do that, but it doesn't
> work for me.

`G T' doesn't work for me either.

But note that I don't neccessarily need (or even want) the
find-full-thread functionality of `G T'.  Being able to move into the
group and tick the article, or just reply to the article with all
group-specific response stuff set up, would be plenty enough for me.

Re: nnir and the new nnimap

[Lars Magne Ingebrigtsen]

Vegard Vesterheim <vegard.vesterheim@uninett.no> writes:

> I have, however, done a simple modification to the actual search
> function (nnir-run-imap).  I like to express my search criteria in IMAP
> syntax directly as IMAP itself supports a query language of sorts. I am
> not sure I see the need to invent another, more limited,
> mini-query-language implemented in elisp, at least not until it can be
> supported with a nice GUI for building complex queries.
>
> The native IMAP search method allows both simple and complex queries, like.
>
>      "from lars"
>      "from lars since 20-Sep-2010"
>      "from lars since 20-Sep-2010 subject imap"
>      "(or from lars from vegard) since 20-Sep-2010 subject imap"
>      "(not (or from lars from vegard)) since 20-Sep-2010 subject imap"
>      "subject imap body query"

I think that sounds very useful.  Could you post the patch here?  And
have you (or are you willing to) sign copyright assignment papers for
the code to the FSF (if the code is long-ish)?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnir and the new nnimap

[Andrew Cohen]

[-- Attachment #1: Type: text/plain, Size: 1308 bytes --]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

    Lars> Vegard Vesterheim <vegard.vesterheim@uninett.no> writes:
    >> I have, however, done a simple modification to the actual search
    >> function (nnir-run-imap).  I like to express my search criteria
    >> in IMAP syntax directly as IMAP itself supports a query language
    >> of sorts. I am not sure I see the need to invent another, more
    >> limited, mini-query-language implemented in elisp, at least not
    >> until it can be supported with a nice GUI for building complex
    >> queries.
    >> 

    Lars> I think that sounds very useful.  Could you post the patch
    Lars> here?  And have you (or are you willing to) sign copyright
    Lars> assignment papers for the code to the FSF (if the code is
    Lars> long-ish)?


I thought this sounded useful as well, so here is a very simple
implementation. If you use "G G" in the group buffer with a prefix arg
you get queried for the search string, and then queried for the field to
search in (e.g. Subject, From, Header). I've changed this so that if you
enter nothing for the field it interprets the search as a raw imap
search.

If you want raw imap searches to be the default for all queries (even
without using a prefix arg) just set nnir-imap-search-field to "".


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnir.el --]
[-- Type: text/x-diff, Size: 1199 bytes --]

diff --git a/lisp/nnir.el b/lisp/nnir.el
index ec22c78..db8b397 100644
--- a/lisp/nnir.el
+++ b/lisp/nnir.el
@@ -345,14 +345,16 @@
 (gnus-declare-backend "nnir" 'mail)
 
 (defvar nnir-imap-search-field "TEXT"
-  "The IMAP search item when doing an nnir search")
+  "The IMAP search item when doing an nnir search. To use raw
+  imap queries by default set this to \"\"")
 
 (defvar nnir-imap-search-arguments
   '(("Whole message" . "TEXT")
     ("Subject" . "SUBJECT")
     ("To" . "TO")
     ("From" . "FROM")
-    (nil . "HEADER \"%s\""))
+    ("Head" . "HEADER \"%s\"")
+    (nil . ""))
   "Mapping from user readable strings to IMAP search items for use in nnir")
 
 (defvar nnir-imap-search-argument-history ()
@@ -981,8 +983,11 @@ details on the language and supported extensions"
 	      (message "Searching %s..." group)
 	      (let ((arts 0)
 		    (result
-		     (nnimap-command "UID SEARCH  %s" 
-				     (nnir-imap-make-query criteria qstring))))
+		     (nnimap-command "UID SEARCH  %s"
+				     (if (string= criteria "")
+					 qstring
+				       (nnir-imap-make-query criteria qstring)
+				       ))))
 		(mapc
 		 (lambda (artnum)
 		   (push (vector group artnum 1) artlist)

Re: nnir and the new nnimap

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> If you want raw imap searches to be the default for all queries (even
> without using a prefix arg) just set nnir-imap-search-field to "".

Thanks; applied.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen