nnir imap with gmail: query encoding

nnir imap with gmail: query encoding

[Carlos Pita]

Hi all,

I'm having some trouble getting nnir to work with non ascii queries.

In my gmail account there is an email containing the word "diría" in the
"facultad" folder. When I search that folder for "diría" there are no
matches. Moreover, when I search the same folder for "diria" there are
no matches either. So I have no way to search the folder for that word.

Is that supported? Should I configure anything encoding-related?

Cheers
--
Carlos

Re: nnir imap with gmail: query encoding

[Carlos Pita]

Doing a raw imap search won't do the trick either:

CHARSET ISO-8859-1 BODY "diría"  -> no matches
CHARSET UTF-8 BODY "diría"  -> no matches

CHARSET ISO-8859-1 BODY "que"  -> fine, lot of matches
CHARSET UTF-8 BODY "que"  -> fin, lot of matches

Here is some info about my emacs encoding:

Coding system for saving this buffer:
  U -- utf-8-emacs

Default coding system (for new files):
  U -- utf-8-unix (alias: mule-utf-8-unix)

Coding system for keyboard input:
  U -- utf-8-unix (alias: mule-utf-8-unix)

Coding system for terminal output:
  U -- utf-8-unix (alias: mule-utf-8-unix)

Coding system for inter-client cut and paste:
  nil
Defaults for subprocess I/O:
  decoding: U -- utf-8-unix (alias: mule-utf-8-unix)

  encoding: U -- utf-8-unix (alias: mule-utf-8-unix)


Priority order for recognizing coding systems when reading files:
  1. utf-8 (alias: mule-utf-8)
  2. iso-2022-7bit 
  3. iso-latin-1 (alias: iso-8859-1 latin-1)
  4. iso-2022-7bit-lock (alias: iso-2022-int-1)
  5. iso-2022-8bit-ss2 
  6. emacs-mule 
  7. raw-text 
  8. iso-2022-jp (alias: junet)
  9. in-is13194-devanagari (alias: devanagari)
  10. chinese-iso-8bit (alias: cn-gb-2312 euc-china euc-cn cn-gb gb2312)
  11. utf-8-auto 
  12. utf-8-with-signature 
  13. utf-16 
  14. utf-16be-with-signature (alias: utf-16-be)
  15. utf-16le-with-signature (alias: utf-16-le)
  16. utf-16be 
  17. utf-16le 
  18. japanese-shift-jis (alias: shift_jis sjis)
  19. chinese-big5 (alias: big5 cn-big5 cp950)
  20. undecided 


Carlos Pita <carlosjosepita@gmail.com> writes:

> Hi all,
>
> I'm having some trouble getting nnir to work with non ascii queries.
>
> In my gmail account there is an email containing the word "diría" in the
> "facultad" folder. When I search that folder for "diría" there are no
> matches. Moreover, when I search the same folder for "diria" there are
> no matches either. So I have no way to search the folder for that word.
>
> Is that supported? Should I configure anything encoding-related?
>
> Cheers
> --
> Carlos

Re: nnir imap with gmail: query encoding

[Eric Abrahamsen]

Carlos Pita <carlosjosepita@gmail.com> writes:

> Doing a raw imap search won't do the trick either:
>
> CHARSET ISO-8859-1 BODY "diría"  -> no matches
> CHARSET UTF-8 BODY "diría"  -> no matches
>
> CHARSET ISO-8859-1 BODY "que"  -> fine, lot of matches
> CHARSET UTF-8 BODY "que"  -> fin, lot of matches

Are you doing the above in a telnet session, or where? IMAP uses a
variant of UTF-7 as its internal coding, maybe try with that and see
what happens?

FWIW, it doesn't look like nnir/IMAP work for me here, either, searching
for Chinese characters.

Eric

> Here is some info about my emacs encoding:
>
> Coding system for saving this buffer:
>   U -- utf-8-emacs
>
> Default coding system (for new files):
>   U -- utf-8-unix (alias: mule-utf-8-unix)
>
> Coding system for keyboard input:
>   U -- utf-8-unix (alias: mule-utf-8-unix)
>
> Coding system for terminal output:
>   U -- utf-8-unix (alias: mule-utf-8-unix)
>
> Coding system for inter-client cut and paste:
>   nil
> Defaults for subprocess I/O:
>   decoding: U -- utf-8-unix (alias: mule-utf-8-unix)
>
>   encoding: U -- utf-8-unix (alias: mule-utf-8-unix)
>
>
> Priority order for recognizing coding systems when reading files:
>   1. utf-8 (alias: mule-utf-8)
>   2. iso-2022-7bit 
>   3. iso-latin-1 (alias: iso-8859-1 latin-1)
>   4. iso-2022-7bit-lock (alias: iso-2022-int-1)
>   5. iso-2022-8bit-ss2 
>   6. emacs-mule 
>   7. raw-text 
>   8. iso-2022-jp (alias: junet)
>   9. in-is13194-devanagari (alias: devanagari)
>   10. chinese-iso-8bit (alias: cn-gb-2312 euc-china euc-cn cn-gb gb2312)
>   11. utf-8-auto 
>   12. utf-8-with-signature 
>   13. utf-16 
>   14. utf-16be-with-signature (alias: utf-16-be)
>   15. utf-16le-with-signature (alias: utf-16-le)
>   16. utf-16be 
>   17. utf-16le 
>   18. japanese-shift-jis (alias: shift_jis sjis)
>   19. chinese-big5 (alias: big5 cn-big5 cp950)
>   20. undecided 
>
>
> Carlos Pita <carlosjosepita@gmail.com> writes:
>
>> Hi all,
>>
>> I'm having some trouble getting nnir to work with non ascii queries.
>>
>> In my gmail account there is an email containing the word "diría" in the
>> "facultad" folder. When I search that folder for "diría" there are no
>> matches. Moreover, when I search the same folder for "diria" there are
>> no matches either. So I have no way to search the folder for that word.
>>
>> Is that supported? Should I configure anything encoding-related?
>>
>> Cheers
>> --
>> Carlos

Re: nnir imap with gmail: query encoding

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

    Eric> Carlos Pita <carlosjosepita@gmail.com> writes:
    >> Doing a raw imap search won't do the trick either:
    >> 
    >> CHARSET ISO-8859-1 BODY "diría" -> no matches CHARSET UTF-8 BODY
    >> "diría" -> no matches
    >> 
    >> CHARSET ISO-8859-1 BODY "que" -> fine, lot of matches CHARSET
    >> UTF-8 BODY "que" -> fin, lot of matches

    Eric> Are you doing the above in a telnet session, or where? IMAP
    Eric> uses a variant of UTF-7 as its internal coding, maybe try with
    Eric> that and see what happens?

    Eric> FWIW, it doesn't look like nnir/IMAP work for me here, either,
    Eric> searching for Chinese characters.

    Eric> Eric


This isn't currently supported in nnir directly, but can be done with a
raw imap query. As I recall the trick is you can't use a quoted string
but have to use a literal; literals need to include the number of octets
(including CR and LF) in brackets following the search term (this is
what I understand from reading the imap RFC).

Re: nnir imap with gmail: query encoding

[Carlos Pita]

>     Eric> Are you doing the above in a telnet session, or where? IMAP
>     Eric> uses a variant of UTF-7 as its internal coding, maybe try with
>     Eric> that and see what happens?
>
>     Eric> FWIW, it doesn't look like nnir/IMAP work for me here, either,
>     Eric> searching for Chinese characters.
>
>     Eric> Eric
>
> This isn't currently supported in nnir directly, but can be done with a
> raw imap query. As I recall the trick is you can't use a quoted string
> but have to use a literal; literals need to include the number of octets
> (including CR and LF) in brackets following the search term (this is
> what I understand from reading the imap RFC).

I've also tested it using python imaplib and there is no way to encode
the query inline, gmail will complain about being unable to parse it as
soon as you move outside the ascii charset, no matter the
enconding. Indeed, you have to use a literal to get it working, but I
have no idea how to patch nnir in order to do that.

Cheers
--
Carlos

Re: nnir imap with gmail: query encoding

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 2725 bytes --]

Andrew Cohen <cohen@bu.edu> writes:

>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>     Eric> Carlos Pita <carlosjosepita@gmail.com> writes:
>     >> Doing a raw imap search won't do the trick either:
>     >> 
>     >> CHARSET ISO-8859-1 BODY "diría" -> no matches CHARSET UTF-8 BODY
>     >> "diría" -> no matches
>     >> 
>     >> CHARSET ISO-8859-1 BODY "que" -> fine, lot of matches CHARSET
>     >> UTF-8 BODY "que" -> fin, lot of matches
>
>     Eric> Are you doing the above in a telnet session, or where? IMAP
>     Eric> uses a variant of UTF-7 as its internal coding, maybe try with
>     Eric> that and see what happens?
>
>     Eric> FWIW, it doesn't look like nnir/IMAP work for me here, either,
>     Eric> searching for Chinese characters.
>
>     Eric> Eric
>
>
> This isn't currently supported in nnir directly, but can be done with a
> raw imap query. As I recall the trick is you can't use a quoted string
> but have to use a literal; literals need to include the number of octets
> (including CR and LF) in brackets following the search term (this is
> what I understand from reading the imap RFC).

Here's a patch that (mostly) makes this work. It's probably not ready
for application, but I'm fairly confident that the basic approach is
sound. I had to make changes to nnir-run-imap, because literals have to
be fed to the server on separate lines.

Things that bear examination:

1. Raw imap queries are still not touched. You couldn't put non-ascii
search terms in raw queries anyway, because, as mentioned above, they
have to be sent as separate lines to the server, and the current search
routine can't do that.

2. The 'coding' let-var that gets sent to the search command (the
argument to CHARSET) is a total guess on my part. It should be easy to
fix if my guess is wrong, though. Do we need to care about the coding
system used in the server's message storage?

3. I've only tested this with my local dovecot (2.2.13). Imap servers
are weird, and edge cases abound. It would be nice if someone could test
this directly with Gmail and Exchange, at least.


What are the future plans for nnir's mini-imap search language? Right
now it doesn't seem like a whole lot is gained by the
nnir-imap-search-arguments/default-search-key setup. It seems like it
would be simpler and more flexible to allow all valid imap search
criteria as lower-cased keys followed by a colon, which would make the
language look a little bit more like notmuch or what have you. Unknown
keys would be considered header values. Or something else like that --
mostly it just seems very limiting that we can specify at most one
criteria to search on.

Anyway, just curious what the plan is.

Eric



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0003-Enable-non-ascii-search-terms-in-imap-searches.patch --]
[-- Type: text/x-diff, Size: 5318 bytes --]

From bb2fd201ed154bebd58059a8de8c991fef1baf1b Mon Sep 17 00:00:00 2001
From: Eric Abrahamsen <eric@ericabrahamsen.net>
Date: Sat, 6 Sep 2014 11:58:19 +0800
Subject: [PATCH 3/3] Enable non-ascii search terms in imap searches

* lisp/nnir.el (nnir-run-imap): Feed search queries by separate lines,
  if necessary.
  (nnir-imap-next-term): Detect and collect byte lengths of non-ascii
  strings.
  (nnir-imap-expr-to-imap): Create an intermediate querystring format
  for non-ascii terms.
---
 lisp/nnir.el | 84 +++++++++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 57 insertions(+), 27 deletions(-)

diff --git a/lisp/nnir.el b/lisp/nnir.el
index e2051df..aba4c58 100644
--- a/lisp/nnir.el
+++ b/lisp/nnir.el
@@ -284,6 +284,8 @@ is `(valuefunc member)'."
 (eval-when-compile
   (autoload 'nnimap-buffer "nnimap")
   (autoload 'nnimap-command "nnimap")
+  (autoload 'nnimap-capability "nnimap")
+  (autoload 'nnimap-wait-for-line "nnimap")
   (autoload 'nnimap-change-group "nnimap")
   (autoload 'nnimap-make-thread-query "nnimap")
   (autoload 'gnus-registry-action "gnus-registry")
@@ -968,33 +970,52 @@ details on the language and supported extensions."
        (catch 'found
          (mapcar
           #'(lambda (group)
-            (let (artlist)
-              (condition-case ()
-                  (when (nnimap-change-group
-                         (gnus-group-short-name group) server)
-                    (with-current-buffer (nnimap-buffer)
-                      (message "Searching %s..." group)
-                      (let ((arts 0)
-                            (result (nnimap-command "UID SEARCH %s"
-                                                    (if (string= criteria "")
-                                                        qstring
-                                                      (nnir-imap-make-query
-                                                       criteria qstring)))))
-                        (mapc
-                         (lambda (artnum)
-                           (let ((artn (string-to-number artnum)))
-                             (when (> artn 0)
-                               (push (vector group artn 100)
-                                     artlist)
-                               (when (assq 'shortcut query)
-                                 (throw 'found (list artlist)))
-                               (setq arts (1+ arts)))))
-                         (and (car result)
-			      (cdr (assoc "SEARCH" (cdr result)))))
-                        (message "Searching %s... %d matches" group arts)))
-                    (message "Searching %s...done" group))
-                (quit nil))
-              (nreverse artlist)))
+	      (let (artlist)
+		(condition-case ()
+		    (when (nnimap-change-group
+			   (gnus-group-short-name group) server)
+		      (with-current-buffer (nnimap-buffer)
+			(message "Searching %s..." group)
+			(let* ((arts 0)
+			       (literal+ (nnimap-capability "LITERAL+"))
+			       (search (split-string
+					(if (string= criteria "")
+					    qstring
+					  (nnir-imap-make-query
+					  criteria qstring))
+					"\n"))
+			       (coding (upcase
+					(replace-regexp-in-string
+					 "-\\(unix\\|dos\\|mac\\)" ""
+					 (symbol-name
+					  (cdr default-process-coding-system)))))
+			       call result)
+			  (setq call (nnimap-send-command
+					"UID SEARCH CHARSET %s %s" coding (pop search)))
+			  (while search ; Non-ascii search terms
+			    (unless literal+
+			      (nnimap-wait-for-line "^\\+\\(.*\\)\n"))
+			    (process-send-string (get-buffer-process (current-buffer)) (pop search))
+			    (process-send-string (get-buffer-process (current-buffer))
+			       (if (nnimap-newlinep nnimap-object)
+				   "\n"
+				 "\r\n")))
+			  (setq result (nnimap-get-response call))
+			  (mapc
+			   (lambda (artnum)
+			     (let ((artn (string-to-number artnum)))
+			       (when (> artn 0)
+				 (push (vector group artn 100)
+				       artlist)
+				 (when (assq 'shortcut query)
+				   (throw 'found (list artlist)))
+				 (setq arts (1+ arts)))))
+			   (and (car result)
+				(cdr (assoc "SEARCH" (cdr result)))))
+			  (message "Searching %s... %d matches" group arts)))
+		      (message "Searching %s...done" group))
+		  (quit nil))
+		(nreverse artlist)))
           groups))))))
 
 (defun nnir-imap-make-query (criteria qstring)
@@ -1062,6 +1083,10 @@ In future the following will be added to the language:
    ;; Composite term: just the fax, mam
    ((eq (car-safe expr) 'not)
     (format "NOT (%s)" (nnir-imap-query-to-imap criteria (rest expr))))
+   ;; Composite term: non-ascii search term
+   ((numberp (car-safe expr))
+    (format "%s {%d%s}\n%s" criteria (car expr)
+	    (if literal+ "+" "") (second expr)))
    ;; Composite term: just expand it all.
    ((and (not (null expr)) (listp expr))
     (format "(%s)" (nnir-imap-query-to-imap criteria expr)))
@@ -1108,6 +1133,11 @@ that the search language can then understand and use."
      ((eq term 'and) 'and)
      ;; negated term
      ((eq term 'not) (list 'not (nnir-imap-next-expr)))
+     ;; non-ascii search string
+     ((and (stringp term)
+	   (not (= (string-bytes term)
+		   (length term))))
+      (list (string-bytes term) term))
      ;; generic term
      (t term))))
 
-- 
2.1.0

Re: nnir imap with gmail: query encoding

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> 2. The 'coding' let-var that gets sent to the search command (the
> argument to CHARSET) is a total guess on my part. It should be easy to
> fix if my guess is wrong, though. Do we need to care about the coding
> system used in the server's message storage?
>
> 3. I've only tested this with my local dovecot (2.2.13). Imap servers
> are weird, and edge cases abound. It would be nice if someone could test
> this directly with Gmail and Exchange, at least.

I've applied your patch, so we'll get some testing.  :-)

> What are the future plans for nnir's mini-imap search language? Right
> now it doesn't seem like a whole lot is gained by the
> nnir-imap-search-arguments/default-search-key setup. It seems like it
> would be simpler and more flexible to allow all valid imap search
> criteria as lower-cased keys followed by a colon, which would make the
> language look a little bit more like notmuch or what have you. Unknown
> keys would be considered header values. Or something else like that --
> mostly it just seems very limiting that we can specify at most one
> criteria to search on.
>
> Anyway, just curious what the plan is.

A simpler search language would be nice...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: nnir imap with gmail: query encoding

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> 2. The 'coding' let-var that gets sent to the search command (the
>> argument to CHARSET) is a total guess on my part. It should be easy to
>> fix if my guess is wrong, though. Do we need to care about the coding
>> system used in the server's message storage?
>>
>> 3. I've only tested this with my local dovecot (2.2.13). Imap servers
>> are weird, and edge cases abound. It would be nice if someone could test
>> this directly with Gmail and Exchange, at least.
>
> I've applied your patch, so we'll get some testing.  :-)

Would this be the source of the flood of nastygrams I'm getting from the
buildbot? I guess we *did* get some testing!

>> What are the future plans for nnir's mini-imap search language? Right
>> now it doesn't seem like a whole lot is gained by the
>> nnir-imap-search-arguments/default-search-key setup. It seems like it
>> would be simpler and more flexible to allow all valid imap search
>> criteria as lower-cased keys followed by a colon, which would make the
>> language look a little bit more like notmuch or what have you. Unknown
>> keys would be considered header values. Or something else like that --
>> mostly it just seems very limiting that we can specify at most one
>> criteria to search on.
>>
>> Anyway, just curious what the plan is.
>
> A simpler search language would be nice...

And a unified one, that the nnir could translate for its various
backends.

You'll forgive me if my patch-writing confidence is at a low ebb,
though. I've had a pretty poor batting average of late.

Re: nnir imap with gmail: query encoding

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Would this be the source of the flood of nastygrams I'm getting from the
> buildbot? I guess we *did* get some testing!

Yup.  :-)  I think I've fixed them now, but have a look over the code.

> You'll forgive me if my patch-writing confidence is at a low ebb,
> though. I've had a pretty poor batting average of late.

Never mind the buildbot.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: nnir imap with gmail: query encoding

[Lars Ingebrigtsen]

The remaining problem is that `default-process-coding-system' isn't
defined in XEmacs.  Can that part of the patch be done in a different
way?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: nnir imap with gmail: query encoding

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> The remaining problem is that `default-process-coding-system' isn't
> defined in XEmacs.  Can that part of the patch be done in a different
> way?

I'll take a look tonight!