Re: gnus-search & imap: always "CHARSET UTF-8" when literal+ is supported

[David Edmondson <dme@dme.org>]

On Friday, 2021-10-22 at 10:47:42 -07, Eric Abrahamsen wrote:

> David Edmondson <dme@dme.org> writes:
>
>> Using current emacs git head, talking to outlook.office365.com over
>> IMAP.
>>
>> Attempts to use gnus-search always fail with the server reporting:
>>
>> (("NO" ("BADCHARSET" "(US-ASCII)") "The" "specified" "charset" "is" "not" "supported."))
>>
>> Looking at gnus-search.el, `gnus-search-imap-search-command' always
>> sends CHARSET UTF-8 if the server supports literal+ (which this one
>> does). Sending US-ASCII (or no charset at all) causes the server to
>> return the required results in simple test cases.
>>
>> Is there a way to determine whether a server supports UTF-8 in searches,
>> and adjust the command sent accordingly? If not, could the use of UTF-8
>> be controlled with (yet another!) variable?
>
> A bit of internet research seems to indicate that Exchange can't handle
> UTF-8 encoded search strings, and also there's no way to test that in
> advance apart from simply seeing if it errors. Awesome!

That's also the impression I gained.

> I think what this means is that it's impossible to search for non-ascii
> text on an Exchange server (can that be true?!). If that's true, then
> the imap search command should be using the presence of a multibyte
> string as the test for whether to use CHARSET UTF-8 or not. You're not
> going to be able to search for a multibyte string, anyway.
>
> Would you try eval'ling the below, and tell me if it works okay when
> searching for a string with no non-ascii characters in it?

This works in a few simple tests, yes. Thanks!

> Also, when you do get the error message above, how does that present to
> the user? Did you have to go digging to find it?

I had to dig. The observed behaviour is that no messages match the
search.

> (cl-defmethod gnus-search-imap-search-command ((engine gnus-search-imap)
> 					       (query string))
>   "Create the IMAP search command for QUERY.
> Currently takes into account support for the LITERAL+ capability.
> Other capabilities could be tested here."
>   (with-slots (literal-plus) engine
>     (when (and literal-plus
> 	       (string-match-p "\n" query))
>       (setq query (split-string query "\n")))
>     (cond
>      ((consp query)
>       ;; We're not really streaming, just need to prevent
>       ;; `nnimap-send-command' from waiting for a response.
>       (let* ((nnimap-streaming t)
> 	     (call
> 	      (nnimap-send-command
> 	       "UID SEARCH CHARSET UTF-8 %s"
> 	       (pop query))))
> 	(dolist (l query)
> 	  (process-send-string (get-buffer-process (current-buffer)) l)
> 	  (process-send-string (get-buffer-process (current-buffer))
> 			       (if (nnimap-newlinep nnimap-object)
> 				   "\n"
> 				 "\r\n")))
> 	(nnimap-get-response call)))
>      (t (nnimap-command "UID SEARCH %s" query)))))

dme.
-- 
I've been waiting for so long, to come here now and sing this song.