Recent nnir update broke search on office365

Recent nnir update broke search on office365

[Dave Goldberg]

All searches come up empty since this commit (per git-bisect)

046825b09ff6e23ec7dfacd451d10985615b9793 is the first bad commit
commit 046825b09ff6e23ec7dfacd451d10985615b9793
Author: Eric Abrahamsen <eric@ericabrahamsen.net>
Date:   Tue Jan 27 16:35:25 2015 +1100

    * nnir.el (nnir-run-imap): Enable non-ASCII IMAP searches.

:040000 040000 5131b4d4212b134d58fdaec4481a41239a69926c
e22212e758aa6a15e13c23ec8fb00f65bcf3a741 M lisp

I should have thought to check my other server (cyrus) - I can do so when I get back to work next week.  I'm happy to help troubleshoot but need guidance on what to look for.

Thanks,
-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: Recent nnir update broke search on office365

[Eric Abrahamsen]

david.goldberg6@verizon.net (Dave Goldberg) writes:

> All searches come up empty since this commit (per git-bisect)
>
> 046825b09ff6e23ec7dfacd451d10985615b9793 is the first bad commit
> commit 046825b09ff6e23ec7dfacd451d10985615b9793
> Author: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date:   Tue Jan 27 16:35:25 2015 +1100
>
>     * nnir.el (nnir-run-imap): Enable non-ASCII IMAP searches.
>
> :040000 040000 5131b4d4212b134d58fdaec4481a41239a69926c
> e22212e758aa6a15e13c23ec8fb00f65bcf3a741 M lisp
>
> I should have thought to check my other server (cyrus) - I can do so
> when I get back to work next week. I'm happy to help troubleshoot but
> need guidance on what to look for.

Yeah, this probably wasn't quite ready. There are a couple of other
problems with that patch that I'm looking at now, and hopefully I'll be
able to come up with something that addresses this as well, if I can
figure out what office365 is doing for encoding. If you have any hints
in that direction, let me know! Sorry for the breakage...

E

Re: Recent nnir update broke search on office365

[Eric Abrahamsen]

david.goldberg6@verizon.net (Dave Goldberg) writes:

> All searches come up empty since this commit (per git-bisect)
>
> 046825b09ff6e23ec7dfacd451d10985615b9793 is the first bad commit
> commit 046825b09ff6e23ec7dfacd451d10985615b9793
> Author: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date:   Tue Jan 27 16:35:25 2015 +1100
>
>     * nnir.el (nnir-run-imap): Enable non-ASCII IMAP searches.
>
> :040000 040000 5131b4d4212b134d58fdaec4481a41239a69926c
> e22212e758aa6a15e13c23ec8fb00f65bcf3a741 M lisp
>
> I should have thought to check my other server (cyrus) - I can do so
> when I get back to work next week. I'm happy to help troubleshoot but
> need guidance on what to look for.

The main problem here is that we're feeding a CHARSET argument to the
imap SEARCH command, but I really had no idea how to determine the
appropriate charset. Right now it's done using a value derived from the
`default-process-coding-system', which is bad because:

1. That variable doesn't exist in xemacs
2. It's probably just the wrong thing to do.

I guess what would be most helpful is if you could tell me your value
for default-process-coding-system, and then do a telnet session with
your exchange server. I'm not sure about exchange but, assuming you're
connecting with ssl, here's what I do with gmail:

openssl s_client -crlf -connect imap.gmail.com:993

Once you're in:

. login email@address.com password
. select INBOX ; or whatever mailbox

(if hitting return doesn't seem to produce a response, try leaving out
the -crlf argument to openssl).

Then you should be able to search with:

. search charset UTF-8 subject "search string"

What I'd like to know is how the different charset arguments affect the
result. Find a string you know is in one of the headers, and try to
search with a few different charsets:

UTF-8
WINDOWS-1252
US-ASCII
ISO-8859-1
whatever you've got in 'default-process-coding-system'

And just see what comes up.

It might also be nice to see the CAPABILITY line that the server gives
you after you connect.

Sorry to give you all this homework! Until someone more knowledgeable
steps up, we're stuck with me muddling through :)

Eric

Re: Recent nnir update broke search on office365

[Eric S Fraga]

On Sunday,  1 Feb 2015 at 14:22, Eric Abrahamsen wrote:
> david.goldberg6@verizon.net (Dave Goldberg) writes:
>
>> All searches come up empty since this commit (per git-bisect)

This happens to me as well in latest gnus (as of yesterday, that
is).  I've backed up to a version that works for now.

[...]

> I guess what would be most helpful is if you could tell me your value
> for default-process-coding-system, and then do a telnet session with
> your exchange server. I'm not sure about exchange but, assuming you're
> connecting with ssl, here's what I do with gmail:

To give you a data point, I have followed your instructions for the IMAP
server I use which is outlook.office365.com (Microsoft :().  The only
charset that works is US-ASCII.  This is rather annoying, to say the
least.

My default-process-coding-system is utf-8-unix.

HTH,
eric

-- 
: Eric S Fraga, GnuPG: 0xFFFCF67D
: in Emacs 24.4.1 + Ma Gnus v0.12 + evil-git-bdeb602
: BBDB version 3.1.2 (2014-04-27 15:05:20 -0500)

Re: Recent nnir update broke search on office365

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Sunday,  1 Feb 2015 at 14:22, Eric Abrahamsen wrote:
>> david.goldberg6@verizon.net (Dave Goldberg) writes:
>>
>>> All searches come up empty since this commit (per git-bisect)
>
> This happens to me as well in latest gnus (as of yesterday, that
> is).  I've backed up to a version that works for now.
>
> [...]
>
>> I guess what would be most helpful is if you could tell me your value
>> for default-process-coding-system, and then do a telnet session with
>> your exchange server. I'm not sure about exchange but, assuming you're
>> connecting with ssl, here's what I do with gmail:
>
> To give you a data point, I have followed your instructions for the IMAP
> server I use which is outlook.office365.com (Microsoft :().  The only
> charset that works is US-ASCII.  This is rather annoying, to say the
> least.
>
> My default-process-coding-system is utf-8-unix.

Oh man. That is helpful, though obviously also very depressing. I can't
believe they wouldn't let you input non-ascii search strings, though,
there must be some way around that...

More research! Probably this feature is currently too uncertain to leave
in trunk, it's going to mess with people's imap usage.

Re: Recent nnir update broke search on office365

[e.fraga]

On Tuesday,  3 Feb 2015 at 21:01, Eric Abrahamsen wrote:

[...]

> More research! Probably this feature is currently too uncertain to leave
> in trunk, it's going to mess with people's imap usage.

Yes, having this feature as it stands is not a good idea.  I cannot use
gnus with it.  Search is an integral part of my work flow.  Please
revert these changes.

thanks,
eric

-- 
: Eric S Fraga, GnuPG: 0xFFFCF67D
: in Emacs 24.4.1 + Ma Gnus v0.12 + evil-git-2748228
: BBDB version 3.0.50 (2013-11-16 11:30:49 -0600)

Re: Recent nnir update broke search on office365

[Dave Goldberg]

> On Tuesday,  3 Feb 2015 at 21:01, Eric Abrahamsen wrote:
>
> [...]
>
>> More research! Probably this feature is currently too uncertain to leave
>> in trunk, it's going to mess with people's imap usage.
>
> Yes, having this feature as it stands is not a good idea.  I cannot use
> gnus with it.  Search is an integral part of my work flow.  Please
> revert these changes.

I had the exact same results with office365 as Eric did.  No problem with searching the cyrus server so I didn't try the by-hand connection.

-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: Recent nnir update broke search on office365

[Katsumi Yamaoka]

On Sun, 01 Feb 2015 14:22:44 +0800, Eric Abrahamsen wrote:
> `default-process-coding-system', which is bad because:

> 1. That variable doesn't exist in xemacs

How about simply making it a user option (the default is utf-8)?
utf-8 is available in XEmacsen[1].

> 2. It's probably just the wrong thing to do.

I think so.  The most reasonable solution might be to use
`default-buffer-file-coding-system' that is used for text
encoding, however the value of it is unreliable.  I don't know
how, but on Cygwin `(set-language-environment "Japanese")' sets
it to `japanese-shift-jis-unix' even though the locale is set to
ja_JP.UTF-8, for instance.  So is `default-process-coding-system'.

[1] XEmacs 21.4 users need to load Mule-UCS: (require 'un-define)

Re: Recent nnir update broke search on office365

[Eric Abrahamsen]

david.goldberg6@verizon.net (Dave Goldberg) writes:

>> On Tuesday,  3 Feb 2015 at 21:01, Eric Abrahamsen wrote:
>>
>> [...]
>>
>>> More research! Probably this feature is currently too uncertain to leave
>>> in trunk, it's going to mess with people's imap usage.
>>
>> Yes, having this feature as it stands is not a good idea.  I cannot use
>> gnus with it.  Search is an integral part of my work flow.  Please
>> revert these changes.
>
> I had the exact same results with office365 as Eric did.  No problem with searching the cyrus server so I didn't try the by-hand connection.

Thanks for checking! I've reverted the change.

Re: Recent nnir update broke search on office365

[Eric Abrahamsen]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> On Sun, 01 Feb 2015 14:22:44 +0800, Eric Abrahamsen wrote:
>> `default-process-coding-system', which is bad because:
>
>> 1. That variable doesn't exist in xemacs
>
> How about simply making it a user option (the default is utf-8)?
> utf-8 is available in XEmacsen[1].
>
>> 2. It's probably just the wrong thing to do.
>
> I think so.  The most reasonable solution might be to use
> `default-buffer-file-coding-system' that is used for text
> encoding, however the value of it is unreliable.  I don't know
> how, but on Cygwin `(set-language-environment "Japanese")' sets
> it to `japanese-shift-jis-unix' even though the locale is set to
> ja_JP.UTF-8, for instance.  So is `default-process-coding-system'.
>
> [1] XEmacs 21.4 users need to load Mule-UCS: (require 'un-define)

There's also a variable `network-coding-system-alist' (nil on my
system). As far as I can tell, the "real" encoding system used for
talking to the imap server ought to be produced by a call that looks
like this:

(find-operation-coding-system 'open-network-stream name work-buffer host
service parameters)

Where the other arguments are produced inside of `open-protocol-stream'
aka `open-network-stream'. I tried eval'ing that while edebugging, and
it produced nil, which either means I did it wrong, or just that some
default value is being used instead -- perhaps just the buffer coding
system of the process buffer?

But even if we figure out a reliable way to find the right coding
system, there's still the issue of the imap servers and their various
coding quirks. I'm still surprised that searching for non-ascii terms is
such a difficult thing to do.

Eric

Re: Recent nnir update broke search on office365

[Eric S Fraga]

On Wednesday,  4 Feb 2015 at 11:02, Eric Abrahamsen wrote:
> Thanks for checking! I've reverted the change.

Thanks.  I've updated my gnus to latest from git and search works fine again.

-- 
: Eric S Fraga, GnuPG: 0xFFFCF67D
: in Emacs 24.4.1 + Ma Gnus v0.12 + evil-git-bdeb602
: BBDB version 3.1.2 (2014-04-27 15:05:20 -0500)

Re: Recent nnir update broke search on office365

[Andreas Schwab]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> (find-operation-coding-system 'open-network-stream name work-buffer host
> service parameters)
>
> Where the other arguments are produced inside of `open-protocol-stream'
> aka `open-network-stream'. I tried eval'ing that while edebugging, and
> it produced nil, which either means I did it wrong, or just that some
> default value is being used instead -- perhaps just the buffer coding
> system of the process buffer?

That just means network-coding-system-alist doesn't contain a match.  It
doesn't try to find a definitive answer.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Recent nnir update broke search on office365

[Eric Abrahamsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> (find-operation-coding-system 'open-network-stream name work-buffer host
>> service parameters)
>>
>> Where the other arguments are produced inside of `open-protocol-stream'
>> aka `open-network-stream'. I tried eval'ing that while edebugging, and
>> it produced nil, which either means I did it wrong, or just that some
>> default value is being used instead -- perhaps just the buffer coding
>> system of the process buffer?
>
> That just means network-coding-system-alist doesn't contain a match.  It
> doesn't try to find a definitive answer.

Yeah, that's what I figured. I might go ask on emacs.help.