Vista, secure imap and firewalls?

Vista, secure imap and firewalls?

[Steinar Bang]

Platform: Vista Enterprise SP1
	  GNU Emacs 22.3.1 (i386-mingw-nt6.0.6001) of 2008-09-06 on SOFT-MJASON
	  No Gnus v0.11 (CVS checkout from yesterday)
	  openssl using SUA, from http://suacommunity.com

Cygwin didn't install for me on Vista, so I had to look for another
source for openssl.  I found the Services for Unix Applications (the
built-in POSIX-thingy in Windows) and enabled it, and downloaded and
installed everything from the SUA community (see URL above).

I have a home computer running dovecot and listening for IMAPS
connections on port 993.  I can connect to this dovecot server from this
Vista machine.  But I can't connect with gnus.

Nor can I connect using the command line command
  openssl s_client -connect homeserver.dyndns.org:993
from a cmd shell on this Vista machine.

The command connects fine, using SUA community openssl, from a Win2008
Server running in a different network, but that is potentially a
different binary, since the 2008 Server is an AMD64 machine.

So I have currently two theories:
 - There is a firewall either on the Vista machine or on the network I'm
   connectiong in (but if so: why is Thunderbird successful in talking
   on that port?)
 - The openssl client on the Vista machine is broken

Any other theories?  Other tips and tricks I could try?

Does anyone successfully connect to an IMAPS server from a Gnus on a
Vista machine?  What about other windowsen?

If so: what openssl do you use?  Are anyone else running SUA community
opensssl? 

Thanx!


- Steinar

Re: Vista, secure imap and firewalls?

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> Nor can I connect using the command line command
>   openssl s_client -connect homeserver.dyndns.org:993
> from a cmd shell on this Vista machine.

> The command connects fine, using SUA community openssl, from a Win2008
> Server running in a different network, but that is potentially a
> different binary, since the 2008 Server is an AMD64 machine.

Using the Vista machine (it's a laptop) in my home LAN, I can connect to
the dovecot server with an openssl command as above.

So there's probably something firewallish between the Vista machine and
the outside world at work.  How and why that firewall lets through
Thunderbird traffic and stops the openssl command line client, I don't
know. 

It could be a characteristic of the traffic perhaps...?  But I would
have thought the Thunderbird SSL support came from openssl...=

Anyway: even with openssl working from the command line, Gnus fails to
open the connection:

Opening nnimap server on myserver...
imap: Connecting to myserver.dyndns.org...
imap: Opening SSL connection with `openssl s_client -quiet -ssl3 -connect %s:%p'...done
Waiting for response from myserver.dyndns.org...done
Unable to open server nnimap+myserver due to: Process imap not running
Unable to open server nnimap+myserver, go offline? (y or n) 
Opening nnimap server on myserver...failed

Here's my gnus-secondary-select-methods:

(setq gnus-secondary-select-methods
      '((nnimap "myserver"
		(nnimap-address "myserver.dyndns.org")
		(nnimap-stream ssl))
 	(nntp "news.gmane.org")
	(nndiary "")
	))

Hm... this setup is for a linux machine.  Do I perhaps need to do
something about line endings?

nnimap with openssl stopping up after connecting in Windows (Was: Vista, secure imap and firewalls?)

[Steinar Bang]

I'm changing the subject to make the problem issue clearer.

Steven E. Harris have seen the same problem as me 
 nntp://news.gmane.org/gmane.emacs.gnus.general/67236

He has the problem when using Cygwin openssl, I have the problem when
using SUA openssl.  I believe both operates in unix mode, ie. with LF as
the line separator (rather than CRLF).

Basically the first "openssl s_client ..." command succeeds in making
the connection, but then everything just hangs.  It typically looks like
this in the *Messages* buffer:
 Opening nnimap server on myserver...
 imap: Connecting to myserver.dyndns.org...
 imap: Opening SSL connection with `openssl s_client -quiet -ssl3 -connect %s:%p'...done
 Waiting for response from myserver.dyndns.org...done
 Unable to open server nnimap+myserver due to: Process imap not running
 Unable to open server nnimap+myserver, go offline? (y or n) 
 Opening nnimap server on myserver...failed

I tried the GnuWin32 openssl (in the hope that it maybe used CRLF for
line separations) but it didn't even get as far as the SUA one.  It
didn't even succeed in connecting to the server before it frose up.

When running the "openssl s_client -connect" command from a cmd shell
both the SUA and GnuWin32 openssl programs succeeded in connecting to
the IMAP server.

Steven thinks the solution lies in setting the coding system for the
buffer where nnimaps reads the server responses (perhaps to
undecided-unix...?), but he hasn't yet figured out to do it.

Steven?  Perhaps you can jump in with some more detail here?

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> I'm changing the subject to make the problem issue clearer.
> Steven E. Harris have seen the same problem as me 
>  nntp://news.gmane.org/gmane.emacs.gnus.general/67236

> He has the problem when using Cygwin openssl, I have the problem when
> using SUA openssl.  I believe both operates in unix mode, ie. with LF as
> the line separator (rather than CRLF).

> Basically the first "openssl s_client ..." command succeeds in making
> the connection, but then everything just hangs.  It typically looks like
> this in the *Messages* buffer:
>  Opening nnimap server on myserver...
>  imap: Connecting to myserver.dyndns.org...
>  imap: Opening SSL connection with `openssl s_client -quiet -ssl3 -connect %s:%p'...done
>  Waiting for response from myserver.dyndns.org...done
>  Unable to open server nnimap+myserver due to: Process imap not running
>  Unable to open server nnimap+myserver, go offline? (y or n) 
>  Opening nnimap server on myserver...failed

Setting imap-log to t, I find the following in the *imap-log* buffer
(I've replaced CR characters with the string "^M" to preserve them in
this posting, I've also anonymized server and domain names):

depth=0 /O=Dovecot mail server/OU=myserver/CN=myserver/emailAddress=root@myserver.mydomain.no
verify error:num=18:self signed certificate
verify return:1
depth=0 /O=Dovecot mail server/OU=myserver/CN=myserver/emailAddress=root@myserver.mydomain.no
verify return:1
* OK Dovecot ready.^M
6 NOOP^M
* BYE Disconnected for inactivity.^M
read:errno=0
7 CAPABILITY^M

I'm not sure what the visible ^M characters signifies...?  Perhaps it
doesn't signify anything at all, except that the character is there in
the dialog, proving that what was sent was CDRLF?

Or perhaps what we have is CRCRLF, where only the first CR is visible?

But all of this is just speculation from my side.

Re: nnimap with openssl stopping up after connecting in Windows

[Steven E. Harris]

Steinar Bang <sb@dod.no> writes:

> Steven thinks the solution lies in setting the coding system for the
> buffer where nnimaps reads the server responses (perhaps to
> undecided-unix...?), but he hasn't yet figured out to do it.

Right. I note that the lines being read from the remote IMAP server show
up in the IMAP process buffer with ^M glyphs at the end. Looking at
these lines in hexl mode, I can see that Emacs has each line ending with

  CR CR LF

That suggests some conflicting assumptions about line endings, maybe
with two steps of conversion going on among the programs involved:

  starttls (Cygwin) -> Emacs (Windows)

That is, the Emacs is a Windows build, but it's calling on starttls
built for Cygwin. I would expect that this starttls would emit lines per
the Unix convention, or maybe it transparently passes through what the
server sends.

The IMAP specification shows line ending with CRLF pairs. If that's what
the server is sending, then I suspect that Emacs reading from starttls
is doing some erroneous line conversion and adding in the extra CR
before the server's CRLF pair.

Here's what Emacs shows in its *imap-log* buffer, sanitized a little:

,----[ *imap-log* buffer ]
| * OK [CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS QUOTA STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
| 1 CAPABILITY
| * CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS QUOTA STARTTLS AUTH=PLAIN AUTH=LOGIN
| 1 OK Capability completed.
| 1 STARTTLS
| * OK [CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS QUOTA STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
| 1 OK Begin TLS negotiation now.
| 2 LOGOUT
| * BYE Logging out
| 2 OK Logout completed.
| 2 CAPABILITY
| 3 LOGIN "seh" "mypassword"
`----

And from the *Messages* buffer:

,----[ *Messages* buffer ]
| Checking new news...
| Opening nnimap server on panix...
| imap: Connecting to mail.panix.com...
| Waiting for response from mail.panix.com...done
| imap: Reconnecting with stream `starttls'...
| Loading starttls...done
| imap: Connecting with STARTTLS...done
| Waiting for response from mail.panix.com...done
| imap: Reconnecting with stream `starttls'...done
| Parsing authinfo file `~/.authinfo'.
| imap: Authenticating to `mail.panix.com' using `login'...
| imap: Plaintext authentication...
| Unable to open server due to: Process imap<1> not running
| Unable to open nnimap:panix, go offline? (y or n) 
| Opening nnimap server on panix...failed
`----

Per the "2 LOGOUT" command above, it looks like the client closes the
connection, and /then/ issues the LOGIN command. I'm not sure why. Or
maybe that LOGOUT is required once the TLS negotiation is complete,
after which the LOGIN command is appropriate.

Here's a similar *imap-log* buffer from a successful login with a Cygwin
XEmacs build (actual ^M characters are replaced here with the character
sequence "^M"):

,----[ *imap-log* buffer (Cygwin) ]
| * OK [CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS QUOTA STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
| 1 CAPABILITY^M
| * CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS QUOTA STARTTLS AUTH=PLAIN AUTH=LOGIN^M
| 1 OK Capability completed.^M
| 1 STARTTLS^M
| * OK [CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS QUOTA STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.^M
| 1 OK Begin TLS negotiation now.^M
| 2 NOOP^M
| 2 OK NOOP completed.^M
| 3 LOGOUT^M
| * BYE Logging out^M
| 3 OK Logout completed.^M
| 2 NOOP^M
| 2 OK NOOP completed.^M
| 3 CAPABILITY^M
| * CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS QUOTA AUTH=PLAIN AUTH=LOGIN^M
| 3 OK Capability completed.^M
| 4 LOGIN "seh" "mypassword"^M
| 4 OK Logged in.^M
| 5 NOOP^M
| 5 OK NOOP completed.^M
| 6 STATUS "INBOX" (UIDVALIDITY UIDNEXT UNSEEN)^M
| * STATUS "INBOX" (UIDNEXT 18324 UIDVALIDITY 1067275254 UNSEEN 0)^M
| 6 OK Status completed.^M
`----

Note here the NOOP commands and the LOGOUT followed by another
CAPABILITY, finally followed by the LOGIN command. There's more
cause-and-effect chatter evident here, whereas the first transcript
above looks more like two parties talking but not hearing one another.

I hope someone more familiar with nnimap, process control, and coding
systems can chime with suggestions for further investigation.

-- 
Steven E. Harris

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> Setting imap-log to t, I find the following in the *imap-log* buffer
> (I've replaced CR characters with the string "^M" to preserve them in
> this posting, I've also anonymized server and domain names):

> depth=0 /O=Dovecot mail server/OU=myserver/CN=myserver/emailAddress=root@myserver.mydomain.no
> verify error:num=18:self signed certificate
> verify return:1
> depth=0 /O=Dovecot mail server/OU=myserver/CN=myserver/emailAddress=root@myserver.mydomain.no
> verify return:1
> * OK Dovecot ready.^M
> 6 NOOP^M
> * BYE Disconnected for inactivity.^M
> read:errno=0
> 7 CAPABILITY^M

[snip!]
> Or perhaps what we have is CRCRLF, where only the first CR is visible?

When I look at this like Steven did, ie. using hexl-mode, I see that this is the
case.

All lines where the "^M" show have the sequence "0d0d0a" (CRCRLF),
while the lines without an "^M" have "0d0a" (CRLF).

Note: I'm using SUA openssl, not the cygwin one (but the behaviour
seems to be identical).

Re: nnimap with openssl stopping up after connecting in Windows

[Simon Josefsson]

Look in the ' *nnimap* '... buffer instead of *imap-log*, it may contain
better encoding of CR/LF/etc.

However, if indeed the data received by Emacs is CRCRLF, try to change
the imap-server-eol variable, e.g.:

(setq gnus-secondary-select-methods
      '((nnimap "myserver"
		(nnimap-address "myserver.dyndns.org")
		(nnimap-stream ssl)
                (imap-server-eol "\r\r\n")
 	(nntp "news.gmane.org")
	(nndiary "")
	))

You may need to modify imap-client-eol as well.

/Simon

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Simon Josefsson <simon@josefsson.org>:

> Look in the ' *nnimap* '... buffer instead of *imap-log*, it may
> contain better encoding of CR/LF/etc.

Is there a particular time one should look in this buffer?  Or is
there a trick to showing some hidden stuff?  This buffer always look
empty when I look into it.

> However, if indeed the data received by Emacs is CRCRLF, try to change
> the imap-server-eol variable, e.g.:

> (setq gnus-secondary-select-methods
>       '((nnimap "myserver"
> 		(nnimap-address "myserver.dyndns.org")
> 		(nnimap-stream ssl)
>               (imap-server-eol "\r\r\n"))
>  	(nntp "news.gmane.org")
> 	(nndiary "")
> 	))

> You may need to modify imap-client-eol as well.

I tried first changing just imap-server-eol as above.  That didn't
work.  It still hung until it timed out.

Then I tried also changing imap-client-eol to "\r\r\n".  That didn't
work either.

Then I tried changing imap-client-eol to "\n" (in the theory of
something changing unix line endings to CRLF).  But that didn't help
either. 

So I guess the important now is to find out what's actually in the '
*nnimap*' buffer...?  I'm assuming it's there but that I have to do
something to see it...?

(Side note: Curiously both variables where shown as holding the value
"^M\n".  Ie. the LF was quoted, unix style, but the CR was unquoted.)

When and how to look in ' *nnimap* ' buffers? (Was: nnimap with openssl stopping up after connecting in Windows)

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> Simon Josefsson <simon@josefsson.org>:
>> Look in the ' *nnimap* '... buffer instead of *imap-log*, it may
>> contain better encoding of CR/LF/etc.

> Is there a particular time one should look in this buffer?  Or is
> there a trick to showing some hidden stuff?  This buffer always look
> empty when I look into it.

I've looked for hidden and invisible text and found nothing there.
Is the stream content from the server supposed to be in this buffer?  Is
it deleted as it is read?  Do I have to run gnus in the debugger and set
a breakpoint somewhere, and then look at the buffer content?  Where
should I put that breakpoint?

Thanx!


- Steinar

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> Simon Josefsson <simon@josefsson.org>:
>> Look in the ' *nnimap* '... buffer instead of *imap-log*, it may
>> contain better encoding of CR/LF/etc.

> Is there a particular time one should look in this buffer?  Or is
> there a trick to showing some hidden stuff?  This buffer always look
> empty when I look into it.

When I looked today, there was content.  Here's a sanitized version of
the content (changed servername and domain, and visible extra CR
replaced with "^M"), and a sligthly less sanitized version (changed
server name and domain), shown in hexl-mode.

What is shows is the same CRCRLF on the IMAP dialogue lines (only server
side this time, but that may be because there is no client side traffic
in imap in the buffer below).

What the hexl-mode version shows, is that all of the other lines have
CRLF.  I've no idea what this indicates.

Note: this is without the imap-client-eol and imap-server-eol settings.

They would perhaps break anyway, since the problem is that part of the
buffer is in CRLF, and the IMAP dialog part of the buffer is CRCRLF...?


First the sanitized buffer content:

depth=0 /O=Dovecot mail server/OU=myserver/CN=myserver/emailAddress=root@myserver.mydomain.no
verify error:num=18:self signed certificate
verify return:1
depth=0 /O=Dovecot mail server/OU=myserver/CN=myserver/emailAddress=root@myserver.mydomain.no
verify return:1
* OK Dovecot ready.^M
* BYE Disconnected for inactivity.^M
read:errno=0

Process imap finished

Then the hexl-mode version:

00000000: 6465 7074 683d 3020 2f4f 3d44 6f76 6563  depth=0 /O=Dovec
00000010: 6f74 206d 6169 6c20 7365 7276 6572 2f4f  ot mail server/O
00000020: 553d 6d79 7365 7276 6572 2f43 4e3d 6d79  U=myserver/CN=my
00000030: 7365 7276 6572 2f65 6d61 696c 4164 6472  server/emailAddr
00000040: 6573 733d 726f 6f74 406d 7973 6572 7665  ess=root@myserve
00000050: 722e 6d79 646f 6d61 696e 2e6e 6f0d 0a76  r.mydomain.no..v
00000060: 6572 6966 7920 6572 726f 723a 6e75 6d3d  erify error:num=
00000070: 3138 3a73 656c 6620 7369 676e 6564 2063  18:self signed c
00000080: 6572 7469 6669 6361 7465 0d0a 7665 7269  ertificate..veri
00000090: 6679 2072 6574 7572 6e3a 310d 0a64 6570  fy return:1..dep
000000a0: 7468 3d30 202f 4f3d 446f 7665 636f 7420  th=0 /O=Dovecot 
000000b0: 6d61 696c 2073 6572 7665 722f 4f55 3d6d  mail server/OU=m
000000c0: 7973 6572 7665 722f 434e 3d6d 7973 6572  yserver/CN=myser
000000d0: 7665 722f 656d 6169 6c41 6464 7265 7373  ver/emailAddress
000000e0: 3d72 6f6f 7440 6d79 7365 7276 6572 2e6d  =root@myserver.m
000000f0: 7964 6f6d 6169 6e2e 6e6f 0d0a 7665 7269  ydomain.no..veri
00000100: 6679 2072 6574 7572 6e3a 310d 0a2a 204f  fy return:1..* O
00000110: 4b20 446f 7665 636f 7420 7265 6164 792e  K Dovecot ready.
00000120: 0d0d 0a2a 2042 5945 2044 6973 636f 6e6e  ...* BYE Disconn
00000130: 6563 7465 6420 666f 7220 696e 6163 7469  ected for inacti
00000140: 7669 7479 2e0d 0d0a 7265 6164 3a65 7272  vity....read:err
00000150: 6e6f 3d30 0d0a 0d0a 5072 6f63 6573 7320  no=0....Process 
00000160: 696d 6170 2066 696e 6973 6865 640d 0a    imap finished..

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> When I looked today, there was content.  Here's a sanitized version of
> the content (changed servername and domain, and visible extra CR
> replaced with "^M"), and a sligthly less sanitized version (changed
> server name and domain), shown in hexl-mode.

> What is shows is the same CRCRLF on the IMAP dialogue lines (only server
> side this time, but that may be because there is no client side traffic
> in imap in the buffer below).

And what this means, is what Steven says earlier in the thread, that
something is changing LFs to CRLFs when reading from a process, so that
when a CRLF actually occurs it's expanded to CRCRLF and breaks the IMAP
client. 

So the fix can be either:
 - Make emacs stop changing LF to CRLF in this case
 - Make the imap dialogue operate with the broken line endings.  I guess
   that would mean understanding CRCRLF where a CRLF is expected, and
   outputting LF where a CRLF would normally be expected (I guess this
   is what Simon's proposal was about...?)

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> So the fix can be either:
>  - Make emacs stop changing LF to CRLF in this case
>  - Make the imap dialogue operate with the broken line endings.  I guess
>    that would mean understanding CRCRLF where a CRLF is expected, and
>    outputting LF where a CRLF would normally be expected (I guess this
>    is what Simon's proposal was about...?)

I've tried the second alternative, by setting:
      ((nnimap "myserver"
		(nnimap-address "myserver.dyndns.org")
		(nnimap-stream ssl)
                (imap-server-eol "\r\r\n")
                (imap-client-eol "\r\r\n"))

(I've tried both CRCRLF and just LF for the imap-client-eol) without
success.  The contents of the " *nnimap* myserver" buffer is

* BYE Disconnected for inactivity.^M
read:errno=0

(sanitized extra CR character)

So I guess the right thing to do here, is to make sure there is no form
of end-of-line translation going on in the nnimap buffer.

I wonder what the correct approach here, is?
  (set-buffer-process-coding-system 'undecided 'undecided)
or maybe
  (set-buffer-file-coding-system 'undecided 'undecided)
?

Or should I use 'undecided-unix rather than 'undecided to emulate the
unix/linux behaviour?

I'll try using set-buffer-process-coding-system first.  I'll hook it
into the imap-disable-multibyte inline function, which seems to be
called in all of the relevant places.

(And, yeah... remember to remove the modified end of line settings on
the "myserver" nnimap server as well)

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> So I guess the right thing to do here, is to make sure there is no form
> of end-of-line translation going on in the nnimap buffer.

> I wonder what the correct approach here, is?
>   (set-buffer-process-coding-system 'undecided 'undecided)

This is what I tried.  Then there was no time out wait, and connecting
to the imap server immediately failed, with the following messages in
the minibuffer:

Opening nnimap server on myserver...
Unable to open server nnimap+myserver due to: No process
Unable to open server nnimap+myserver, go offline? (y or n) 
Opening nnimap server on myserver...failed

Trying to run (imap-disable-multibyte) in the *scratch* buffer gives the
following stack trace:
Debugger entered--Lisp error: (error "No process")
  signal(error ("No process"))
  error("No process")
  set-buffer-process-coding-system(undecided undecided)
  (progn (set-buffer-process-coding-system (quote undecided) (quote undecided)))
  (if (fboundp (quote set-buffer-process-coding-system)) (progn (set-buffer-process-coding-system ... ...)))
  (when (fboundp (quote set-buffer-process-coding-system)) (set-buffer-process-coding-system (quote undecided) (quote undecided)))
  imap-disable-multibyte()
  eval((imap-disable-multibyte))
  eval-last-sexp-1(t)
  eval-last-sexp(t)
  eval-print-last-sexp()
  call-interactively(eval-print-last-sexp)

So I guess set-buffer-file-coding-system is the one to try next... 

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>> So I guess the right thing to do here, is to make sure there is no form
>> of end-of-line translation going on in the nnimap buffer.

>> I wonder what the correct approach here, is?
>> (set-buffer-process-coding-system 'undecided 'undecided)

> So I guess set-buffer-file-coding-system is the one to try next... 

That didn't work either.

But perhaps I did the wrong thing by piggy-backing on the
imap-disable-multibyte function?

Perhaps the only place to set the coding system, are in the two calls to
the imap-disable-multibyte function that aren't applied to the
imap-log-buffer?  I.e. the two calls to imap-disable-multibyte that are
inside the imap-open function?

And perhaps the calls should be to set-buffer-process-coding-system?

I think I will try to create a new inline function that sets the buffer
process coding system to undecided both for outgoing and incoming, and
call that from the two places in imap-open that imap-disable-multibyte
is called from.

(I'm waving around blindly here, as readers of this articles may have
guessed.  Any assistance from anyone who can tell what actually makes
sense would be appreciated)

Re: nnimap with openssl stopping up after connecting in Windows

[Steven E. Harris]

Steinar Bang <sb@dod.no> writes:

> Or should I use 'undecided-unix rather than 'undecided to emulate the
> unix/linux behaviour?

Another one to try is 'binary, which should disable all conversion. I'm
not sure that we /are/ trying to disable all conversion, but if so,
'binary is the right one.

-- 
Steven E. Harris

Re: nnimap with openssl stopping up after connecting in Windows

[Steven E. Harris]

Steinar Bang <sb@dod.no> writes:

> I wonder what the correct approach here, is?
>   (set-buffer-process-coding-system 'undecided 'undecided)
> or maybe
>   (set-buffer-file-coding-system 'undecided 'undecided)
> ?

There's also `imap-coding-system-for-read' and
`imap-coding-system-for-write'. Their values are bound to the related
`coding-system-for-read' and `coding-system-for-write' before starting
up the subprocess to talk to the IMAP server. (See, for example,
`imap-ssl-open'.)

They're both bound to 'binary by default, which is what I would have
proposed setting them to for our experiment.

-- 
Steven E. Harris

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> "Steven E. Harris" <seh@panix.com>:

> Another one to try is 'binary, which should disable all conversion. I'm
> not sure that we /are/ trying to disable all conversion, but if so,
> 'binary is the right one.

Right.  I'd thought I'd heard about 'binary somewhere.  But it wasn't on
the list, so I thought I'd imagined that.  I'll try using that in my
experiment.

That is: I'll try 'undecided first to see what happens.  I'm also
planning on trying out 'undecided-unix to emulate the behaviour on unix
platforms.

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> "Steven E. Harris" <seh@panix.com>:

> There's also `imap-coding-system-for-read' and
> `imap-coding-system-for-write'. Their values are bound to the related
> `coding-system-for-read' and `coding-system-for-write' before starting
> up the subprocess to talk to the IMAP server. (See, for example,
> `imap-ssl-open'.)

> They're both bound to 'binary by default, which is what I would have
> proposed setting them to for our experiment.

Hm... that's not promising.  But I'll try my experiments anyway.

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> Right.  I'd thought I'd heard about 'binary somewhere.  But it wasn't
> on the list, so I thought I'd imagined that.  I'll try using that in
> my experiment.

> That is: I'll try 'undecided first to see what happens.  I'm also
> planning on trying out 'undecided-unix to emulate the behaviour on
> unix platforms.

Calling set-buffer-process-coding-system fails, even when just calling it in
imap-open.  So that's probably not the right thing either.

It fails with the 
 Unable to open server nnimap+myserver due to: No process
message.

So I'll try set-buffer-file-coding-system 'undecided-unix and 'binary
(I've already tried 'undecided without any effect), but I'm not holding
my breath, since this looks like it's related to saving and loading
files, and there is no such thing taking place in the nnimap buffers...

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> So I'll try set-buffer-file-coding-system 'undecided-unix and 'binary
> (I've already tried 'undecided without any effect), but I'm not
> holding my breath, since this looks like it's related to saving and
> loading files, and there is no such thing taking place in the nnimap
> buffers...

Neither 'undecided-unix nor 'binary as arguments to
set-buffer-file-coding-system had any effect (not surprisingly...).

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

On a side note: I tried plain IMAP from inside my home LAN.  And that
worked.

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> On a side note: I tried plain IMAP from inside my home LAN.  And that
> worked.

(Side note: What this _does_ mean is that the escape hatch of using SSH
port forwarding may be an option.  It's clumsy, but it is an
option... also it is a way around the fact that, as mentioned at the
start of the original thread, something is stopping the openssl
connections from the work LAN (but not the IMAPS connections made by
Thunderbird and Opera, to my home IMAP server, for some strange reason))

Re: nnimap with openssl stopping up after connecting in Windows

[Steven E. Harris]

Steinar Bang <sb@dod.no> writes:

> So the fix can be either:
>  - Make emacs stop changing LF to CRLF in this case

I'm happy to report that I'm posting this from within Emacs, and I think
I found the problem with this whole Windows-Emacs-with-Cygwin-tunneling
setup. It's the shell!

Check the value of `shell-file-name' in your Emacs instance. If, like
me, you started Emacs from within a Cygwin shell, this variable will
bear a value such as "zsh". But try this:

,----
| (let ((shell-file-name "C:/Program Files/emacs-22.3/bin/cmdproxy.exe"))
|   (gnus))
`----

or even just

,----
| (let ((shell-file-name "cmdproxy"))
|   (gnus))
`----

and see what happens. That's all it took -- after about four months of
occasional noodling -- to make it work.

There's also the variable `shell-command-switch' which may need
adjustment for some values of `shell-file-name'. Its default value of
"-c" seems to work properly with "cmdproxy".


[Some time passes...]

Well, maybe not. I still can't get this to work with STARTTLS, and now
I'm noticing that even if I set `shell-file-name' back to "zsh", nnimap
still works with openssl. If I include the form

  (nnimap-stream ssl)

in my select method definition for the IMAP server, it works. Without
that form, it tries to use STARTTLS, finds a Cygwin-compiled starttls
program, and fails as described previously.

I'd like to be more happy about this, but I'm confused.

-- 
Steven E. Harris

Re: nnimap with openssl stopping up after connecting in Windows

[Steven E. Harris]

[I apologize if this message already made it to the list once. My Emacs
 email-related configuration is not yet straightened out.]


Steinar Bang <sb@dod.no> writes:

> So the fix can be either:
>  - Make emacs stop changing LF to CRLF in this case

I'm happy to report that I'm posting this from within Emacs, and I think
I found the problem with this whole Windows-Emacs-with-Cygwin-tunneling
setup. It's the shell!

Check the value of `shell-file-name' in your Emacs instance. If, like
me, you started Emacs from within a Cygwin shell, this variable will
bear a value such as "zsh". But try this:

,----
| (let ((shell-file-name "C:/Program Files/emacs-22.3/bin/cmdproxy.exe"))
|   (gnus))
`----

or even just

,----
| (let ((shell-file-name "cmdproxy"))
|   (gnus))
`----

and see what happens. That's all it took -- after about four months of
occasional noodling -- to make it work.

There's also the variable `shell-command-switch' which may need
adjustment for some values of `shell-file-name'. Its default value of
"-c" seems to work properly with "cmdproxy".

[Some time passes...]

Well, maybe not. I still can't get this to work with STARTTLS, and now
I'm noticing that even if I set `shell-file-name' back to "zsh", nnimap
still works with openssl. If I include the form

  (nnimap-stream ssl)

in my select method definition for the IMAP server, it works. Without
that form, it tries to use STARTTLS, finds a Cygwin-compiled starttls
program, and fails as described previously.

I'd like to be more happy about this, but I'm confused.

-- 
Steven E. Harris

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> "Steven E. Harris" <seh@panix.com>:

> Check the value of `shell-file-name' in your Emacs instance. If, like
> me, you started Emacs from within a Cygwin shell,

Nope.  At least not intentionally.  My emacs is
 GNU Emacs 22.3.1 (i386-mingw-nt6.0.6001) of 2008-09-06 on SOFT-MJASON
and is started from the start menu.

> this variable will bear a value such as "zsh". But try this:

> ,----
> | (let ((shell-file-name "C:/Program Files/emacs-22.3/bin/cmdproxy.exe"))
> | (gnus))
> `----

This is the value it already has unfortunally...:-/
`C-h v shell-file-name' gives:

shell-file-name is a variable defined in `C source code'.
Its value is 
"C:/Program Files/emacs-22.3/bin/cmdproxy.exe"

[snip!]
> [Some time passes...]

> Well, maybe not. I still can't get this to work with STARTTLS, and now
> I'm noticing that even if I set `shell-file-name' back to "zsh",
> nnimap still works with openssl. If I include the form

>   (nnimap-stream ssl)

That's also what I already have...:-/

	(nnimap "myserver"
		(nnimap-address "myserver.dyndns.org")
		(nnimap-stream ssl))

> in my select method definition for the IMAP server, it works. Without
> that form, it tries to use STARTTLS, finds a Cygwin-compiled starttls
> program, and fails as described previously.

With ssl there it finds a SUA Community compiled openssl (since Cygwin
wouldn't install for me on Vista (with "the BLODA explanation" as the
closest I got).
	http://www.suacommunity.com/

> I'd like to be more happy about this, but I'm confused.

So am I.  Perhaps we don't have the same problem, even...? :-/

Re: nnimap with openssl stopping up after connecting in Windows

[Steven E. Harris]

Steinar Bang <sb@dod.no> writes:

>> I'd like to be more happy about this, but I'm confused.
>
> So am I.  Perhaps we don't have the same problem, even...? :-/

Perhaps. I'm also not confident that when I shut down this current Emacs
instance and start up another one the IMAP connections will continue
working. I'll have more time tonight or over the weekend to experiment
more.

-- 
Steven E. Harris

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> "Steven E. Harris" <seh@panix.com>:

> Steinar Bang <sb@dod.no> writes:
>>> I'd like to be more happy about this, but I'm confused.
>> 
>> So am I.  Perhaps we don't have the same problem, even...? :-/

> Perhaps. I'm also not confident that when I shut down this current
> Emacs instance and start up another one the IMAP connections will
> continue working. I'll have more time tonight or over the weekend to
> experiment more.

I tried the openssl binary linked to from here: http://www.openssl.org/related/binaries.html
(the "Win32 OpenSSL v0.9.8i Light").

The idea was that this program may be running as a native win32
application, and may (hopefully) not trigger the LF->CRLF translation
(which, to recap, isn't the actual problem.  The CRCRLF that results
from a CRLF hitting the conversion _is_ the problem (or at least I
think it is)).

It's another shot in the dark, but what the heck.

What I did:
 - I renamed the SUA community openssl executable to openssl_disabled.
 - I verified that I had no openssl in the path.
 - I added C:\OpenSSL\bin to the path.
 - I started a new cmd window, and verified that I now had the openssl
   executable in the path.
 - I ran the command "openssl s_client -connect myhost.dyndns.org:993"
   and verified that it connected to the IMAP server in what looked (to
   me) like a meaningful way
 - I started a new GNU emacs, and ran `M-x gnus-slave' in it, and it was
   less succsessful than the SUA openssl in opening the connection (it
   never said that it was able to open a connection and then timed out,
   like the SUA openssl does).  The " *nnimap* myserver" buffer
   contains: 

Process imap finished

Process imap<1> exited abnormally with code 1

Process imap<2> exited abnormally with code 1

 - Line separations in the " *nnimap* myserver" buffer are 0a, according
   to hexl-mode (ie. LF), and the status line displays "(Unix)" on the
   left side of the line
 - I started `M-x shell RET' and ran the command "openssl s_client
   -connect myhost.dyndns.org:993" and verified that it was able to open
   a connection to the IMAP server
 - I also started `M-x shell RET' on the emacs process running `M-x gnus
   RET' that was started before changing the path, and tried the same
   command, and verified that it had no openssl in its path

So... I know I have the new openssl executable in a way that can be
executed from inside emacs, but I still haven't been able to test if it
behaves nicer than SUA openssl (and presumably also cygwin openssl?) in
the " *nnimap* myhost" buffer.

Could it be that this openssl communicates with cmd.exe in another way
than using stdout and stdin...?

The speculations continue...

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

Ok.  Now I also can report a success.  This time with cygwin openssl.

My Vista laptop have been blue screening a couple of times.  Apparently
Vista isn't supposed to do that, so IT support have been on me to turn
my laptop in for reinstallation (if it continues to blue screen they
will look at the hardware).

Since reinstallation is a bit of a hazzle, I've been dragging my heels
since mid-November.  But on Wednesday I finally turned it in, and got it
back on Friday.

And this time I wasn't hit with BLODA[1] when installing cygwin, so I
got cygwin openssl.

I modified the imap-ssl-program to use the full path to the cygwin
openssl.exe (I don't like to put cygwin in the PATH.  I prefer to put
GnuWin32 executables there to avoid line ending confusion for diff and
the like) and switched the nnimap server back to
       ((nnimap "myserver"
		(nnimap-address "myserver.dyndns.org")
                (nnimap-stream ssl))
and started Gnus, and it came up and connected to the nnimap server.
...well almost.

It failed at first, but when I entered the server buffer and opened the
nnimap server it opened.

I'm probably running a hook that us run too late...?  It's now running in
gnus-started-hook.  I tried gnus-open-hook and gnus-starting-hook, but
both are too late.

Does anyone know of a good hook candidate for me to use...?

Here's the emacs lisp code from my .emacs file:
(if windows-emacs
    (progn
      (defun use-cygwin-openssl-for-nnimap ()
        (let ((cygwin-openssl "C:\\cygwin\\bin\\openssl.exe"))
          (if (file-exists-p cygwin-openssl)
              (setq imap-ssl-program
                    (mapcar '(lambda (element)
                               (let ((newelement element))
                                 (if (string-match "openssl " element)
                                     (setq newelement (replace-match (concat cygwin-openssl " ") t t element)))
                                 newelement))
                            imap-ssl-program)))))
          (add-hook 'gnus-started-hook 'use-cygwin-openssl-for-nnimap)))


[1] <http://cygwin.com/faq/faq.using.html#faq.using.bloda>

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> I'm probably running a hook that us run too late...?  It's now running
> in gnus-started-hook.  I tried gnus-open-hook and gnus-starting-hook,
> but both are too late.

> Does anyone know of a good hook candidate for me to use...?

I'm now running it from gnus-load-hook, and have added a (require 'imap)
to ensure that imap-ssl-program is defined before running the function
that modifies the list elements.

Then it works for the initial Gnus startup as well.

The code now looks like this:
(if windows-emacs
    (progn
      (defun use-cygwin-openssl-for-nnimap ()
        (let ((cygwin-openssl "C:\\cygwin\\bin\\openssl.exe"))
          (if (file-exists-p cygwin-openssl)
              (progn
                (require 'imap)
                (setq imap-ssl-program
                      (mapcar '(lambda (element)
                                 (let ((newelement element))
                                   (if (string-match "openssl " element)
                                       (setq newelement (replace-match (concat cygwin-openssl " ") t t element))
                                     newelement)))
                              imap-ssl-program))))))
          (add-hook 'gnus-load-hook 'use-cygwin-openssl-for-nnimap)))


The windows-emacs variable is defined like this:

(defvar windows-emacs (string-match "mingw" (emacs-version))
  "Hold a numerical value if this is an Emacs running on Windows, and
nil if this isn't windows")

Re: nnimap with openssl stopping up after connecting in Windows

[Reiner Steib]

On Sun, Jan 11 2009, Steinar Bang wrote:

>>>>>> Steinar Bang <sb@dod.no>:
>
>> I'm probably running a hook that us run too late...?  It's now running
>> in gnus-started-hook.  I tried gnus-open-hook and gnus-starting-hook,
>> but both are too late.
>
>> Does anyone know of a good hook candidate for me to use...?
>
> I'm now running it from gnus-load-hook, and have added a (require 'imap)
> to ensure that imap-ssl-program is defined before running the function
> that modifies the list elements.
>
> Then it works for the initial Gnus startup as well.

Doesn't it work if you put the stuff in ~/.gnus.el?

> The code now looks like this:
> (if windows-emacs
>     (progn
>       (defun use-cygwin-openssl-for-nnimap ()
>         (let ((cygwin-openssl "C:\\cygwin\\bin\\openssl.exe"))

You may use "C:/cygwin/bin/openssl.exe".

>           (if (file-exists-p cygwin-openssl)
>               (progn
>                 (require 'imap)
>                 (setq imap-ssl-program
>                       (mapcar '(lambda (element)
>                                  (let ((newelement element))
>                                    (if (string-match "openssl " element)
>                                        (setq newelement (replace-match (concat cygwin-openssl " ") t t element))
>                                      newelement)))
>                               imap-ssl-program))))))
>           (add-hook 'gnus-load-hook 'use-cygwin-openssl-for-nnimap)))
>
> The windows-emacs variable is defined like this:
> (defvar windows-emacs (string-match "mingw" (emacs-version))

What's wrong with the variable `system-type'.  Typically use:
  (memq system-type '(windows-nt ms-dos))

,----[ <f1> v system-type RET ]
| system-type is a variable defined in `src/emacs.c'.
| Its value is gnu/linux
| 
| Documentation:
| Value is symbol indicating type of operating system you are using.
| Special values:
|   `gnu/linux'   compiled for a GNU/Linux system.
|   `darwin'      compiled for Darwin (GNU-Darwin, Mac OS X, ...).
|   `macos'       compiled for Mac OS 9.
|   `ms-dos'      compiled as an MS-DOS application.
|   `windows-nt'  compiled as a native W32 application.
|   `cygwin'      compiled using the Cygwin library.
|   `vax-vms' or
|   `axp-vms'     compiled for a (Open)VMS system.
| Anything else indicates some sort of Unix system.
`----

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnimap with openssl stopping up after connecting in Windows

[Steven E. Harris]

Steinar Bang <sb@dod.no> writes:

> The windows-emacs variable is defined like this:
>
> (defvar windows-emacs (string-match "mingw" (emacs-version))
>   "Hold a numerical value if this is an Emacs running on Windows, and
>    nil if this isn't windows")

I've had these similar ones around for a while:

,----
| (defvar running-on-cygwin-p (memq system-type '(cygwin cygwin32)))
| (defvar running-on-mswindows-p
|         (memq system-type '(windows-nt ms-windows cygwin cygwin32)))
`----

Note that for these, running on Cygwin implies Windows, so the
Windows-but-not-Cygwin cases need to take both variables into account.

-- 
Steven E. Harris

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Reiner Steib <reinersteib+gmane@imap.cc>:

> Doesn't it work if you put the stuff in ~/.gnus.el?

Nope.  First thing I tried.  Undefined imap-ssl-program.

>> (let ((cygwin-openssl "C:\\cygwin\\bin\\openssl.exe"))

> You may use "C:/cygwin/bin/openssl.exe".

Ok, that's definitely more visually pleasing.

(Is there any other reasons that it's better...?)

> What's wrong with the variable `system-type'.  Typically use:
>   (memq system-type '(windows-nt ms-dos))

Nothing.  Was that present in 2004 on 20.4?  That's when the variable
appeared in my setup, according to "svn blame".  I looked for something
like that back then, but didn't find it.

I'm a slow adopter of new emacsen.  I stayed with emacs 18 for as long
as it was viable and probably a bit beyond that.  I stayed with 19.34
for _ages_. :-)

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>>>>> Reiner Steib <reinersteib+gmane@imap.cc>:
>> Doesn't it work if you put the stuff in ~/.gnus.el?

> Nope.  First thing I tried.  Undefined imap-ssl-program.

But yes, now that I do (require 'imap) anyway, I guess I could put it in
there, and avoid the hazzle of using a hook.

(one good thing about using the gnus-load-hook like I do now is that it
is done only once, but since the match and replace is already set up to
avoid repeated replacement (that's what the trailing space on the match
is about))

So yes, I think I'll try that to simplify things.

Re: nnimap with openssl stopping up after connecting in Windows

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:
>>>>> Steinar Bang <sb@dod.no>:
>>>>> Reiner Steib <reinersteib+gmane@imap.cc>:

>>> Doesn't it work if you put the stuff in ~/.gnus.el?

>> Nope.  First thing I tried.  Undefined imap-ssl-program.

> But yes, now that I do (require 'imap) anyway, I guess I could put it
> in there, and avoid the hazzle of using a hook.

> (one good thing about using the gnus-load-hook like I do now is that
> it is done only once, but since the match and replace is already set
> up to avoid repeated replacement (that's what the trailing space on
> the match is about))

> So yes, I think I'll try that to simplify things.

Done.  The code looks like this:

; Access cygwin openssl directly, if available
(if windows-emacs
    (let ((cygwin-openssl "C:/cygwin/bin/openssl.exe"))
      (if (file-exists-p cygwin-openssl)
          (progn
            (require 'imap) ; Need this to define the imap-ssl-program variable
            (setq imap-ssl-program
                  (mapcar '(lambda (element)
                             (let ((newelement element))
                               (if (string-match "openssl " element) ; NOTE the trailing space in the match avoid repeated replacement of the "openssl" on each Gnus startup
                                   (setq newelement (replace-match (concat cygwin-openssl " ") t t element))
                                 newelement)))
                          imap-ssl-program))))))