IMAP problems

IMAP problems

[Frank Schmitt]

Hello

I've problems connecting to an IMAP server with Gnus. It seems
something goes wrong when authenticating with the server, but what?

Here's what I have in my .gnus:

(add-to-list 'gnus-secondary-select-methods 
                         '(nnimap "mail1"
                                  (nnimap-address "mail1.kens.com")))

When Gnus starts, the connection to the server isn't established, if I
go to server buffer status is set to "denied".

Here's what *imap-log* says
 
* OK Courier-IMAP ready. Copyright 1998-2001 Double Precision, Inc.  See COPYING for distribution information.

1 CAPABILITY

* CAPABILITY IMAP4rev1 CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT STARTTLS

1 OK CAPABILITY completed

* OK Courier-IMAP ready. Copyright 1998-2001 Double Precision, Inc.  See COPYING for distribution information.

1 STARTTLS

1 OK Begin SSL/TLS negotation now.

2 LOGOUT

* BYE Courier-IMAP server shutting down

2 OK LOGOUT completed

2 CAPABILITY

3 LOGIN "myusername" ""


and here the contents of *imap-debug*:

======================================================================
1 -> imap-opened: buffer=" *nnimap* mail1"
1 <- imap-opened: nil
======================================================================
1 -> imap-open: server="mail1.kens.com" port=nil stream=nil auth=nil buffer=" *nnimap* mail1"
| 2 -> imap-opened: buffer=" *nnimap* mail1"
| 2 <- imap-opened: nil
| 2 -> imap-open-1: buffer=" *nnimap* mail1"
| | 3 -> imap-network-open: name="imap" buffer=" *nnimap* mail1" server="mail1.kens.com" port=nil
| | | 4 -> imap-parse-greeting: 
| | | 4 <- imap-parse-greeting: nil
| | | 4 -> imap-parse-greeting: 
| | | 4 <- imap-parse-greeting: nonauth
| | 3 <- imap-network-open: #<process imap>
| 2 <- imap-open-1: #<process imap>
| 2 -> imap-capability: identifier=AUTH=GSSAPI buffer=" *nnimap* mail1"
| | 3 -> imap-send-command-wait: command="CAPABILITY" buffer=nil
| | | 4 -> imap-send-command: command="CAPABILITY" buffer=nil
| | | | 5 -> imap-send-command-1: cmdstr="1 CAPABILITY"
| | | | 5 <- imap-send-command-1: nil
| | | 4 <- imap-send-command: 1
| | | 4 -> imap-wait-for-tag: tag=1 buffer=nil
| | | | 5 -> imap-arrival-filter: proc=#<process imap> string="* CAPABILITY IMAP4rev1 CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT STARTTLS

1 OK CAPABILITY completed

"
| | | | | 6 -> imap-find-next-line: 
| | | | | 6 <- imap-find-next-line: 113
| | | | | 6 -> imap-parse-response: 
| | | | | | 7 -> imap-parse-resp-text: 
| | | | | | | 8 -> imap-parse-resp-text-code: 
| | | | | | | 8 <- imap-parse-resp-text-code: nil
| | | | | | 7 <- imap-parse-resp-text: nil
| | | | | 6 <- imap-parse-response: nil
| | | | | 6 -> imap-find-next-line: 
| | | | | 6 <- imap-find-next-line: 98
| | | | | 6 -> imap-parse-response: 
| | | | | 6 <- imap-parse-response: (IMAP4REV1 CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT STARTTLS)
| | | | | 6 -> imap-find-next-line: 
| | | | | 6 <- imap-find-next-line: 28
| | | | | 6 -> imap-parse-response: 
| | | | | | 7 -> imap-parse-resp-text: 
| | | | | | | 8 -> imap-parse-resp-text-code: 
| | | | | | | 8 <- imap-parse-resp-text-code: nil
| | | | | | 7 <- imap-parse-resp-text: nil
| | | | | 6 <- imap-parse-response: nil
| | | | | 6 -> imap-find-next-line: 
| | | | | 6 <- imap-find-next-line: nil
| | | | 5 <- imap-arrival-filter: nil
| | | 4 <- imap-wait-for-tag: OK
| | 3 <- imap-send-command-wait: OK
| 2 <- imap-capability: nil
| 2 -> imap-capability: identifier=AUTH=KERBEROS_V4 buffer=" *nnimap* mail1"
| 2 <- imap-capability: nil
| 2 -> imap-capability: identifier=STARTTLS buffer=" *nnimap* mail1"
| 2 <- imap-capability: (STARTTLS)
| 2 -> imap-open-1: buffer=#<buffer  *temp*>
| | 3 -> imap-parse-greeting: 
| | 3 <- imap-parse-greeting: nil
| | 3 -> imap-parse-greeting: 
| | 3 <- imap-parse-greeting: nonauth
| | 3 -> imap-send-command-wait: command="STARTTLS" buffer=nil
| | | 4 -> imap-send-command: command="STARTTLS" buffer=nil
| | | | 5 -> imap-send-command-1: cmdstr="1 STARTTLS"
| | | | 5 <- imap-send-command-1: nil
| | | 4 <- imap-send-command: 1
| | | 4 -> imap-wait-for-tag: tag=1 buffer=nil
| | | | 5 -> imap-arrival-filter: proc=#<process imap<1>> string="1 OK Begin SSL/TLS negotation now.

"
| | | | | 6 -> imap-find-next-line: 
| | | | | 6 <- imap-find-next-line: 113
| | | | | 6 -> imap-parse-response: 
| | | | | | 7 -> imap-parse-resp-text: 
| | | | | | | 8 -> imap-parse-resp-text-code: 
| | | | | | | 8 <- imap-parse-resp-text-code: nil
| | | | | | 7 <- imap-parse-resp-text: nil
| | | | | 6 <- imap-parse-response: nil
| | | | | 6 -> imap-find-next-line: 
| | | | | 6 <- imap-find-next-line: 37
| | | | | 6 -> imap-parse-response: 
| | | | | | 7 -> imap-parse-resp-text: 
| | | | | | | 8 -> imap-parse-resp-text-code: 
| | | | | | | 8 <- imap-parse-resp-text-code: nil
| | | | | | 7 <- imap-parse-resp-text: nil
| | | | | 6 <- imap-parse-response: nil
| | | | | 6 -> imap-find-next-line: 
| | | | | 6 <- imap-find-next-line: nil
| | | | 5 <- imap-arrival-filter: nil
| | | 4 <- imap-wait-for-tag: OK
| | 3 <- imap-send-command-wait: OK
| 2 <- imap-open-1: #<process imap<1>>
| 2 -> imap-close: buffer=" *nnimap* mail1"
| | 3 -> imap-opened: buffer=nil
| | 3 <- imap-opened: (open run)
| | 3 -> imap-send-command-wait: command="LOGOUT" buffer=nil
| | | 4 -> imap-send-command: command="LOGOUT" buffer=nil
| | | | 5 -> imap-send-command-1: cmdstr="2 LOGOUT"
| | | | 5 <- imap-send-command-1: nil
| | | 4 <- imap-send-command: 2
| | | 4 -> imap-wait-for-tag: tag=2 buffer=nil
| | | | 5 -> imap-arrival-filter: proc=#<process imap> string="* BYE Courier-IMAP server shutting down

2 OK LOGOUT completed

"
| | | | | 6 -> imap-find-next-line: 
| | | | | 6 <- imap-find-next-line: 42
| | | | | 6 -> imap-parse-response: 
| | | | | | 7 -> imap-parse-resp-text: 
| | | | | | | 8 -> imap-parse-resp-text-code: 
| | | | | | | 8 <- imap-parse-resp-text-code: nil
| | | | | | 7 <- imap-parse-resp-text: nil
| | | | | 6 <- imap-parse-response: nil
| | | | | 6 -> imap-find-next-line: 
| | | | | 6 <- imap-find-next-line: 24
| | | | | 6 -> imap-parse-response: 
| | | | | | 7 -> imap-parse-resp-text: 
| | | | | | | 8 -> imap-parse-resp-text-code: 
| | | | | | | 8 <- imap-parse-resp-text-code: nil
| | | | | | 7 <- imap-parse-resp-text: nil
| | | | | 6 <- imap-parse-response: nil
| | | | | 6 -> imap-find-next-line: 
| | | | | 6 <- imap-find-next-line: nil
| | | | 5 <- imap-arrival-filter: nil
| | | 4 <- imap-wait-for-tag: OK
| | 3 <- imap-send-command-wait: OK
| 2 <- imap-close: t
| 2 -> imap-opened: buffer=" *nnimap* mail1"
| 2 <- imap-opened: (run)

-- 
Did you ever realize how much text fits in eighty columns? If you now consider
that a signature usually consists of up to four lines, this gives you enough
space to spread a tremendous amount of information with your messages. So seize
this opportunity and don't waste your signature with bullshit nobody will read.

Re: IMAP problems

[Simon Josefsson]

Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:

> Hello
>
> I've problems connecting to an IMAP server with Gnus. It seems
> something goes wrong when authenticating with the server, but what?
>
> Here's what I have in my .gnus:
>
> (add-to-list 'gnus-secondary-select-methods 
>                          '(nnimap "mail1"
>                                   (nnimap-address "mail1.kens.com")))
>
> When Gnus starts, the connection to the server isn't established, if I
> go to server buffer status is set to "denied".
>
> Here's what *imap-log* says
>  
...
> 2 CAPABILITY
>
> 3 LOGIN "myusername" ""

Nothing after this?  No response to the LOGIN command?  The IMAP trace
looked fine until then.  Perhaps your 'startls' binary is broken?
Type 'starttls mail1.kens.com 143' and type '. starttls' into it, do
'kill -ALRM <pidofstarttls>' and then type '. capability' followed by
'login myusername yourpassword'.  You should be logged in then.

> | 2 <- imap-opened: (run)

Looks fine too, the only reason I can find is that 'starttls' somehow
dies after a while.

Re: IMAP problems

[Frank Schmitt]

Simon Josefsson <jas@extundo.com> writes:

> ...
>> 2 CAPABILITY
>>
>> 3 LOGIN "myusername" ""
>
> Nothing after this?  No response to the LOGIN command?  The IMAP trace
> looked fine until then.  Perhaps your 'startls' binary is broken?
> Type 'starttls mail1.kens.com 143' and type '. starttls' into it, do
> 'kill -ALRM <pidofstarttls>' and then type '. capability' followed by
> 'login myusername yourpassword'.  You should be logged in then.
>
>> | 2 <- imap-opened: (run)
>
> Looks fine too, the only reason I can find is that 'starttls' somehow
> dies after a while.

I think, I found where the problem is: I'm working under MS Windows,
using a starttls compiled by me under the cygwin environment. Could it
be possible that there are CRLF vs. CR problems? I've got this idea
because the IMAP-LOG buffer has extra linefeeds in it (every line looks
like this:
foo bar foo bar^M
schlurps murks^M
)

-- 
Did you ever realize how much text fits in eighty columns? If you now consider
that a signature usually consists of up to four lines, this gives you enough
space to spread a tremendous amount of information with your messages. So seize
this opportunity and don't waste your signature with bullshit nobody will read.

Re: IMAP problems

[Frank Schmitt]

Simon Josefsson <jas@extundo.com> writes:

> Nothing after this?  No response to the LOGIN command?  The IMAP trace
> looked fine until then.  Perhaps your 'startls' binary is broken?
> Type 'starttls mail1.kens.com 143' and type '. starttls' into it, do
> 'kill -ALRM <pidofstarttls>' and then type '. capability' followed by
> 'login myusername yourpassword'.  You should be logged in then.

I just tried this. starttls exits quietly after the ". capability".

-- 
Did you ever realize how much text fits in eighty columns? If you now consider
that a signature usually consists of up to four lines, this gives you enough
space to spread a tremendous amount of information with your messages. So seize
this opportunity and don't waste your signature with bullshit nobody will read.

Re: IMAP problems

[Simon Josefsson]

Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> ...
>>> 2 CAPABILITY
>>>
>>> 3 LOGIN "myusername" ""
>>
>> Nothing after this?  No response to the LOGIN command?  The IMAP trace
>> looked fine until then.  Perhaps your 'startls' binary is broken?
>> Type 'starttls mail1.kens.com 143' and type '. starttls' into it, do
>> 'kill -ALRM <pidofstarttls>' and then type '. capability' followed by
>> 'login myusername yourpassword'.  You should be logged in then.
>>
>>> | 2 <- imap-opened: (run)
>>
>> Looks fine too, the only reason I can find is that 'starttls' somehow
>> dies after a while.
>
> I think, I found where the problem is: I'm working under MS Windows,
> using a starttls compiled by me under the cygwin environment. Could it
> be possible that there are CRLF vs. CR problems? I've got this idea
> because the IMAP-LOG buffer has extra linefeeds in it (every line looks
> like this:
> foo bar foo bar^M
> schlurps murks^M
> )

The imap-log should look like that.  And since you got passed the
first CAPABILITY exchange, I doubt this is the problem, but it is
possible.  OTOH, I have never heard of anyone successfully using
starttls under cygwin, so I'm betting on a b0rken starttls binary.
Does it work if you use it manually if you follow the instructions
above?

Re: IMAP problems

[Simon Josefsson]

Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Nothing after this?  No response to the LOGIN command?  The IMAP trace
>> looked fine until then.  Perhaps your 'startls' binary is broken?
>> Type 'starttls mail1.kens.com 143' and type '. starttls' into it, do
>> 'kill -ALRM <pidofstarttls>' and then type '. capability' followed by
>> 'login myusername yourpassword'.  You should be logged in then.
>
> I just tried this. starttls exits quietly after the ". capability".

Oh, too bad I didn't wage any money on the bet, then. :-)

Gnus should ideally notice that the process died, and imap.el do
contain such logic, but I think you ran into a Emacs bug too.  If you
can find out why `process-status' inside `imap-starttls-open' return
'run even though the process has died, and report that to the Emacs
for Windows people, and they fix it, Gnus will happily continue the
login procedure using the normal network connection.

Or, you could remove the 'starttls' binary.

Re: IMAP problems

[Frank Schmitt]

Simon Josefsson <jas@extundo.com> writes:

> Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:
>
>> I just tried this. starttls exits quietly after the ". capability".
>
> Oh, too bad I didn't wage any money on the bet, then. :-)
[...]
> Or, you could remove the 'starttls' binary.

I was to lazy to investigate this any further and switched over to
openssl instead which works nicely now. (OK, I had the problem that
saying g in group buffer didn't find new mails which i solved by
convincing the administrator of the IMAP server that 
Copyright 1998-2003 Double Precision, Inc.
looks much better than
Copyright 1998-2001 Double Precision, Inc.

-- 
Did you ever realize how much text fits in eighty columns? If you now consider
that a signature usually consists of up to four lines, this gives you enough
space to spread a tremendous amount of information with your messages. So seize
this opportunity and don't waste your signature with bullshit nobody will read.

STARTTLS rewrite (was: Re: IMAP problems)

[Simon Josefsson]

Since starttls doesn't seem to work well under Windows, and isn't part
of many systems (e.g., Debian), I wrote a new starttls.el that uses
GNUTLS instead.  imap.el's starttls support is rewritten as well, but
it should support both the old starttls.el and the new one.  The new
starttls.el is currently in contrib/, and if a few people can test it
and verify that it works, I think we could consider replacing
lisp/starttls.el with contrib/starttls.el a bugfix and make the change
before the next release.

So please put (load "~/gnus/contrib/starttls") or something to that
effect in your .emacs or .gnus, if you use STARTTLS with IMAP, and
want to test the new GNUTLS approach.  Don't forget to say if it
worked or not, especially if you use Windows.

Also let me know if the imap.el change broke something in your current
setup.

Bummer: when I tried to send this, I noticed that the smtpmail.el
STARTTLS support doesn't like the new starttls.el.  Will fix.

Re: STARTTLS rewrite

[Simon Josefsson]

I've added some more changes to the starttls code now, and it requires
a patched GNUTLS to work (see post to gnutls-devel).  smtpmail.el
still not fixed.  Just FYI.

Re: STARTTLS rewrite

[Frank Schmitt]

Simon Josefsson <jas@extundo.com> writes:

> So please put (load "~/gnus/contrib/starttls") or something to that
> effect in your .emacs or .gnus, if you use STARTTLS with IMAP, and
> want to test the new GNUTLS approach.  Don't forget to say if it
> worked or not, especially if you use Windows.

Well, I'm using Windows and failed to get gnutls running, as there
doesn't seem to be binaries around and I wasn't able to compile
libgcrypt which is a requirement for gnutls. (I tried first under
MinGW, then under Cygwin with both libgcrypt 1.1.12 and 1.1.42)

-- 
Did you ever realize how much text fits in eighty columns? If you now consider
that a signature usually consists of up to four lines, this gives you enough
space to spread a tremendous amount of information with your messages. So seize
this opportunity and don't waste your signature with bullshit nobody will read.

Re: STARTTLS rewrite

[Simon Josefsson]

Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> So please put (load "~/gnus/contrib/starttls") or something to that
>> effect in your .emacs or .gnus, if you use STARTTLS with IMAP, and
>> want to test the new GNUTLS approach.  Don't forget to say if it
>> worked or not, especially if you use Windows.
>
> Well, I'm using Windows and failed to get gnutls running, as there
> doesn't seem to be binaries around and I wasn't able to compile
> libgcrypt which is a requirement for gnutls. (I tried first under
> MinGW, then under Cygwin with both libgcrypt 1.1.12 and 1.1.42)

Thanks for trying.  Still, I think the chances of eventually getting
GNUTLS up and running under Windows will be higher than starttls.

There seems to be some CRLF troubles when APPENDing articles as well,
so it isn't ready for prime time.