STARTTLS support

STARTTLS support

[Eythan Weg]

Hi,

It seems that the new nnimap does not support (nnimap-stream starttls):
the *nnimap buffer is empty.  No other sign except for a denied status
in the servers buffer.  Starttls is supported in the old version.

Am I wrong?

Eythan

Re: STARTTLS support

[CHENG Gao]

*On Thu, 23 Sep 2010 12:29:55 -0400
* Also sprach eythanweg@gmail.com (Eythan Weg):

> Hi,
>
> It seems that the new nnimap does not support (nnimap-stream starttls):
> the *nnimap buffer is empty.  No other sign except for a denied status
> in the servers buffer.  Starttls is supported in the old version.
>
> Am I wrong?
>
> Eythan

If I understand correctly, it supports only "network" and "ssl". Other
options mentioned in manual are all deprecated. Means manual need be updated?

Re: STARTTLS support

[Lars Magne Ingebrigtsen]

eythanweg@gmail.com (Eythan Weg) writes:

> It seems that the new nnimap does not support (nnimap-stream starttls):
> the *nnimap buffer is empty.  No other sign except for a denied status
> in the servers buffer.  Starttls is supported in the old version.

I've now added starttls support, but I have no starttls-enabled servers
to test with, so I have no idea whether it works or not.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[Eythan Weg]

Lars Magne Ingebrigtsen <larsi@gnus.org>
Thu, 23 Sep 2010 19:43:22 +0200

   eythanweg@gmail.com (Eythan Weg) writes:
   
   > It seems that the new nnimap does not support (nnimap-stream starttls):
   > the *nnimap buffer is empty.  No other sign except for a denied status
   > in the servers buffer.  Starttls is supported in the old version.
   
   I've now added starttls support, but I have no starttls-enabled servers
   to test with, so I have no idea whether it works or not.


Thank you.  Here is a response in the Messages buffer:
 
Unable to open server nnimap+ivytech.edu due to: Symbol's function
definition  is void: starttls-open-stream   


Eythan  

Re: STARTTLS support

[Dave Goldberg]

> Lars Magne Ingebrigtsen <larsi@gnus.org>
> Thu, 23 Sep 2010 19:43:22 +0200

>    eythanweg@gmail.com (Eythan Weg) writes:
   
>> It seems that the new nnimap does not support (nnimap-stream starttls):
>> the *nnimap buffer is empty.  No other sign except for a denied status
>> in the servers buffer.  Starttls is supported in the old version.
   
>    I've now added starttls support, but I have no starttls-enabled servers
>    to test with, so I have no idea whether it works or not.


> Thank you.  Here is a response in the Messages buffer:
 
> Unable to open server nnimap+ivytech.edu due to: Symbol's function
> definition  is void: starttls-open-stream   


Same thing happened to me.

-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: STARTTLS support

[Lars Magne Ingebrigtsen]

eythanweg@gmail.com (Eythan Weg) writes:

> Unable to open server nnimap+ivytech.edu due to: Symbol's function
> definition  is void: starttls-open-stream   

I've now added an autoload cookie for that.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[Dave Goldberg]

> eythanweg@gmail.com (Eythan Weg) writes:
>> Unable to open server nnimap+ivytech.edu due to: Symbol's function
>> definition  is void: starttls-open-stream   

> I've now added an autoload cookie for that.

Well now I'm getting a bit further but still no dice.  Here's the trace from my message log.

Reading /home/dsg/.newsrc.eld...
Loading starttls...
Loading starttls...done
Opening STARTTLS connection to `cyrus.mywork.org:imap'...
Opening STARTTLS connection to `cyrus.mywork.org:imap'...done
STARTTLS negotiation failed: *** Starting TLS handshake
*** Fatal error: A TLS packet with unexpected length was received.
*** Handshake has failed

Process *nnimap* exited abnormally with code 1

Loading auth-source...
Loading auth-source...done
/home/dsg/.authinfo.: 0% (0/128)
Loading passwd...
Loading passwd...done
/home/dsg/.authinfo.: 100% (128/128)
/home/dsg/.authinfo.: 0% (0/128)
/home/dsg/.authinfo.: 100% (128/128)
Unable to open server nnimap+cyrus due to: Buffer *nnimap cyrus.mywork.org nil  *nntpd** has no process
Opening STARTTLS connection to `cyrus.mywork.org:imap'...
Opening STARTTLS connection to `cyrus.mywork.org:imap'...done
STARTTLS negotiation failed: *** Starting TLS handshake
*** Fatal error: A TLS packet with unexpected length was received.
*** Handshake has failed

Process *nnimap* exited abnormally with code 1

Buffer *nnimap cyrus.mywork.org nil  *nntpd**<2> has no process

I guess I need to look at the changes to auth-source because it's now
clearly reading my .authinfo.gpg but still prompting me for username
and password, along with whatever the TLS issue is.

Thanks,
-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: STARTTLS support

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> STARTTLS negotiation failed: *** Starting TLS handshake

I'm now getting a STARTTLS test account, which should make it easier to
debug this.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[Norbert Koch]

* Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Dave Goldberg <david.goldberg6@verizon.net> writes:
>
> > STARTTLS negotiation failed: *** Starting TLS handshake
>
> I'm now getting a STARTTLS test account, which should make it easier to
> debug this.

On my system I get in the nnimap buffer:

----------------------------

* OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS LOGINDISABLED
  AUTH=DIGEST-MD5 AUTH=CRAM-MD5 AUTH=NTLM SASL-IR] mm0004 Cyrus IMAP
  v2.3.14 server ready
 
Process *nnimap* segmentation fault 

----------------------------

The messages so far have been

Opening nnimap server on mmlab...failed 
Denied server nnimap+mmlab 
Opening nnimap server on mmlab... 
Opening nnimap server on mmlab...failed 
Loading auth-source...done 
Loading auth-source... 
Opening STARTTLS connection to `imap.mmlab.de:imap' 
Loading starttls...done 
Loading starttls... 
Opening nnimap server on mmlab... 
Reading active file from mmlab via nnimap... 


norbert.
-- 

Re: STARTTLS support

[Lars Magne Ingebrigtsen]

Norbert Koch <viteno@xemacs.org> writes:

> Process *nnimap* segmentation fault 

Segmentation fault?  That seems excessive.  :-)

Anyway, I've now tested STARTTLS with a test server, and ironed out a
bug in the connection code, so it now Works For Me.  Give it a whirl...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[Eythan Weg]

Lars Magne Ingebrigtsen <larsi@gnus.org>
Sat, 25 Sep 2010 16:12:13 +0200

   Norbert Koch <viteno@xemacs.org> writes:
   
   > Process *nnimap* segmentation fault 
   
   Segmentation fault?  That seems excessive.  :-)
   
   Anyway, I've now tested STARTTLS with a test server, and ironed out a
   bug in the connection code, so it now Works For Me.  Give it a whirl...
   


Works for me too. But the nnimap buffer seems to record the complete
content of every mail.  The ssl stream (with gmail) does not do that, just
the headers.

Anyway, is there a  to control or even avoid these buffers?  

Thanks...

Eythan 

Re: STARTTLS support

[Eythan Weg]

eythanweg@gmail.com (Eythan Weg)
Sat, 25 Sep 2010 11:44:56 -0400

   Lars Magne Ingebrigtsen <larsi@gnus.org>
   Sat, 25 Sep 2010 16:12:13 +0200
   
      Norbert Koch <viteno@xemacs.org> writes:
      
      > Process *nnimap* segmentation fault 
      
      Segmentation fault?  That seems excessive.  :-)
      
      Anyway, I've now tested STARTTLS with a test server, and ironed out a
      bug in the connection code, so it now Works For Me.  Give it a whirl...
      
   
   
   Works for me too. But the nnimap buffer seems to record the complete
   content of every mail.  The ssl stream (with gmail) does not do that, just
   the headers.

I restarted emacs and the starttls and ssl nnimap buffers  behave similarly.
Is there anything that controls what's shown in those buffers?

Eythan 

Re: STARTTLS support

[Lars Magne Ingebrigtsen]

eythanweg@gmail.com (Eythan Weg) writes:

> I restarted emacs and the starttls and ssl nnimap buffers  behave similarly.
> Is there anything that controls what's shown in those buffers?

They are internal buffers used for communication with the server.
You're not supposed to look in them for anything but debugging
purposes.  (They're supposed to start with a space, so you don't see
them at all, but during the debugging phase it's easier to have them
readily accessible.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[CHENG Gao]

*On Sat, 25 Sep 2010 16:12:13 +0200
* Also sprach Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Norbert Koch <viteno@xemacs.org> writes:
>
>> Process *nnimap* segmentation fault 
>
> Segmentation fault?  That seems excessive.  :-)
>
> Anyway, I've now tested STARTTLS with a test server, and ironed out a
> bug in the connection code, so it now Works For Me.  Give it a whirl...

Thanks. It workes for me too.

Please forgive me for asking one stupid question: what's the benefit of
STARTTLS over TLS (if server supports STARTTLS over IMAP)?

Re: STARTTLS support

[Lars Magne Ingebrigtsen]

CHENG Gao <chenggao@cyberhut.org> writes:

> Please forgive me for asking one stupid question: what's the benefit of
> STARTTLS over TLS (if server supports STARTTLS over IMAP)?

Isn't STARTTLS just a hack to provide encryption without having to open
more ports in the firewall, sort of?  STARTTLS is just that you contact
a server the normal way, then say "STARTTLS", and then you start
negotiating the encryption and stuff.

TLS is just contacting a socket and then doing the encryption
negotiation immediately, without saying "STARTTLS" in clear text first.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[Andreas Schwab]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Isn't STARTTLS just a hack to provide encryption without having to open
> more ports in the firewall, sort of?

Not at all, STARTTLS is the recommended way to use TLS.  See RFC2595,
section 7.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: STARTTLS support

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> Isn't STARTTLS just a hack to provide encryption without having to
LMI> open more ports in the firewall, sort of?

From the security perspective, having to use a separate port for TLS is
considered the hack.

It is better to have a single port.  And, when the starttls command is
issued first, nothing which goes over the wire before TLS starts gives
away any more info than the TLS handshake itself.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: STARTTLS support

[CHENG Gao]

*On Sat, 25 Sep 2010 21:24:01 +0200
* Also sprach Lars Magne Ingebrigtsen <larsi@gnus.org>:

> CHENG Gao <chenggao@cyberhut.org> writes:
>
>> Please forgive me for asking one stupid question: what's the benefit of
>> STARTTLS over TLS (if server supports STARTTLS over IMAP)?
>
> Isn't STARTTLS just a hack to provide encryption without having to open
> more ports in the firewall, sort of?  STARTTLS is just that you contact
> a server the normal way, then say "STARTTLS", and then you start
> negotiating the encryption and stuff.
>
> TLS is just contacting a socket and then doing the encryption
> negotiation immediately, without saying "STARTTLS" in clear text first.

Thanks for your explanation.

I am wondering if starttls.el and tls.el could be merged. To my
understanding, they are to some extent same thing (anyway starttls is
tls). According to manual starttls is recommended way (if server
supports it), and gnutls is recommended tool. So IMHO starttls.el could
be merged into tls.el, and change starttls-program default to gnutls
instead of homebrewed starttls. Or even remove support of starttls (last
version is in 2005)?

Re: STARTTLS support

[Ted Zlatanov]

On Sun, 26 Sep 2010 12:20:08 +0800 CHENG Gao <chenggao@cyberhut.org> wrote: 

CG> I am wondering if starttls.el and tls.el could be merged. To my
CG> understanding, they are to some extent same thing (anyway starttls is
CG> tls). According to manual starttls is recommended way (if server
CG> supports it), and gnutls is recommended tool. So IMHO starttls.el could
CG> be merged into tls.el, and change starttls-program default to gnutls
CG> instead of homebrewed starttls. Or even remove support of starttls (last
CG> version is in 2005)?

Let me finish the GnuTLS integration work (currently stuck because of a
strange handshake problem) and then we can work on a sensible way to
have a single interface in Emacs to handle all the TLS modes.  I'll make
sure to CC ding and emacs-devel for that.  Meanwhile, if you or anyone
else wants to take a shot at fixing it, I've checked in my latest
(broken) GnuTLS work into the Emacs repo.

Ted

Re: STARTTLS support

[CHENG Gao]

*On Sun, 26 Sep 2010 01:17:54 -0500
* Also sprach Ted Zlatanov <tzz@lifelogs.com>:

> On Sun, 26 Sep 2010 12:20:08 +0800 CHENG Gao <chenggao@cyberhut.org> wrote: 
>
> CG> I am wondering if starttls.el and tls.el could be merged. To my
> CG> understanding, they are to some extent same thing (anyway starttls is
> CG> tls). According to manual starttls is recommended way (if server
> CG> supports it), and gnutls is recommended tool. So IMHO starttls.el could
> CG> be merged into tls.el, and change starttls-program default to gnutls
> CG> instead of homebrewed starttls. Or even remove support of starttls (last
> CG> version is in 2005)?
>
> Let me finish the GnuTLS integration work (currently stuck because of a
> strange handshake problem) and then we can work on a sensible way to
> have a single interface in Emacs to handle all the TLS modes.  I'll make
> sure to CC ding and emacs-devel for that.  Meanwhile, if you or anyone
> else wants to take a shot at fixing it, I've checked in my latest
> (broken) GnuTLS work into the Emacs repo.
>
> Ted

I just pulled and built. I confirm it works, just I need to input login
info to connect to imap servers. Should I set some variables to make it
use .authinfo(.gpg)?

contrib/ssl.el? (was: Re: STARTTLS support)

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 682 bytes --]

On Sun, Sep 26 2010, CHENG Gao wrote:

> I am wondering if starttls.el and tls.el could be merged. To my
> understanding, they are to some extent same thing (anyway starttls is
> tls). According to manual starttls is recommended way (if server
> supports it), and gnutls is recommended tool. So IMHO starttls.el could
> be merged into tls.el, and change starttls-program default to gnutls
> instead of homebrewed starttls. Or even remove support of starttls (last
> version is in 2005)?

That make me wonder: is also contrib/ssl.el any useful to any user?
Maybe XEmacs user, but I doubt so?

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: contrib/ssl.el?

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> That make me wonder: is also contrib/ssl.el any useful to any user?
> Maybe XEmacs user, but I doubt so?

I doubt so, too.  I think it can probably be removed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[Ted Zlatanov]

On Sun, 26 Sep 2010 15:17:55 +0800 CHENG Gao <chenggao@cyberhut.org> wrote: 

CG> I just pulled and built. I confirm it works, just I need to input login
CG> info to connect to imap servers. Should I set some variables to make it
CG> use .authinfo(.gpg)?

Wait, what?  gnutls.el's version of `starttls-open-stream' and
`open-ssl-stream' work for you?  Are you sure you're not using the
regular ssl.el and starttls.el?  Follow the instructions in my e-mail to
emacs-devel to be sure you're testing the right things.  Those functions
were failing for me last night at the handshake stage.

Ted

Re: STARTTLS support

[CHENG Gao]

*On Sun, 26 Sep 2010 07:18:34 -0500
* Also sprach Ted Zlatanov <tzz@lifelogs.com>:

> On Sun, 26 Sep 2010 15:17:55 +0800 CHENG Gao <chenggao@cyberhut.org> wrote: 
>
> CG> I just pulled and built. I confirm it works, just I need to input login
> CG> info to connect to imap servers. Should I set some variables to make it
> CG> use .authinfo(.gpg)?
>
> Wait, what?  gnutls.el's version of `starttls-open-stream' and
> `open-ssl-stream' work for you?  Are you sure you're not using the
> regular ssl.el and starttls.el?  Follow the instructions in my e-mail to
> emacs-devel to be sure you're testing the right things.  Those functions
> were failing for me last night at the handshake stage.
>
> Ted

I am not sure but I suppose so.

I use same gnus settings and same .authinfo.gpg. Before your gnutls
source is pulled, I need not input my login info to access my imap
servers. But now I have to input login info for each server. So I
suppose Gnus is using your new gnutls code. But I am not 100% sure.

Re: STARTTLS support

[Lars Magne Ingebrigtsen]

CHENG Gao <chenggao@cyberhut.org> writes:

> I use same gnus settings and same .authinfo.gpg. Before your gnutls
> source is pulled, I need not input my login info to access my imap
> servers. But now I have to input login info for each server. So I
> suppose Gnus is using your new gnutls code. But I am not 100% sure.

I don't think the gnutls stuff is involved with any password prompting.

(setq debug-on-quit t) and `C-g' when it asks you.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[CHENG Gao]

*On Sun, 26 Sep 2010 07:18:34 -0500
* Also sprach Ted Zlatanov <tzz@lifelogs.com>:

> On Sun, 26 Sep 2010 15:17:55 +0800 CHENG Gao <chenggao@cyberhut.org> wrote: 
>
> CG> I just pulled and built. I confirm it works, just I need to input login
> CG> info to connect to imap servers. Should I set some variables to make it
> CG> use .authinfo(.gpg)?
>
> Wait, what?  gnutls.el's version of `starttls-open-stream' and
> `open-ssl-stream' work for you?  Are you sure you're not using the
> regular ssl.el and starttls.el?  Follow the instructions in my e-mail to
> emacs-devel to be sure you're testing the right things.  Those functions
> were failing for me last night at the handshake stage.
>
> Ted

Sorry I was wrong.

I added (require 'gnutls) into .gnus (in face I did a brutal force thing
- temporarily deleted tls.el and starttls.el), and I got fail messages.

So no, it doesn't work.

Re: STARTTLS support

[Dave Goldberg]

> Anyway, I've now tested STARTTLS with a test server, and ironed out a
> bug in the connection code, so it now Works For Me.  Give it a whirl...

It's working great for me now (git pull as of 5 minutes ago - I'm not
familiar with git - is there a way to report a version snapshot?).
auth-source works as well (unlike another report I am only prompted
once for the passphrase to .authinfo.gpg and everything else works
like a charm.  

One minor glitch, utf7 is apparently not autoloaded nor required so I
have to do that manually before starting gnus or it fails at some
point (it's not been consistent) with utf7-encode being undefined.

Thanks
-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: STARTTLS support

[Florian Ragwitz]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

Dave Goldberg <david.goldberg6@verizon.net> writes:
> > Anyway, I've now tested STARTTLS with a test server, and ironed out a
> > bug in the connection code, so it now Works For Me.  Give it a
> > whirl...

It works just fine for me now as well. starttls in Gnus 5.13 never did.

> It's working great for me now (git pull as of 5 minutes ago - I'm not
> familiar with git - is there a way to report a version snapshot?).

Usually `git describe` is most useful, but that relies on annotated tags
in the repository, which currently aren't there for Gnus. As an
alternative, the commit IDs are a good way of identifying a certain
snapshot as well, although you'll have to have a repository around to
make some sense of it. You can retrieve that with, for example,
`git rev-parse HEAD'.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: STARTTLS support

[Frank Schmitt]

Dave Goldberg <david.goldberg6@verizon.net> writes:

>> Anyway, I've now tested STARTTLS with a test server, and ironed out a
>> bug in the connection code, so it now Works For Me.  Give it a whirl...
>
> It's working great for me now (git pull as of 5 minutes ago - I'm not
> familiar with git - is there a way to report a version snapshot?).
> auth-source works as well (unlike another report I am only prompted
> once for the passphrase to .authinfo.gpg and everything else works
> like a charm.  
>
> One minor glitch, utf7 is apparently not autoloaded nor required so I
> have to do that manually before starting gnus or it fails at some
> point (it's not been consistent) with utf7-encode being undefined.

You have (require 'gnus-load) in ~/.emacs?

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: STARTTLS support

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> One minor glitch, utf7 is apparently not autoloaded nor required so I
> have to do that manually before starting gnus or it fails at some
> point (it's not been consistent) with utf7-encode being undefined.

Fixed now.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[Tibor Simko]

On Mon, 27 Sep 2010, Florian Ragwitz wrote:
> Usually `git describe` is most useful, but that relies on annotated
> tags in the repository, which currently aren't there for Gnus.

`git describe --tags' matches non-annotated tags too.

Best regards
-- 
Tibor Simko

Re: STARTTLS support

[Dave Goldberg]

> One minor glitch, utf7 is apparently not autoloaded nor required so I
> have to do that manually before starting gnus or it fails at some
> point (it's not been consistent) with utf7-encode being undefined.

One other issue.  GCC is not working on my cyrus server - it is on Exchange.  From a recent attempt, I get this in the debug log:

* FLAGS (%Answered %Flagged %Draft %Deleted %Seen gnus-save gnus-forward gnus-dormant $Forwarded $MDNSent gnus-expire $Label1 $Label2 $Label3 $Label4 $Label5 NonJunk Junk)
* OK [PERMANENTFLAGS (%Answered %Flagged %Draft %Deleted %Seen gnus-save gnus-forward gnus-dormant $Forwarded $MDNSent gnus-expire $Label1 $Label2 $Label3 $Label4 $Label5 NonJunk Junk %*)]  
* 228 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 974143455]  
* OK [UIDNEXT 96583]  
* OK [NOMODSEQ] Sorry, modsequences have not been enabled on this mailbox
* OK [URLMECH INTERNAL]
1959 OK [READ-WRITE] Completed
* 1 FETCH (FLAGS (%Flagged %Seen) UID 64158)
* 2 FETCH (FLAGS (%Flagged %Seen) UID 87316)
* 3 FETCH (FLAGS (%Flagged %Seen) UID 87832)
[lots more of these]
1960 OK Completed (0.000 sec)

My guess is that 228 EXISTS is an error from the server that it thinks
the message I'm trying to store already is already there, but it is
not.  This is a GCC from a brand new message, not a reply or anything
so there is no existing message id in there and looking at the
Exchange GCC folder, the message-id's I generate are unique.  Also, I
tried to B c a message from the Exchange folder to Cyrus and got the
same result.

Thanks,
-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: STARTTLS support

[Steinar Bang]

>>>>> Dave Goldberg <david.goldberg6@verizon.net>:

> ...  GCC is not working on my cyrus server - it is on Exchange.

That sentence confused me.  Isn't cyrus and Exchange two different
IMAP servers?

Or do you have cyrus proxying an Exchange server?  I didn't know cyrus
could do proxying?

Re: STARTTLS support

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> One other issue.  GCC is not working on my cyrus server - it is on
> Exchange.  From a recent attempt, I get this in the debug log:

That's the server communication buffer, and not the debug log, and it's
the response to a different command, so it doesn't help much.  :-)

> Also, I tried to B c a message from the Exchange folder to Cyrus and
> got the same result.

I'm wondering whether different servers use a vaguely different syntax
for ACCEPT.  Do any of them require a "." on a line of its own to end
the message?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[Dave Goldberg]

>>>>> Dave Goldberg <david.goldberg6@verizon.net>:
>> ...  GCC is not working on my cyrus server - it is on Exchange.

> That sentence confused me.  Isn't cyrus and Exchange two different
> IMAP servers?

Sorry for not being clear.  I have two nnimap servers, one cyrus and
one Exchange.  I GCC to both of them (basically I use the cyrus server
as a fast access cache of what I'm dealing with in the current month
and Exchange is the corporate email archive).  The GCC's are properly
being written to the Sent Items folder on Exchange but not to cyrus.



-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: STARTTLS support

[Dave Goldberg]

> Dave Goldberg <david.goldberg6@verizon.net> writes:
>> One other issue.  GCC is not working on my cyrus server - it is on
>> Exchange.  From a recent attempt, I get this in the debug log:

> That's the server communication buffer, and not the debug log, and it's
> the response to a different command, so it doesn't help much.  :-)

What is the debug log?  I didn't see anything obvious in M-x
list-buffers.

-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: STARTTLS support

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> Sorry for not being clear.  I have two nnimap servers, one cyrus and
> one Exchange.  I GCC to both of them (basically I use the cyrus server
> as a fast access cache of what I'm dealing with in the current month
> and Exchange is the corporate email archive).  The GCC's are properly
> being written to the Sent Items folder on Exchange but not to cyrus.

I've now pushed a change that sends "." at the end of the ACCEPT.  Could
you check whether that helps?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> What is the debug log?  I didn't see anything obvious in M-x
> list-buffers.

There's just a log of commands sent to the server, and that's called
"*imap log*".  There's no log of the stuff we get back from the server.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[Gijs Hillenius]

On 27 Sep 2010, Lars Magne Ingebrigtsen wrote:

> Dave Goldberg <david.goldberg6@verizon.net> writes:
>
>> Sorry for not being clear.  I have two nnimap servers, one cyrus and
>> one Exchange.  I GCC to both of them (basically I use the cyrus server
>> as a fast access cache of what I'm dealing with in the current month
>> and Exchange is the corporate email archive).  The GCC's are properly
>> being written to the Sent Items folder on Exchange but not to cyrus.
>
> I've now pushed a change that sends "." at the end of the ACCEPT.  Could
> you check whether that helps?

Ha! at least I am not alone on this. my cyrus also is not gnus-gcc'ing
my sent message.

I whirled a new gnus just now. Still not. (sorry). Is there a switch I
can throw to increase the verbosity in the *imap log*?

Re: STARTTLS support

[Lars Magne Ingebrigtsen]

Gijs Hillenius <gijs@hillenius.net> writes:

> I whirled a new gnus just now. Still not. (sorry). Is there a switch I
> can throw to increase the verbosity in the *imap log*?

Nope, sorry.

Try edebugging through `nnimap-request-accept-article' and see what
happens. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: STARTTLS support

[Dave Goldberg]

> Gijs Hillenius <gijs@hillenius.net> writes:
>> I whirled a new gnus just now. Still not. (sorry). Is there a switch I
>> can throw to increase the verbosity in the *imap log*?

> Nope, sorry.

> Try edebugging through `nnimap-request-accept-article' and see what
> happens. 

OK, I'll try that when I get back to work tomorrow.  Unfortunately
remote access to work is limited to an interface that doesn't work
with Gnus.

-- 
Dave Goldberg
david.goldberg6@verizon.net

nnimap-request-accept-article vs Cyrus IMAP

[Dave Goldberg]

>> Gijs Hillenius <gijs@hillenius.net> writes:
>>> I whirled a new gnus just now. Still not. (sorry). Is there a switch I
>>> can throw to increase the verbosity in the *imap log*?

>> Nope, sorry.

>> Try edebugging through `nnimap-request-accept-article' and see what
>> happens. 

> OK, I'll try that when I get back to work tomorrow.  Unfortunately
> remote access to work is limited to an interface that doesn't work
> with Gnus.

I stepped through nnimap-request-accept-article on both Cyrus and
Exchange servers while doing GCC.  In the Exchange case, the function
winds up with an "OK APPEND completed" message from the server.  In
the Cyrus case I get "NO Message contains bare newlines" back from the
server.  

I did this with M-x debug-on-entry, which was quite time consuming,
and if I've missed reporting anything important, I have to do it
again.  Is there a way to debug the function such that it runs through
but the steps and results are logged to a buffer?  That would make it
much easier to look for things like this.

Thanks,
-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: nnimap-request-accept-article vs Cyrus IMAP

[Daniel Pittman]

Dave Goldberg <david.goldberg6@verizon.net> writes:
>>> Gijs Hillenius <gijs@hillenius.net> writes:
>>>> I whirled a new gnus just now. Still not. (sorry). Is there a switch I
>>>> can throw to increase the verbosity in the *imap log*?
>
>>> Nope, sorry.
>
>>> Try edebugging through `nnimap-request-accept-article' and see what
>>> happens. 
>
>> OK, I'll try that when I get back to work tomorrow.  Unfortunately
>> remote access to work is limited to an interface that doesn't work
>> with Gnus.
>
> I stepped through nnimap-request-accept-article on both Cyrus and Exchange
> servers while doing GCC.  In the Exchange case, the function winds up with
> an "OK APPEND completed" message from the server.  In the Cyrus case I get
> "NO Message contains bare newlines" back from the server.

Oh, Cyrus, you picky thing.  It hates non-RFC email with a passion and will
reject it out of hand where other tools (eg: *everything* else) will accept
it and just cope.[1]

In this case the message that is being added ends, as noted, with something
other than the spec-mandated CR-LF sequence.  The same sort of fault will
occur for anything with non-7-bit characters in headers[2], and so on.


Gnus, or the IMAP thing, is going to have to be much more careful about
encoding of email as it goes in to IMAP for this to work.

        Daniel

Footnotes: 
[1]  They may fail in other ways, by guesstimating size, or not being able to
     search or sort that email, etc, though.

[2]  ...though there is a setting to have it mutilate them that /may/ apply
     other than in the LMTP protocol.

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: nnimap-request-accept-article vs Cyrus IMAP

[Robert Pluim]

Daniel Pittman <daniel@rimspace.net> writes:

> Oh, Cyrus, you picky thing.  It hates non-RFC email with a passion and will
> reject it out of hand where other tools (eg: *everything* else) will accept
> it and just cope.[1]

You say that like it's a bad thing :)

Robert

Re: nnimap-request-accept-article vs Cyrus IMAP

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> I stepped through nnimap-request-accept-article on both Cyrus and
> Exchange servers while doing GCC.  In the Exchange case, the function
> winds up with an "OK APPEND completed" message from the server.  In
> the Cyrus case I get "NO Message contains bare newlines" back from the
> server.  

Ah, right.  I've now done a \n->\r\n conversion in that function, so it
should possibly work now.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap-request-accept-article vs Cyrus IMAP

[Dave Goldberg]

> Dave Goldberg <david.goldberg6@verizon.net> writes:
>> I stepped through nnimap-request-accept-article on both Cyrus and
>> Exchange servers while doing GCC.  In the Exchange case, the function
>> winds up with an "OK APPEND completed" message from the server.  In
>> the Cyrus case I get "NO Message contains bare newlines" back from the
>> server.  

> Ah, right.  I've now done a \n->\r\n conversion in that function, so it
> should possibly work now.

Well it certainly does do the GCC, but I get this backtrace after the message is written:

Debugger entered--Lisp error: (wrong-type-argument number-char-or-marker-p nil)
  gnus-add-to-range(((1 . 96703)) (nil))
  gnus-compute-read-articles("nnimap+cyrus:INBOX.in-box" (nil))
  gnus-group-make-articles-read("nnimap+cyrus:INBOX.in-box" (nil))
  gnus-group-mark-article-read("nnimap+cyrus:INBOX.in-box" nil)
  gnus-inews-do-gcc()
  gnus-agent-possibly-do-gcc()
  run-hooks(gnus-agent-possibly-do-gcc)
  message-send(nil)
  message-send-and-exit(nil)
  call-interactively(message-send-and-exit)

I guess this is triggered by my having gnus-gcc-mark-as-read set to t?

Thanks,
-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: nnimap-request-accept-article vs Cyrus IMAP

[Dave Goldberg]

> Well it certainly does do the GCC, but I get this backtrace after the message is written:


> [...]

> I guess this is triggered by my having gnus-gcc-mark-as-read set to t?

Yup, that's the trigger.  If I set gnus-gcc-mark-as-read to nil, the
process runs to completion.

-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: nnimap-request-accept-article vs Cyrus IMAP

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 878 bytes --]

On Wed, Sep 29 2010, Dave Goldberg wrote:

> Well it certainly does do the GCC, but I get this backtrace after the message is written:
>
> Debugger entered--Lisp error: (wrong-type-argument number-char-or-marker-p nil)
>   gnus-add-to-range(((1 . 96703)) (nil))
>   gnus-compute-read-articles("nnimap+cyrus:INBOX.in-box" (nil))
>   gnus-group-make-articles-read("nnimap+cyrus:INBOX.in-box" (nil))
>   gnus-group-mark-article-read("nnimap+cyrus:INBOX.in-box" nil)
>   gnus-inews-do-gcc()
>   gnus-agent-possibly-do-gcc()
>   run-hooks(gnus-agent-possibly-do-gcc)
>   message-send(nil)
>   message-send-and-exit(nil)
>   call-interactively(message-send-and-exit)
>
> I guess this is triggered by my having gnus-gcc-mark-as-read set to t?

Same backtrace here, but I use Dovecot. ;)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: nnimap-request-accept-article vs Cyrus IMAP

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

>   gnus-group-mark-article-read("nnimap+cyrus:INBOX.in-box" nil)

The article there is nil, which means that this didn't return anything.
Can you find a Message-ID from the group below, and then try evaling the
following: 

(nnimap-find-article-by-message-id "nnimap+cyrus:INBOX.in-box" message-id)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap-request-accept-article vs Cyrus IMAP

[Gijs Hillenius]

On 29 Sep 2010, Lars Magne Ingebrigtsen wrote:

> Dave Goldberg <david.goldberg6@verizon.net> writes:
>
>> I stepped through nnimap-request-accept-article on both Cyrus and
>> Exchange servers while doing GCC.  In the Exchange case, the function
>> winds up with an "OK APPEND completed" message from the server.  In
>> the Cyrus case I get "NO Message contains bare newlines" back from the
>> server.  
>
> Ah, right.  I've now done a \n->\r\n conversion in that function, so it
> should possibly work now.

Works here.

Thanks, Lars!

Re: nnimap-request-accept-article vs Cyrus IMAP

[Dave Goldberg]

> Dave Goldberg <david.goldberg6@verizon.net> writes:
>> gnus-group-mark-article-read("nnimap+cyrus:INBOX.in-box" nil)

> The article there is nil, which means that this didn't return anything.
> Can you find a Message-ID from the group below, and then try evaling the
> following: 

> (nnimap-find-article-by-message-id "nnimap+cyrus:INBOX.in-box" message-id)

It returns nil.  And from other reports this does not seem to be a
cyrus specific issue.


-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: nnimap-request-accept-article vs Cyrus IMAP

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

>> (nnimap-find-article-by-message-id "nnimap+cyrus:INBOX.in-box" message-id)

> It returns nil.  And from other reports this does not seem to be a
> cyrus specific issue.

What's in the *nnimap cyrus ...* buffer after evaling that?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap-request-accept-article vs Cyrus IMAP

[Dave Goldberg]

> Dave Goldberg <david.goldberg6@verizon.net> writes:
>>> (nnimap-find-article-by-message-id "nnimap+cyrus:INBOX.in-box" message-id)

>> It returns nil.  And from other reports this does not seem to be a
>> cyrus specific issue.

> What's in the *nnimap cyrus ...* buffer after evaling that?

NO Mailbox does not exist^M.

So perhaps my earlier speculation about it being the search string is
wrong.

-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: nnimap-request-accept-article vs Cyrus IMAP

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

>>>> (nnimap-find-article-by-message-id "nnimap+cyrus:INBOX.in-box" message-id)
>
>>> It returns nil.  And from other reports this does not seem to be a
>>> cyrus specific issue.
>
>> What's in the *nnimap cyrus ...* buffer after evaling that?
>
> NO Mailbox does not exist^M.
>
> So perhaps my earlier speculation about it being the search string is
> wrong.

My mistake.  It should be:

(nnimap-find-article-by-message-id "INBOX.in-box" message-id)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap-request-accept-article vs Cyrus IMAP

[Dave Goldberg]

> Dave Goldberg <david.goldberg6@verizon.net> writes:
>>>>> (nnimap-find-article-by-message-id "nnimap+cyrus:INBOX.in-box" message-id)
>> 
>>>> It returns nil.  And from other reports this does not seem to be a
>>>> cyrus specific issue.
>> 
>>> What's in the *nnimap cyrus ...* buffer after evaling that?
>> 
>> NO Mailbox does not exist^M.
>> 
>> So perhaps my earlier speculation about it being the search string is
>> wrong.

> My mistake.  It should be:

> (nnimap-find-article-by-message-id "INBOX.in-box" message-id)

So it is the search string having ^M at the end after all.  If I run
the function exactly as you say, then I get the article number.  If I
run it with the ^M at the end of the message-id string, it fails with

BAD Missing required argument to Search header

and the function returns nil.

Thanks,
-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: nnimap-request-accept-article vs Cyrus IMAP

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> So it is the search string having ^M at the end after all.  If I run
> the function exactly as you say, then I get the article number. 

Yup.  Fixed and pushed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap-request-accept-article vs Cyrus IMAP

[Dave Goldberg]

> Dave Goldberg <david.goldberg6@verizon.net> writes:
>> So it is the search string having ^M at the end after all.  If I run
>> the function exactly as you say, then I get the article number. 

> Yup.  Fixed and pushed.

Still erroring out with the same backtrace.  The IMAP search is
working now, but the function (nnimap-find-article-by-message-id) is
returning nil.

Thanks,
-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: nnimap-request-accept-article vs Cyrus IMAP

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> Still erroring out with the same backtrace.  The IMAP search is
> working now, but the function (nnimap-find-article-by-message-id) is
> returning nil.

D'oh!  That was a pretty stupid coding error.

I've now fixed and tested and everything.  So I hope it works this
time.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap-request-accept-article vs Cyrus IMAP

[Dave Goldberg]

>> Dave Goldberg <david.goldberg6@verizon.net> writes:
>>> So it is the search string having ^M at the end after all.  If I run
>>> the function exactly as you say, then I get the article number. 

>> Yup.  Fixed and pushed.

> Still erroring out with the same backtrace.  The IMAP search is
> working now, but the function (nnimap-find-article-by-message-id) is
> returning nil.

I just ran nnimap-find-article-by-message-id under edebug.  The IMAP
search is not working - I was apparently looking at the buffer after
an erase and subsequent processing.

In response to the EXAMINE I get

224 NO Mailbox does not exist

And then in response to the UID SEARCH I get

225 BAD Please select a mailbox first


-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: nnimap-request-accept-article vs Cyrus IMAP

[Lars Magne Ingebrigtsen]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> I just ran nnimap-find-article-by-message-id under edebug.  The IMAP
> search is not working - I was apparently looking at the buffer after
> an erase and subsequent processing.
>
> In response to the EXAMINE I get
>
> 224 NO Mailbox does not exist

Huh?  What did "*imap log*" say it sent?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap-request-accept-article vs Cyrus IMAP

[Dave Goldberg]

> Dave Goldberg <david.goldberg6@verizon.net> writes:
>> I just ran nnimap-find-article-by-message-id under edebug.  The IMAP
>> search is not working - I was apparently looking at the buffer after
>> an erase and subsequent processing.
>> 
>> In response to the EXAMINE I get
>> 
>> 224 NO Mailbox does not exist

> Huh?  What did "*imap log*" say it sent?

Sigh, ignore that.  I think something must have got out of sync after
I messed around with edebugging that first time.  After installing the
latest changes and going through it again, the search is indeed
successful and the function correctly returns the article number.

Thanks,
-- 
Dave Goldberg
david.goldberg6@verizon.net