nnimap ports and protocols

nnimap ports and protocols

[Lars Magne Ingebrigtsen]

In the absence of user-specified ports and protocols, would it make
sense just to try the likely ones?

That is, test (in order):

1) imaps
2) imap + starttls
3) imap

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap ports and protocols

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> In the absence of user-specified ports and protocols, would it make sense
> just to try the likely ones?

You wouldn't be the first person to do it, but watch out for firewalls that
eat the SYN packet silently: having Gnus hang for a significant timeout on the
first connection attempt would suck.  Doing that every time...

        Daniel

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: nnimap ports and protocols

[Ted Zlatanov]

On Fri, 01 Oct 2010 11:00:13 +1000 Daniel Pittman <daniel@rimspace.net> wrote: 

DP> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> In the absence of user-specified ports and protocols, would it make sense
>> just to try the likely ones?

DP> You wouldn't be the first person to do it, but watch out for firewalls that
DP> eat the SYN packet silently: having Gnus hang for a significant timeout on the
DP> first connection attempt would suck.  Doing that every time...

The nnimap stream parameter should be a list the user can customize,
attempted in order.  The default can be what Lars listed.

Then Gnus can ask the user if they want to customize the nnimap stream
parameter when a connection fails.

Ted

Re: nnimap ports and protocols

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1297 bytes --]

On Fri, 01 Oct 2010 08:43:46 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Fri, 01 Oct 2010 11:00:13 +1000 Daniel Pittman <daniel@rimspace.net> wrote: 
DP> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>> In the absence of user-specified ports and protocols, would it make sense
>>> just to try the likely ones?

DP> You wouldn't be the first person to do it, but watch out for firewalls that
DP> eat the SYN packet silently: having Gnus hang for a significant timeout on the
DP> first connection attempt would suck.  Doing that every time...

TZ> The nnimap stream parameter should be a list the user can customize,
TZ> attempted in order.  The default can be what Lars listed.

TZ> Then Gnus can ask the user if they want to customize the nnimap stream
TZ> parameter when a connection fails.

Attached is a patch that:

1) has FIXMEs all over

2) removes the dependency on nnimap-shell-program in favor of a more
structured nnimap-stream

3) makes nnimap-stream an alist of (stream . helper) where the helper is
currently only meaningful for the 'shell stream.  I plan to use it for
GnuTLS as well.

4) provides a default nnimap-stream that tries ssl, then network, then
shell

I'm sure this can be done better, but wanted to show how I thought
streams should work for nnimap.

Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnimap-stream.patch --]
[-- Type: text/x-diff, Size: 3921 bytes --]

diff --git a/lisp/nnimap.el b/lisp/nnimap.el
index 70aa573..89fcff2 100644
--- a/lisp/nnimap.el
+++ b/lisp/nnimap.el
@@ -54,10 +54,17 @@
 If nnimap-stream is `ssl', this will default to `imaps'.  If not,
 it will default to `imap'.")
 
-(defvoo nnimap-stream 'ssl
+(defvoo nnimap-stream '((ssl . "")
+                        (network . "")
+                        (shell .  (if (boundp 'imap-shell-program)
+                                      (if (listp imap-shell-program)
+                                          (car imap-shell-program)
+                                        imap-shell-program)
+                                    "ssh %s imapd")))
   "How nnimap will talk to the IMAP server.
 Values are `ssl', `network', `starttls' or `shell'.")
 
+;; FIXME: remove this in favor of nnimap-stream parameters
 (defvoo nnimap-shell-program (if (boundp 'imap-shell-program)
 				 (if (listp imap-shell-program)
 				     (car imap-shell-program)
@@ -245,12 +252,12 @@ textual parts.")
     (push (current-buffer) nnimap-process-buffers)
     (current-buffer)))
 
-(defun nnimap-open-shell-stream (name buffer host port)
+(defun nnimap-open-shell-stream (name buffer host port program)
   (let ((process-connection-type nil))
     (start-process name buffer shell-file-name
 		   shell-command-switch
 		   (format-spec
-		    nnimap-shell-program
+		    program
 		    (format-spec-make
 		     ?s host
 		     ?p port)))))
@@ -293,9 +300,13 @@ textual parts.")
       (let* ((coding-system-for-read 'binary)
 	     (coding-system-for-write 'binary)
 	     (port nil)
+             ;; FIXME: this should cycle through the streams and if one fails, offer to customize them
+             (stream-struct (or (car-safe nnimap-stream) '(starttls . "")))
+             (stream (car stream-struct))
+             (stream-helper (cdr stream-struct))
 	     (ports
 	      (cond
-	       ((eq nnimap-stream 'network)
+	       ((eq stream 'network)
 		(open-network-stream
 		 "*nnimap*" (current-buffer) nnimap-address
 		 (setq port
@@ -304,19 +315,20 @@ textual parts.")
 			       "imap"
 			     "143"))))
 		'("143" "imap"))
-	       ((eq nnimap-stream 'shell)
+	       ((eq stream 'shell)
 		(nnimap-open-shell-stream
 		 "*nnimap*" (current-buffer) nnimap-address
-		 (setq port (or nnimap-server-port "imap")))
+		 (setq port (or nnimap-server-port "imap"))
+                 stream-helper)
 		'("imap"))
-	       ((eq nnimap-stream 'starttls)
+	       ((eq stream 'starttls)
 		(let ((tls-program (nnimap-extend-tls-programs)))
 		  (open-tls-stream
 		   "*nnimap*" (current-buffer) nnimap-address
 		   (setq port (or nnimap-server-port "imap"))
 		   'starttls))
 		'("imap"))
-	       ((eq nnimap-stream 'ssl)
+	       ((eq stream 'ssl)
 		(open-tls-stream
 		 "*nnimap*" (current-buffer) nnimap-address
 		 (setq port
@@ -332,7 +344,7 @@ textual parts.")
 		      (memq (process-status (nnimap-process nnimap-object))
 			    '(open run))))
 	    (nnheader-report 'nnimap "Unable to contact %s:%s via %s"
-			     nnimap-address port nnimap-stream)
+			     nnimap-address port stream)
 	  (gnus-set-process-query-on-exit-flag (nnimap-process nnimap-object) nil)
 	  (if (not (setq connection-result (nnimap-wait-for-connection)))
 	      (nnheader-report 'nnimap
@@ -352,9 +364,9 @@ textual parts.")
 	      (push (format "%s" nnimap-server-port) ports))
 	    ;; If this is a STARTTLS-capable server, then sever the
 	    ;; connection and start a STARTTLS connection instead.
-	    (when (and (eq nnimap-stream 'network)
+	    (when (and (eq stream 'network)
 		       (member "STARTTLS" (nnimap-capabilities nnimap-object)))
-	      (let ((nnimap-stream 'starttls))
+	      (let ((nnimap-stream '(starttls))) ; FIXME, should be handled with the right nnimap-stream to begin with
 		(let ((tls-process
 		       (nnimap-open-connection buffer)))
 		  ;; If the STARTTLS connection was successful, we

Re: nnimap ports and protocols

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> That is, test (in order):
>
> 1) imaps
> 2) imap + starttls
> 3) imap

I started up Mail on my iMac for the first time, just to see what a
modern MUA does.

It first asked me for my address and my password.  Then it tried
doing...  something based on that address -- I'm guessing it probed for
pop3 and imap ports.

Then it said it couldn't find anything, and asked me to choose an
incoming mail server.  It defaulted to pop3, but had three other
choices: IMAP, Exchange 2007, and Exchange 2007 IMAP (!).  Weird.

Anyway, I chose IMAP, and it presented me with a new dialogue saying
that the certificate was invalid, so it had contacted the imaps port.
(Since the server in question doesn't have an open imap port.)  I
clicked "continue", and it presented me with an overview of everything,
and under the "IMAP" section "ssl encryption" was ticked on.

So I'm guessing it probed imaps and then imap, just like my clever
plan.  And then stored the result.

Which reminds me.  Stuff like this should be stored.  If nnimap has
determined that server X is an imaps server, or whatever, then it
shouldn't have to probe every time Emacs is run.

Does Emacs have any good mechanism for saving data like this?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap ports and protocols

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Which reminds me.  Stuff like this should be stored.  If nnimap has
> determined that server X is an imaps server, or whatever, then it
> shouldn't have to probe every time Emacs is run.
>
> Does Emacs have any good mechanism for saving data like this?

I'd use the customize framework. This gives the added benefit of a ready
GUI for manually tweaking the saved info.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: nnimap ports and protocols

[Lars Magne Ingebrigtsen]

Frank Schmitt <ich@frank-schmitt.net> writes:

> I'd use the customize framework. This gives the added benefit of a ready
> GUI for manually tweaking the saved info.

This data wouldn't be variables as such.  It'd be, like "connecting to
imap server foo is done on port bar using method zot".  Does that work
in the customize framework?  I'm not sure how I would express that...

Or to take a different example -- using tls.el to connect to a server.
Then it would be nice if we'd tried gnutls-cli once, and saw it fail,
and that openssl worked, that we could save that in some way.

*scratches head*

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap ports and protocols

[Ted Zlatanov]

On Fri, 01 Oct 2010 17:34:07 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Frank Schmitt <ich@frank-schmitt.net> writes:
>> I'd use the customize framework. This gives the added benefit of a ready
>> GUI for manually tweaking the saved info.

LMI> This data wouldn't be variables as such.  It'd be, like "connecting to
LMI> imap server foo is done on port bar using method zot".  Does that work
LMI> in the customize framework?  I'm not sure how I would express that...

LMI> Or to take a different example -- using tls.el to connect to a server.
LMI> Then it would be nice if we'd tried gnutls-cli once, and saw it fail,
LMI> and that openssl worked, that we could save that in some way.

My example on extending nnimap-stream could handle this (posted earlier
this morning in this thread).

Another approach is to add a customizeable alist to control this
preference.

Yet another is to save a "nnimap-stream" token in the netrc file.

Ted

Re: nnimap ports and protocols

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> My example on extending nnimap-stream could handle this (posted earlier
> this morning in this thread).

Yes, that would work, but I find that variables that have complex
syntaxes (and we have enough of them) are something users fiddle with
and complain when they break, so if that could be avoided, it would be a
plus. 

> Yet another is to save a "nnimap-stream" token in the netrc file.

This is my current idea I had like five minutes ago, so it should be
great.  :-)  To take nnimap-stream as the example:

1) we'd change the default to nil, signalling that we want Gnus to
figure it out on its own.  If it has a value, then the user has
obviously chosen it, which makes the rest of the stuff irrelevant.

2) if it's nil, then we're in DWIM mode.  We then call
(nnimap-guess-connection-method server).  It'd try the stuff we talked
about earlier, and return its result.

3) this is saved in a new Gnus/Emacs file, called...  er...
~/emacs.d/emacs-memoize, which would have a format like

(nnimap connection-method ("imaps" ssl))

one per line, so that the user can edit it by hand, if necessary.  The
meaning here would be DOMAIN (so that many different packages can use
the same mechanism), TOKEN (that has some meaning in that domain), and
the VALUE.

4) nnimap would call this with the macro
(gnus-memoize 'nnimap 'connection-method
  (nnimap-guess-connection-method server))

So it's like a normal memoizer, only it saves stuff to a file, too.

Does this sound halfway sane?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap ports and protocols

[Ted Zlatanov]

On Fri, 01 Oct 2010 18:25:42 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> My example on extending nnimap-stream could handle this (posted earlier
>> this morning in this thread).

LMI> Yes, that would work, but I find that variables that have complex
LMI> syntaxes (and we have enough of them) are something users fiddle with
LMI> and complain when they break, so if that could be avoided, it would be a
LMI> plus.

I see.  That's reasonable.

>> Yet another is to save a "nnimap-stream" token in the netrc file.

LMI> This is my current idea I had like five minutes ago, so it should be
LMI> great.  :-)  To take nnimap-stream as the example:

LMI> 1) we'd change the default to nil, signalling that we want Gnus to
LMI> figure it out on its own.  If it has a value, then the user has
LMI> obviously chosen it, which makes the rest of the stuff irrelevant.

OK.

LMI> 2) if it's nil, then we're in DWIM mode.  We then call
LMI> (nnimap-guess-connection-method server).  It'd try the stuff we talked
LMI> about earlier, and return its result.

LMI> 3) this is saved in a new Gnus/Emacs file, called...  er...
LMI> ~/emacs.d/emacs-memoize, which would have a format like

LMI> (nnimap connection-method ("imaps" ssl))

LMI> one per line, so that the user can edit it by hand, if necessary.  The
LMI> meaning here would be DOMAIN (so that many different packages can use
LMI> the same mechanism), TOKEN (that has some meaning in that domain), and
LMI> the VALUE.

That seems like a reinvention of the customize facility.  We can do a
non-interactive customization of `gnus-connection-preferences' for
instance and save the result.  I think that's better.

Also the connection preferences need to be keyed by server name/server
address regex with t as the default.

LMI> 4) nnimap would call this with the macro
LMI> (gnus-memoize 'nnimap 'connection-method
LMI>   (nnimap-guess-connection-method server))

LMI> So it's like a normal memoizer, only it saves stuff to a file, too.

LMI> Does this sound halfway sane?

If Customize is inadequate, I would put this in .newsrc.eld, not in a
special new file.  And I would allow `gnus-connection-preferences' to be
a function called with the server name.

Ted

Re: nnimap ports and protocols

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> That seems like a reinvention of the customize facility.  We can do a
> non-interactive customization of `gnus-connection-preferences' for
> instance and save the result.  I think that's better.

That's be a whopping long alist that wouldn't be very understandable to
the customize interface.  I think...

> Also the connection preferences need to be keyed by server name/server
> address regex with t as the default.

No, it would just store what it decided to do for exactly that server.

> If Customize is inadequate, I would put this in .newsrc.eld, not in a
> special new file.  And I would allow `gnus-connection-preferences' to be
> a function called with the server name.

It could be stored in the .newsrc.eld, but we'd then have to supply a
mode to edit it, because the user will want to clear stuff out.  

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap ports and protocols

[Ted Zlatanov]

On Fri, 01 Oct 2010 20:29:11 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> If Customize is inadequate, I would put this in .newsrc.eld, not in a
>> special new file.  And I would allow `gnus-connection-preferences' to be
>> a function called with the server name.

LMI> It could be stored in the .newsrc.eld, but we'd then have to supply a
LMI> mode to edit it, because the user will want to clear stuff out.  

That could be mapped in the Server buffer.  We already have a standard
way of editing forms in Gnus (server edit, topic/group parameter edit,
etc.) so this is not a bad way, usability-wise.

Ted

Re: nnimap ports and protocols

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> That could be mapped in the Server buffer.  We already have a standard
> way of editing forms in Gnus (server edit, topic/group parameter edit,
> etc.) so this is not a bad way, usability-wise.

Yeah, that's true.  I think that's the way to go, then.

What about questions from the tls layer about "do you want to accept
this invalid certificate from the server?" and stuff?  If that's to be
stored in the server, then open-gnutls-stream should return both the
stream and, er, something that says something about this.  Perhaps a
plist of properties, like :certificate :invalid-but-accepted or
something.

Or is that a more global thing?  "I accept this certificate" sounds a
bit more global...  and it's user-generated, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap ports and protocols

[Ted Zlatanov]

On Fri, 01 Oct 2010 20:47:19 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> That could be mapped in the Server buffer.  We already have a standard
>> way of editing forms in Gnus (server edit, topic/group parameter edit,
>> etc.) so this is not a bad way, usability-wise.

LMI> Yeah, that's true.  I think that's the way to go, then.

LMI> What about questions from the tls layer about "do you want to accept
LMI> this invalid certificate from the server?" and stuff?  If that's to be
LMI> stored in the server, then open-gnutls-stream should return both the
LMI> stream and, er, something that says something about this.  Perhaps a
LMI> plist of properties, like :certificate :invalid-but-accepted or
LMI> something.

"don't worry about the rilly technical stuff" ;)

The GnuTLS layer should handle all of that.  Gnus should not know about
certificate management; just like passwords or accepting SSH keys,
that's completely irrelevant to reading mail.

So I think Fgnutls_boot will need work and so on.  It's not bad.  I may
use auth-source.el to store the accepted certificates in a netrc file or
in the Secrets API.

Ted

Re: nnimap ports and protocols

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> The GnuTLS layer should handle all of that.  Gnus should not know about
> certificate management; just like passwords or accepting SSH keys,
> that's completely irrelevant to reading mail.

Yeah, but I was just wondering where the gnutls layer would stash
information like that.  We certainly don't want the user to be queried
about the invalid certificate every time she connects to the IMAP
server, so it needs to be saved somewhere.

> So I think Fgnutls_boot will need work and so on.  It's not bad.  I may
> use auth-source.el to store the accepted certificates in a netrc file or
> in the Secrets API.

The netrc file is for Humans, I think.  Perhaps you can stash this in
Customize?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap ports and protocols

[Ted Zlatanov]

On Fri, 01 Oct 2010 21:17:45 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> The GnuTLS layer should handle all of that.  Gnus should not know about
>> certificate management; just like passwords or accepting SSH keys,
>> that's completely irrelevant to reading mail.

LMI> Yeah, but I was just wondering where the gnutls layer would stash
LMI> information like that.  We certainly don't want the user to be queried
LMI> about the invalid certificate every time she connects to the IMAP
LMI> server, so it needs to be saved somewhere.

>> So I think Fgnutls_boot will need work and so on.  It's not bad.  I may
>> use auth-source.el to store the accepted certificates in a netrc file or
>> in the Secrets API.

LMI> The netrc file is for Humans, I think.  Perhaps you can stash this in
LMI> Customize?  :-)

Maybe.  There's also the "trust file" concept which contains accepted
certificate authorities as a bundle.  Maybe I can create a trust file to
contain accepted server certificates.  GnuTLS doesn't allow multiple
trust files so this may get complicated.  I'll ask gnutls-devel.

Ted

Re: nnimap ports and protocols

[Sivaram Neelakantan]

On Fri, Oct 01 2010,Lars Magne Ingebrigtsen Lars Magne Ingebrigtsen wrote:


[snipped 23 lines]

> 3) this is saved in a new Gnus/Emacs file, called...  er...
> ~/emacs.d/emacs-memoize, which would have a format like
>
> (nnimap connection-method ("imaps" ssl))
>
> one per line, so that the user can edit it by hand, if necessary.  The
> meaning here would be DOMAIN (so that many different packages can use
> the same mechanism), TOKEN (that has some meaning in that domain), and
> the VALUE.
>
> 4) nnimap would call this with the macro
> (gnus-memoize 'nnimap 'connection-method
>   (nnimap-guess-connection-method server))
>
> So it's like a normal memoizer, only it saves stuff to a file, too.
>
> Does this sound halfway sane?

As long as a info message goes into the *Messages* buffer telling the
user where it's trying to pick up the last used method/credentials
from i.e. the file it read to guess the login method, the approach
looks good.

There's nothing quite maddening like "smart" choices made mysteriously
and you don't know where to root around to fix it. :-)



 sivaram
 --