mail-source.el patch to use netrc-parse

mail-source.el patch to use netrc-parse

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1336 bytes --]

The attached patch modifies mail-source.el to use a netrc (~/.authinfo)
file for IMAP fetching.  It will be triggered when
mail-source-authinfo-file is not nil.  It will override the :user and
:password parameters set in mail-sources unconditionally.  This allows
users to store their passwords for an IMAP mail source in the authinfo
file, encrypted if necessary.  I use this for my IMAP fetching and it
worked fine; please test (also if you don't use the feature, make sure
your IMAP fetching works OK).  I won't commit until I get confirmations
it works, since mail fetching is a sensitive area for improvements.

I plan to add this to POP and webmail fetching as well, possibly through
mail-source-bind instead of in each individual fetching function.
Please let me know what you think.

Also, I think instead of the current variables

(setq
 nnimap-authinfo-file "~/.authinfo.enc"
 nntp-authinfo-file "~/.authinfo.enc"
 mail-source-authinfo-file "~/.authinfo.enc"
 smtpmail-auth-credentials "~/.authinfo.enc")

Gnus should have a single specification:

(setq
 authinfo-files '(('nnimap "~/.imap-authinfo.enc")
                  (t "~/.authinfo.enc")))

Then we can derive each of those variables at runtime, if they are not
set:

(setq nnimap-authinfo-file
 (or nnimap-authinfo-file (gnus-get-authinfo-file 'nnimap)))

WDYT?

Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: mail-source.authinfo.patch --]
[-- Type: text/x-diff, Size: 6432 bytes --]

Index: mail-source.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/mail-source.el,v
retrieving revision 7.26
diff -r7.26 mail-source.el
38a39,40
>   (autoload 'netrc-parse "netrc")
>   (autoload 'netrc-machine-user-or-password "netrc")
339a342,348
> (defcustom mail-source-authinfo-file nil
>   "Authinfo file.  
> When set, it will override :user and :password for a mail source
> if that source's server is set in the authinfo file.  See netrc.el"
>   :group 'mail-source
>   :type 'file)
> 
1017,1077c1026,1045
<     (mail-source-run-script
<      prescript (format-spec-make ?p password ?t mail-source-crash-box
< 				 ?s server ?P port ?u user)
<      prescript-delay)
<     (let ((from (format "%s:%s:%s" server user port))
< 	  (found 0)
< 	  (buf (generate-new-buffer " *imap source*"))
< 	  (mail-source-string (format "imap:%s:%s" server mailbox))
< 	  (imap-shell-program (or (list program) imap-shell-program))
< 	  remove)
<       (if (and (imap-open server port stream authentication buf)
< 	       (imap-authenticate
< 		user (or (cdr (assoc from mail-source-password-cache))
< 			 password) buf)
< 	       (imap-mailbox-select mailbox nil buf))
< 	  (let ((coding-system-for-write mail-source-imap-file-coding-system)
< 		str)
< 	    (with-temp-file mail-source-crash-box
< 	      ;; Avoid converting 8-bit chars from inserted strings to
< 	      ;; multibyte.
< 	      (mm-disable-multibyte)
< 	      ;; remember password
< 	      (with-current-buffer buf
< 		(when (and imap-password
< 			   (not (assoc from mail-source-password-cache)))
< 		  (push (cons from imap-password) mail-source-password-cache)))
< 	      ;; if predicate is nil, use all uids
< 	      (dolist (uid (imap-search (or predicate "1:*") buf))
< 		(when (setq str
< 			    (if (imap-capability 'IMAP4rev1 buf)
< 				(caddar (imap-fetch uid "BODY.PEEK[]"
< 						    'BODYDETAIL nil buf))
< 			      (imap-fetch uid "RFC822.PEEK" 'RFC822 nil buf)))
< 		  (push uid remove)
< 		  (insert "From imap " (current-time-string) "\n")
< 		  (save-excursion
< 		    (insert str "\n\n"))
< 		  (while (let ((case-fold-search nil))
< 			   (re-search-forward "^From " nil t))
< 		    (replace-match ">From "))
< 		  (goto-char (point-max))))
< 	      (nnheader-ms-strip-cr))
< 	    (incf found (mail-source-callback callback server))
< 	    (mail-source-delete-crash-box)
< 	    (when (and remove fetchflag)
< 	      (setq remove (nreverse remove))
< 	      (imap-message-flags-add
< 	       (imap-range-to-message-set (gnus-compress-sequence remove))
< 	       fetchflag nil buf))
< 	    (if dontexpunge
< 		(imap-mailbox-unselect buf)
< 	      (imap-mailbox-close nil buf))
< 	    (imap-close buf))
< 	(imap-close buf)
< 	;; We nix out the password in case the error
< 	;; was because of a wrong password being given.
< 	(setq mail-source-password-cache
< 	      (delq (assoc from mail-source-password-cache)
< 		    mail-source-password-cache))
< 	(error "IMAP error: %s" (imap-error-text buf)))
<       (kill-buffer buf)
---
>     (let* ((list ((when mail-source-authinfo-file
> 		    (gnus-message 7 "Parsing authinfo file `%s'."
> 				  mail-source-authinfo-file)
> 		    (netrc-parse mail-source-authinfo-file))))
> 	   (user (if mail-source-authinfo-file
> 		     (netrc-machine-user-or-password
> 		      "login"
> 		      list
> 		      (list server)
> 		      (list port)
> 		      (list "imap" "imaps"))
> 		   user))
> 	   (password (if mail-source-authinfo-file
> 			 (netrc-machine-user-or-password
> 			  "password"
> 			  list
> 			  (list server)
> 			  (list port)
> 			  (list "imap" "imaps"))
> 		       password)))
1079,1082c1047,1111
<        postscript
<        (format-spec-make ?p password ?t mail-source-crash-box
< 			 ?s server ?P port ?u user))
<       found)))
---
>        prescript (format-spec-make ?p password ?t mail-source-crash-box
> 				   ?s server ?P port ?u user)
>        prescript-delay)
>       (let ((from (format "%s:%s:%s" server user port))
> 	    (found 0)
> 	    (buf (generate-new-buffer " *imap source*"))
> 	    (mail-source-string (format "imap:%s:%s" server mailbox))
> 	    (imap-shell-program (or (list program) imap-shell-program))
> 	    remove)
> 	(if (and (imap-open server port stream authentication buf)
> 		 (imap-authenticate
> 		  user (or (cdr (assoc from mail-source-password-cache))
> 			   password) buf)
> 		 (imap-mailbox-select mailbox nil buf))
> 	    (let ((coding-system-for-write mail-source-imap-file-coding-system)
> 		  str)
> 	      (with-temp-file mail-source-crash-box
> 		;; Avoid converting 8-bit chars from inserted strings to
> 		;; multibyte.
> 		(mm-disable-multibyte)
> 		;; remember password
> 		(with-current-buffer buf
> 		  (when (and imap-password
> 			     (not (assoc from mail-source-password-cache)))
> 		    (push (cons from imap-password) mail-source-password-cache)))
> 		;; if predicate is nil, use all uids
> 		(dolist (uid (imap-search (or predicate "1:*") buf))
> 		  (when (setq str
> 			      (if (imap-capability 'IMAP4rev1 buf)
> 				  (caddar (imap-fetch uid "BODY.PEEK[]"
> 						      'BODYDETAIL nil buf))
> 				(imap-fetch uid "RFC822.PEEK" 'RFC822 nil buf)))
> 		    (push uid remove)
> 		    (insert "From imap " (current-time-string) "\n")
> 		    (save-excursion
> 		      (insert str "\n\n"))
> 		    (while (let ((case-fold-search nil))
> 			     (re-search-forward "^From " nil t))
> 		      (replace-match ">From "))
> 		    (goto-char (point-max))))
> 		(nnheader-ms-strip-cr))
> 	      (incf found (mail-source-callback callback server))
> 	      (mail-source-delete-crash-box)
> 	      (when (and remove fetchflag)
> 		(setq remove (nreverse remove))
> 		(imap-message-flags-add
> 		 (imap-range-to-message-set (gnus-compress-sequence remove))
> 		 fetchflag nil buf))
> 	      (if dontexpunge
> 		  (imap-mailbox-unselect buf)
> 		(imap-mailbox-close nil buf))
> 	      (imap-close buf))
> 	  (imap-close buf)
> 	  ;; We nix out the password in case the error
> 	  ;; was because of a wrong password being given.
> 	  (setq mail-source-password-cache
> 		(delq (assoc from mail-source-password-cache)
> 		      mail-source-password-cache))
> 	  (error "IMAP error: %s" (imap-error-text buf)))
> 	(kill-buffer buf)
> 	(mail-source-run-script
> 	 postscript
> 	 (format-spec-make ?p password ?t mail-source-crash-box
> 			   ?s server ?P port ?u user))
> 	found))))

Re: mail-source.el patch to use netrc-parse

[Reiner Steib]

On Tue, Feb 05 2008, Ted Zlatanov wrote:

> The attached patch

Please use unified (or context) diffs.

> modifies mail-source.el to use a netrc (~/.authinfo) file for IMAP
> fetching.  It will be triggered when mail-source-authinfo-file is
> not nil.  It will override the :user and
> :password parameters set in mail-sources unconditionally.

Is it good to override it?  Wouldn't it be more natural if
`mail-source-authinfo-file' is used if no user/password is given in
`mail-sources'?

> This allows users to store their passwords for an IMAP mail source
> in the authinfo file, encrypted if necessary.  

I think this is useful.

> I use this for my IMAP fetching and it worked fine; please test
> (also if you don't use the feature, make sure your IMAP fetching
> works OK).  I won't commit until I get confirmations it works, since
> mail fetching is a sensitive area for improvements.
>
> I plan to add this to POP and webmail fetching as well, possibly through
> mail-source-bind instead of in each individual fetching function.
> Please let me know what you think.

I'm not sure if I understand what you have in mind.

> Also, I think instead of the current variables
>
> (setq
>  nnimap-authinfo-file "~/.authinfo.enc"
>  nntp-authinfo-file "~/.authinfo.enc"
>  mail-source-authinfo-file "~/.authinfo.enc"
>  smtpmail-auth-credentials "~/.authinfo.enc")
>
> Gnus should have a single specification:

`smtpmail.el', (pop3.el) and `mail-source.el' are not Gnus specific.
So it should be a general Emacs facility.  So please include
emacs-devel in further discussions.  As all ~/.authinfo parsing is
done via `netrc.el', it might make sense to define it therein?

> (setq
>  authinfo-files '(('nnimap "~/.imap-authinfo.enc")
>                   (t "~/.authinfo.enc")))

Maybe a simple value "~/.authinfo" should be equivalent
to '((t "~/.authinfo")).

> Then we can derive each of those variables at runtime, if they are not
> set:
>
> (setq nnimap-authinfo-file
>  (or nnimap-authinfo-file (gnus-get-authinfo-file 'nnimap)))
>
> WDYT?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: mail-source.el patch to use netrc-parse

[Ted Zlatanov]

Reiner Steib wrote:
> Ted Zlatanov wrote:
> > modifies mail-source.el to use a netrc (~/.authinfo) file for IMAP
> > fetching.  It will be triggered when mail-source-authinfo-file is
> > not nil.  It will override the :user and
> > :password parameters set in mail-sources unconditionally.

> Is it good to override it?  Wouldn't it be more natural if
> `mail-source-authinfo-file' is used if no user/password is given in
> `mail-sources'?

I think you're right.

> > This allows users to store their passwords for an IMAP mail source
> > in the authinfo file, encrypted if necessary.  

> I think this is useful.

> > I use this for my IMAP fetching and it worked fine; please test
> > (also if you don't use the feature, make sure your IMAP fetching
> > works OK).  I won't commit until I get confirmations it works, since
> > mail fetching is a sensitive area for improvements.
> >
> > I plan to add this to POP and webmail fetching as well, possibly through
> > mail-source-bind instead of in each individual fetching function.
> > Please let me know what you think.

> I'm not sure if I understand what you have in mind.

Right now it's a special case for IMAP.  Instead, I could set up
mail-source-bind to read the authentication info and bind the
corresponding variables for every mail source available.  But also see
my authinfo proposal, sent separately.

Ted

global authinfo mechanism in Emacs, Gnus, Tramp (was: mail-source.el patch to use netrc-parse)

[Ted Zlatanov]

Reiner Steib wrote:
> On Tue, Feb 05 2008, Ted Zlatanov wrote:
> > Also, I think instead of the current variables
> >
> > (setq
> >  nnimap-authinfo-file "~/.authinfo.enc"
> >  nntp-authinfo-file "~/.authinfo.enc"
> >  mail-source-authinfo-file "~/.authinfo.enc"
> >  smtpmail-auth-credentials "~/.authinfo.enc")
> >
> > Gnus should have a single specification:

> `smtpmail.el', (pop3.el) and `mail-source.el' are not Gnus specific.
> So it should be a general Emacs facility.  So please include
> emacs-devel in further discussions.  As all ~/.authinfo parsing is
> done via `netrc.el', it might make sense to define it therein?

> > (setq
> >  authinfo-files '(('nnimap "~/.imap-authinfo.enc")
> >                   (t "~/.authinfo.enc")))

> Maybe a simple value "~/.authinfo" should be equivalent
> to '((t "~/.authinfo")).

Agreed.  Note also I want to transition netrc.el to use EasyPG instead
of encrypt.el, so changes are coming there anyhow.

> > Then we can derive each of those variables at runtime, if they are not
> > set:
> >
> > (setq nnimap-authinfo-file
> >  (or nnimap-authinfo-file (gnus-get-authinfo-file 'nnimap)))
> >
> > WDYT?

I'd deprecate the special variables, personally, and make
gnus-get-authinfo-file respect them but use the global authinfo
otherwise.  Also, the global authinfo could have non-file sources,
e.g. hard-coded strings or shell commands.  I think that's valuable.
I'm tired of specifying special variables for authentication everywhere
in Emacs and in Gnus.  I see a *lot* of questions about setting those
up in Gnus, as well, so I think we can do better.

With a single specification we could also extend it to specify EasyPG
encryption and other things.  Right now, specifying IMAP vs. POP
vs. SMTP (TLS and regular) authentication information is annoying.
Tramp could also use this mechanism.  I've cc-ed emacs-devel and
tramp-devel.

Logically the structure of the global authinfo would be:

entry:
 protocol-or-purpose: ['pop, 'imap, 'imaps, 'ssh, t]
  hardcoded: (:user "me" :password 'ask :cache 60)
  file: (:name "/a/b/c" :epg-parameters () :other-parameters ())
  command: (:program "/bin/true" :parameters ())

This is an initial idea, so don't get hung up on the format.  I
intentionally didn't specify it in Customize format.

If something like this already exists in Emacs, please let me know.

Ted

Re: global authinfo mechanism in Emacs, Gnus, Tramp

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> I'm tired of specifying special variables for authentication everywhere
Ted> in Emacs and in Gnus.  I see a *lot* of questions about setting those
Ted> up in Gnus, as well, so I think we can do better.

Additionally it would be nice not to have to set the variables by hand
at all... Emacs could remember the passwords in a keyring.

Ted> entry:
Ted>  protocol-or-purpose: ['pop, 'imap, 'imaps, 'ssh, t]
Ted>   hardcoded: (:user "me" :password 'ask :cache 60)
Ted>   file: (:name "/a/b/c" :epg-parameters () :other-parameters ())
Ted>   command: (:program "/bin/true" :parameters ())

Gnome's keyring manager uses (more or less) an application-specified
alist and returns the best match.  That way different users of the
keyring can store the information they need there.

Tom

Re: global authinfo mechanism in Emacs, Gnus, Tramp

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> Can Emacs ask the Gnome keyring manager over some protocol for a
Ted> user's authentication alist?

Emacs could either use the keyring API directly, or it could use a
helper program.  When I was toying with keyring stuff I went the
latter route.  I can send 'ekeyring.c' (and 'keyring.el' for that
matter) if you want it.

FWIW the hard part about this project is just going through all the
existing elisp and wiring it up to the keyring.

Tom

Re: global authinfo mechanism in Emacs, Gnus, Tramp

[Ted Zlatanov]

On Thu, 28 Feb 2008 09:12:04 -0700 Tom Tromey <tromey@redhat.com> wrote: 

>>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> entry:
Ted> protocol-or-purpose: ['pop, 'imap, 'imaps, 'ssh, t]
Ted> hardcoded: (:user "me" :password 'ask :cache 60)
Ted> file: (:name "/a/b/c" :epg-parameters () :other-parameters ())
Ted> command: (:program "/bin/true" :parameters ())

Tom> Gnome's keyring manager uses (more or less) an application-specified
Tom> alist and returns the best match.  That way different users of the
Tom> keyring can store the information they need there.

Can Emacs ask the Gnome keyring manager over some protocol for a user's
authentication alist?  That would be great for those who use Gnome.
Sorry, I'm not one of them so I don't know the protocols and data
formats available.

Ted

Re: global authinfo mechanism in Emacs, Gnus, Tramp

[Ted Zlatanov]

I set up a first draft of the auth-source.el package in Gnus.  I'll do a
bit more customization before it's even close to usable, but if anyone
wants you can at least see how customize will look.  The goal is
simplicity, so I intentionally am not specifying ports and
SSL/TLS/etc. options.

Ted

Re: mail-source.el patch to use netrc-parse

[Ted Zlatanov]

On Thu, 28 Feb 2008 09:38:21 -0600 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> Right now it's a special case for IMAP.  Instead, I could set up
TZ> mail-source-bind to read the authentication info and bind the
TZ> corresponding variables for every mail source available.  But also see
TZ> my authinfo proposal, sent separately.

I've now set up mail-source-bind to always check auth-sources (through
auth-source.el) for IMAP, POP, and everything else mail-source.el
supports.  The user name and password will override any others that have
been set up; if this is incorrect I can rearrange the priority but I
think it's sensible and will make it easier for users to migrate to an
authinfo-centered setup.

Ted