pop3.el broken

pop3.el broken

[Pavel Janík]

Hi,

this change

2002-03-12  Katsumi Yamaoka  <yamaoka@jpl.org>

	* pop3.el (pop3-open-server): Set process buffer unibyte.

is the cause of \202 appearing in e-mails fetched by POP. Reverting it
fixed the problem for me.

Please also consider applying something like this:

--- message.el.~6.218.~	Wed Mar 13 07:04:53 2002
+++ message.el	Wed Mar 13 11:54:35 2002
@@ -3062,8 +3062,8 @@
 	 ;; free for -inject-arguments -- a big win for the user and for us
 	 ;; since we don't have to play that double-guessing game and the user
 	 ;; gets full control (no gestapo'ish -f's, for instance).  --sj
-         (if (functionp 'message-qmail-inject-args)
-             (funcall 'message-qmail-inject-args)
+         (if (message-functionp message-qmail-inject-args)
+             (funcall message-qmail-inject-args)
            message-qmail-inject-args)))
     ;; qmail-inject doesn't say anything on it's stdout/stderr,
     ;; we have to look at the retval instead
-- 
Pavel Janík

Yours is the third report of this - it's definitely a bug in ext3.
                  -- Andrew Morton in linux-kernel

Re: pop3.el broken

[Stainless Steel Rat]

* Pavel@Janik.cz (Pavel Janík)  on Wed, 13 Mar 2002
| 2002-03-12  Katsumi Yamaoka  <yamaoka@jpl.org>

| 	* pop3.el (pop3-open-server): Set process buffer unibyte.

| is the cause of \202 appearing in e-mails fetched by POP. Reverting it
| fixed the problem for me.

Yes, it is.  pop3's process buffer must be multibyte and the coding system
must be binary.  Nothing else will work correctly on a MULE Emacs or XEmacs.
-- 
Rat <ratinox@peorth.gweep.net>    \ Happy Fun Ball may stick to certain types
Minion of Nathan - Nathan says Hi! \ of skin.
PGP Key: at a key server near you!  \ 
       That and five bucks will get you a small coffee at Starbucks.

Re: pop3.el broken

[Bill White]

On Wed Mar 13 2002 at 04:58, Pavel@Janik.cz (Pavel Janík) said:

> Hi,
>
> this change
>
> 2002-03-12  Katsumi Yamaoka  <yamaoka@jpl.org>
>
> 	* pop3.el (pop3-open-server): Set process buffer unibyte.
>
> is the cause of \202 appearing in e-mails fetched by POP. Reverting
> it fixed the problem for me.

Dunno about \202, but I'm suddenly seeing lots of \201s appearing in
messages fetched via pop.  They are always just before 8-bit
characters.

Cheers -

bw
-- 
Bill White . billw@wolfram.com . http://members.wri.com/billw
"No ma'am, we're musicians."

Re: pop3.el broken

[Bill White]

On Wed Mar 13 2002 at 10:20, Bill White <billw@wolfram.com> said:

> On Wed Mar 13 2002 at 04:58, Pavel@Janik.cz (Pavel Janík) said:
>
>> Hi,
>>
>> this change
>>
>> 2002-03-12  Katsumi Yamaoka  <yamaoka@jpl.org>
>>
>> 	* pop3.el (pop3-open-server): Set process buffer unibyte.
>>
>> is the cause of \202 appearing in e-mails fetched by POP. Reverting
>> it fixed the problem for me.
>
> Dunno about \202, but I'm suddenly seeing lots of \201s appearing in
> messages fetched via pop.  They are always just before 8-bit
> characters.

See <url:http://members.wri.com/billw/201.jpg>

Cheers -

bw
-- 
Bill White . billw@wolfram.com . http://members.wri.com/billw
"No ma'am, we're musicians."

Re: pop3.el broken

[Bill White]

On Wed Mar 13 2002 at 04:58, Pavel@Janik.cz (Pavel Janík) said:

> Hi,
>
> this change
>
> 2002-03-12  Katsumi Yamaoka  <yamaoka@jpl.org>
>
> 	* pop3.el (pop3-open-server): Set process buffer unibyte.
>
> is the cause of \202 appearing in e-mails fetched by POP. Reverting
> it fixed the problem for me.

Reverting fixed the \201 problem for me, too.  Will this be fixed
soon?  If not soon, how do I avoid picking up revision 6.11 of pop3.el
when I 'cvs up'?

Cheers -

bw
-- 
Bill White . billw@wolfram.com . http://members.wri.com/billw
"No ma'am, we're musicians."

Re: pop3.el broken

[Paul Jarc]

Bill White <billw@wolfram.com> wrote:
> how do I avoid picking up revision 6.11 of pop3.el when I 'cvs up'?

"cvs -r 6.10 update lisp/pop3.el".  Future invocations of "cvs update"
will remember that pop3.el should stay at 6.10.  When it's fixed
later, do "cvs update -A lisp/pop3.el" to make it forget and get the
latest version.


paul

Re: pop3.el broken

[Karl Kleinpaste]

prj@po.cwru.edu (Paul Jarc) writes:
> "cvs -r 6.10 update lisp/pop3.el"

FTR, that should be
"cvs update -r 6.10 lisp/pop3.el"
because (at least in the latest CVS, as updated by RedHat) order is
important, and the former version gives annoying diagnostics.

Re: pop3.el broken

[Simon Josefsson]

Pavel@Janik.cz (Pavel Janík) writes:

> 	* pop3.el (pop3-open-server): Set process buffer unibyte.
>
> is the cause of \202 appearing in e-mails fetched by POP. Reverting it
> fixed the problem for me.

I reverted it in CVS.

Katsumi, was the apop bug you reported the reason for the change?  If
so, I can't reproduce the following:

(md5 "dingnusdingnusdingnusdingnusding")
 => "e071fa3c89ab48d83f55162fa456881d"

(md5 (string-as-multibyte "dingnusdingnusdingnusdingnusding"))
 => "6486539a971f92213ffd3538f9ad8fea"

My emacs seem to return e071f... for both cases, regardless of whether
the buffer is multibyte or not.  

> Please also consider applying something like this:

Thanks, done.

Re: pop3.el broken

[Katsumi Yamaoka]

I'm sorry to all for my carelessness.

>>>>> In <iluelioqlyq.fsf@extundo.com> 
>>>>>	Simon Josefsson <jas@extundo.com> wrote:
> Pavel@Janik.cz (Pavel Janík) writes:

>> 	* pop3.el (pop3-open-server): Set process buffer unibyte.

>> is the cause of \202 appearing in e-mails fetched by POP. Reverting
>> it fixed the problem for me.

> I reverted it in CVS.

> Katsumi, was the apop bug you reported the reason for the change?
> If so, I can't reproduce the following:

> (md5 "dingnusdingnusdingnusdingnusding")
>  => "e071fa3c89ab48d83f55162fa456881d"

> (md5 (string-as-multibyte "dingnusdingnusdingnusdingnusding"))
>  => "6486539a971f92213ffd3538f9ad8fea"

> My emacs seem to return e071f... for both cases, regardless of
> whether the buffer is multibyte or not.

I found that the difference is caused by Mule-UCS.  I'll
consider how to fix the problem.
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: pop3.el broken

[Stainless Steel Rat]

* Katsumi Yamaoka <yamaoka@jpl.org>  on Wed, 13 Mar 2002
| I found that the difference is caused by Mule-UCS.  I'll
| consider how to fix the problem.

If you are looking at changing pop3.el because something is broken, stop.
If it does not fit in the pop3-movemail function then the problem is
somewhere not in pop3.el.  I tried making the process buffer unibyte four
years ago.  It will not work.  The MULE environment will force it multibyte
anyway (see the recent batch of \201 sightings).

The buffer must remain multibyte in a MULE environment, and must have
coding-system 'binary.  That is the only way it works.

The bug you've found is in MULE-UCS, not pop3.el.
-- 
Rat <ratinox@peorth.gweep.net>    \ Caution: Happy Fun Ball may suddenly
Minion of Nathan - Nathan says Hi! \ accelerate to dangerous speeds.
PGP Key: at a key server near you!  \ 
       That and five bucks will get you a small coffee at Starbucks.

Re: pop3.el broken

[Katsumi Yamaoka]

>>>>> In <m3sn73agyk.fsf@peorth.gweep.net> 
>>>>>	Stainless Steel Rat <ratinox@peorth.gweep.net> wrote:

> * Katsumi Yamaoka <yamaoka@jpl.org>  on Wed, 13 Mar 2002
> | I found that the difference is caused by Mule-UCS.  I'll
> | consider how to fix the problem.

> If you are looking at changing pop3.el because something is broken, stop.

Ok, I agree.

> If it does not fit in the pop3-movemail function then the problem is
> somewhere not in pop3.el.  I tried making the process buffer unibyte four
> years ago.  It will not work.  The MULE environment will force it multibyte
> anyway (see the recent batch of \201 sightings).

> The buffer must remain multibyte in a MULE environment, and must have
> coding-system 'binary.  That is the only way it works.

> The bug you've found is in MULE-UCS, not pop3.el.

I'm afraid so it is.  I've sent a question to the mule-ja list
this morning (in Japan).  Now I'm using the following advice for
the temporary repairs.

(defadvice md5 (before treat-7bit-chars-as-unibyte activate)
  "Treat 7bit chars as unibyte."
  (if (string-match "^[\000-\177]+" (ad-get-arg 0))
      (ad-set-arg 0 (string-as-unibyte (ad-get-arg 0)))))
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: pop3.el broken

[Simon Josefsson]

On 13 Mar 2002, Stainless Steel Rat wrote:

> If it does not fit in the pop3-movemail function then the problem is
> somewhere not in pop3.el.  I tried making the process buffer unibyte four
> years ago.  It will not work.  The MULE environment will force it multibyte
> anyway (see the recent batch of \201 sightings).
> 
> The buffer must remain multibyte in a MULE environment, and must have
> coding-system 'binary.  That is the only way it works.

FWIW, imap.el uses unibyte buffers, so it can be made to work.  OTOH, I'm
somewhat inclined to change this, to reduce the amount of code that has
anything to do with Mule.  OTTH, don't fix what aint broken.

(Nnimap uses string-as-multibyte on the data handed from imap.el to get it 
in a format acceptable by Gnus.)

Re: pop3.el broken

[Stainless Steel Rat]

* Simon Josefsson <jas@extundo.com>  on Thu, 14 Mar 2002
| FWIW, imap.el uses unibyte buffers, so it can be made to work.

The difference is that pop3.el uses only one buffer for the entire session.
Everything happens in the process buffer.  MULE becomes a liability when
operating on data streams like that.
-- 
Rat <ratinox@peorth.gweep.net>    \ Ingredients of Happy Fun Ball include an
Minion of Nathan - Nathan says Hi! \ unknown glowing substance which fell to
PGP Key: at a key server near you!  \ Earth, presumably from outer space.
       That and five bucks will get you a small coffee at Starbucks.

Re: pop3.el broken

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 1790 bytes --]

The problem that the return value of `md5' for a multibyte string
and a unibyte string differ under Emacs 21 is caused by Mule-UCS.
However, it is not a bug of Mule-UCS.  What we should do is to
improve pop3.el after all.  I will attach a new patch.

The built-in function `md5' will encode a given string if it is a
multibyte string.  The coding-system for encoding is decided by
the following way:

(symbol-value (car coding-category-list))

You can find those procedures in fns.c.

Mule-UCS will put the coding category `coding-category-utf-16-le'
in the forefront of `coding-category-list' by default.  The
default value of `coding-category-utf-16-le' is `utf-16-le'.  And
a string extracted from a multibyte buffer will be a multibyte
string even if it does not contain non-ascii characters.  Thus
the code

(md5 (concat pop3-timestamp pass))

and

(md5 (encode-coding-string (concat pop3-timestamp pass) 'utf-16-le))

are equivalent and a string will be altered to 16-bit characters
as follows:

(encode-coding-string "dingnusdingnus" 'utf-16-le)
 => "\377\376d^@i^@n^@g^@n^@u^@s^@d^@i^@n^@g^@n^@u^@s^@"

Even if Mule-UCS is not used, a string as the first argument of
`md5' is always encoded by some coding-system, but almost all
coding-systems will not alter an ascii text.  Exceptionally, it
is a very rare case but it is possible that if a person has set
the value of `coding-category-iso-7' with the coding-system
`iso-2022-jp-1978-irv' (which is known as "old JIS") and push
`coding-category-iso-7' into `coding-category-list', she or he
would not be able to use the apop authentication with pop3.el.
In my opinion that it should never be her or his fault.


2002-03-15  Katsumi Yamaoka  <yamaoka@jpl.org>

	* pop3.el (pop3-string-as-unibyte): New macro.
	(pop3-apop): Use it.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: pop3.el.diff --]
[-- Type: text/x-patch, Size: 1006 bytes --]

--- pop3.el~	2002-03-13 21:56:17.780672000 +0000
+++ pop3.el	2002-03-15 12:58:08.964942000 +0000
@@ -307,6 +307,13 @@
     (if (not (and response (string-match "+OK" response)))
 	(pop3-quit process))))
 
+(eval-when-compile
+  (defmacro pop3-string-as-unibyte (string)
+    "Return a unibyte string with the same individual bytes as STRING."
+    (if (fboundp 'string-as-unibyte)
+	(list 'string-as-unibyte string)
+      string)))
+
 (defun pop3-apop (process user)
   "Send alternate authentication information to the server."
   (let ((pass pop3-password))
@@ -314,7 +321,8 @@
 	(setq pass
 	      (pop3-read-passwd (format "Password for %s: " pop3-maildrop))))
     (if pass
-	(let ((hash (pop3-md5 (concat pop3-timestamp pass))))
+	(let ((hash (pop3-md5 (pop3-string-as-unibyte
+			       (concat pop3-timestamp pass)))))
 	  (pop3-send-command process (format "APOP %s %s" user hash))
 	  (let ((response (pop3-read-response process t)))
 	    (if (not (and response (string-match "+OK" response)))

Re: pop3.el broken

[Simon Josefsson]

On Fri, 15 Mar 2002, Katsumi Yamaoka wrote:

> The problem that the return value of `md5' for a multibyte string
> and a unibyte string differ under Emacs 21 is caused by Mule-UCS.
> However, it is not a bug of Mule-UCS.  What we should do is to
> improve pop3.el after all.  I will attach a new patch.

This patch looks OK.  Out of curiosity, is it sufficient to, instead, pass
'binary as CODING-SYSTEM to `md5'?  It seems as if the problem really is 
`md5' guessing incorrectly about a good coding system to use, using hints 
provided by Mule-UCS.  Converting the string into unibyte before calling 
md5, as your patch does, is one solution, but maybe the string can be 
passed directly to `md5' if you provide it with a better coding system to 
encode the string as.

> Even if Mule-UCS is not used, a string as the first argument of
> `md5' is always encoded by some coding-system, but almost all
> coding-systems will not alter an ascii text.  Exceptionally, it
> is a very rare case but it is possible that if a person has set
> the value of `coding-category-iso-7' with the coding-system
> `iso-2022-jp-1978-irv' (which is known as "old JIS") and push
> `coding-category-iso-7' into `coding-category-list', she or he
> would not be able to use the apop authentication with pop3.el.
> In my opinion that it should never be her or his fault.

The problem really is that POP3/APOP does not define in what encoding the
password should be encoded into before inputting it to the hash function.  
It is left as a customization things, which means that only ASCII will
interoperate unless you configure your client to what the server uses.

The best would be if the protocol said "the password should be encoded
into UTF-8" before hashed.  Hm.  Perhaps pop3.el should always encode the 
password as UTF-8 instead of using string-as-unibyte.  At least it seems 
more configurable, if you make the 'utf-8 choice customizable.  But I 
haven't tried it, so maybe passing 'utf-8 to `md5' does not work.  (I 
remember that I really did not understand what I was doing when I wrote 
`md5'.)

Re: pop3.el broken

[Stainless Steel Rat]

* Simon Josefsson <jas@extundo.com>  on Fri, 15 Mar 2002
| The problem really is that POP3/APOP does not define in what encoding the
| password should be encoded into before inputting it to the hash function.
| It is left as a customization things, which means that only ASCII will
| interoperate unless you configure your client to what the server uses.

Actually, er... as I recall, POP3 passwords are not 8-bit clean (they are
also Unix passwords).  Simple ASCII is all that is allowed.  Same goes for
APOP and I think KPOP as well.  You are doing something very, very wrong if
you try to use 8-bit characters in passwords like that.
-- 
Rat <ratinox@peorth.gweep.net>    \ Do not use Happy Fun Ball on concrete.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 
       That and five bucks will get you a small coffee at Starbucks.

Re: pop3.el broken

[Simon Josefsson]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> * Simon Josefsson <jas@extundo.com>  on Fri, 15 Mar 2002
> | The problem really is that POP3/APOP does not define in what encoding the
> | password should be encoded into before inputting it to the hash function.
> | It is left as a customization things, which means that only ASCII will
> | interoperate unless you configure your client to what the server uses.
>
> Actually, er... as I recall, POP3 passwords are not 8-bit clean (they are
> also Unix passwords).  Simple ASCII is all that is allowed.

Then passing 'ascii as the CODING-SYSTEM for `md5' seem like the right
thing.  If we want to extend the spec a little with the most likely
politically correct coding system this week, we could change it into
'utf-8 (if the Emacs supports CODING-SYSTEM and the utf-8 coding
system), which is compatible with ascii for ASCII-only characters, so
it shouldn't break correct usage.  The coding system to use should be
customizable, of course.

> Same goes for APOP and I think KPOP as well.  You are doing
> something very, very wrong if you try to use 8-bit characters in
> passwords like that.

APOP/KPOP doesn't transfer the password, so perhaps this area is
underspecified.  KPOP uses Kerberos, which "supports" non-ASCII
characters, but only if the client and server uses the same charsets.
AFAICR, they are moving towards UTF-8.

Re: pop3.el broken

[Simon Josefsson]

Simon Josefsson <jas@extundo.com> writes:

>> * Simon Josefsson <jas@extundo.com>  on Fri, 15 Mar 2002
>> | The problem really is that POP3/APOP does not define in what encoding the
>> | password should be encoded into before inputting it to the hash function.
>> | It is left as a customization things, which means that only ASCII will
>> | interoperate unless you configure your client to what the server uses.
>>
>> Actually, er... as I recall, POP3 passwords are not 8-bit clean (they are
>> also Unix passwords).  Simple ASCII is all that is allowed.
>
> Then passing 'ascii as the CODING-SYSTEM for `md5' seem like the right
> thing.

...except that there isn't an ascii CODING-SYSTEM.  However, using
raw-text gives the expected results under the emacsen I could test
with (xemacs 21.4 nomule, xemacs 21.5 mule, emacs 21.1 native md5 and
elisp md5 and external md5sum).

Re: pop3.el broken

[Stainless Steel Rat]

* Simon Josefsson <jas@extundo.com>  on Sat, 16 Mar 2002
| ...except that there isn't an ascii CODING-SYSTEM.  However, using
| raw-text gives the expected results under the emacsen I could test
| with (xemacs 21.4 nomule, xemacs 21.5 mule, emacs 21.1 native md5 and
| elisp md5 and external md5sum).

I believe raw-text (and raw?) do not exist in XEmacs 20.  What happens if
you use binary (which I know works for everything).
-- 
Rat <ratinox@peorth.gweep.net>    \ Do not taunt Happy Fun Ball.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 
       That and five bucks will get you a small coffee at Starbucks.

Re: pop3.el broken

[Katsumi Yamaoka]

Let's assume that the following premises are correct.

1. A POP3 server will use only ascii characters to generate a
   timestamp string in its banner greeting, which is described
   in RFC1460.  There is no problem that pop3.el uses the
   coding-system binary and a multibyte buffer to transfer such
   a string from a server.

2. A password string may be either an ascii-only string or a
   string which includes non-ascii characters.  The latter
   should be encoded with a reasonable coding-system before
   passing it to md5bin (which is mentioned below).

   I don't know how to set a non-ascii password using the
   popauth command, and I think the use of a non-ascii password
   is not popular, though.

3. Both a timestamp string and a password string do not contain
   newlines.

4. There is the imaginary function md5bin which will not encode
   a given string.  It can be provided with one of the built-in
   md5, the lisp function md5 and the external process md5 which
   is called with the coding-system binary.

Do you have any doubts?  If none, pop3.el can be modified
something like follows.  It can be simplified if we don't have
to handle to encode a password string.


(defvar pop3-apop-passwd-coding-system nil
  "*Coding system used to encode a non-ascii password for APOP.")

(defvar pop3-md5-binary nil)

(defmacro pop3-make-apop-auth (password)
  "Return a string for the apop authentication corresponding to the value
of `pop3-timestamp' and PASSWORD."
  (if (and nil;(fboundp 'md5)
	   (subrp (symbol-function 'md5)))
      (if (and (featurep 'xemacs)
	       (< (function-max-args 'md5) 4))
	  (if (featurep 'mule)
	      ;; XEmacs 20 with MULE
	      `(let ((pass ,password))
		 (md5 (concat pop3-timestamp
			      (if pop3-apop-passwd-coding-system
				  (encode-coding-string
				   pass pop3-apop-passwd-coding-system)
				pass))))
	    ;; XEmacs 20 without MULE
	    `(md5 (concat pop3-timestamp ,password)))
	(if (featurep 'mule)
	    ;; Emacs 21 or XEmacs 21 with MULE
	    `(let ((pass ,password))
	       (md5 (concat pop3-timestamp
			    (if pop3-apop-passwd-coding-system
				(encode-coding-string
				 pass pop3-apop-passwd-coding-system)
			      pass))
		    nil nil 'binary))
	  ;; XEmacs 21 without MULE
	  `(md5 (concat pop3-timestamp ,password) nil nil
		'binary)))
    (let ((fn '(if (not pop3-md5-binary)
		   (setq pop3-md5-binary
			 (if (condition-case nil
				 (require 'md5)
			       (error nil))
			     'md5
			   (lambda (string)
			     (with-temp-buffer
			       (insert string)
			       (let ((coding-system-for-write 'binary))
				 (call-process-region (point-min) (point-max)
						      pop3-md5-program
						      t (current-buffer))
				 (buffer-substring 1 33)))))))))
      (if (featurep 'mule)
	  `(let ((pass ,password))
	     ,fn
	     (funcall pop3-md5-binary
		      (concat pop3-timestamp
			      (if pop3-apop-passwd-coding-system
				  (encode-coding-string
				   pass pop3-apop-passwd-coding-system)
				pass))))
	`(progn
	   ,fn
	   (funcall pop3-md5-binary (concat pop3-timestamp ,password)))))))

(defun pop3-apop (process user)
[...]
	;;(let ((hash (pop3-md5 (concat pop3-timestamp pass))))
	(let ((hash (pop3-make-apop-auth pass)))
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: pop3.el broken

[Stainless Steel Rat]

* Katsumi Yamaoka <yamaoka@jpl.org>  on Mon, 18 Mar 2002
|    The latter should be encoded with a reasonable coding-system before
|    passing it to md5bin (which is mentioned below).

A pass phrase is raw data, not text.  It should not be treated as something
a human can read.  It should -never- be smashed through any coding system.

Something you should be aware of before applying any patches: pop3.el is
maintained directly by the FSF since they forked it in 1999 and never
bothered to tell me about it.  It is part of Emacs, not Gnus.  All changes
need to go through the FSF; but your XEmacs code will most probably be
stripped.  XEmacs has a vastly different fork that incorporates much of
Franklin Lee's work but will never be merged into the FSF's code tree.

Wrapping APOP auth with reasonable advice may be the sanest thing to do.
It certainly requires the least ammount of new code.
-- 
Rat <ratinox@peorth.gweep.net>    \ Do not taunt Happy Fun Ball.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 
       That and five bucks will get you a small coffee at Starbucks.

Re: pop3.el broken

[Simon Josefsson]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> |    The latter should be encoded with a reasonable coding-system before
> |    passing it to md5bin (which is mentioned below).
>
> A pass phrase is raw data, not text.  It should not be treated as something
> a human can read.  It should -never- be smashed through any coding system.

Huh?  A password is a text string, entered on the keyboard.  As such,
it must be encoded in some character set in order to be input to a
hash function.  pop3 has implicitely encoded it as ASCII
traditionally, assuming that (md5 "foobar") encodes the characters
"foobar" as ASCII characters before applying md5 on it.  This is no
longer the case, (md5 "foobar") can end up encoding the characters as
UTF-16 before running md5 if you have configured Emacs to use that
encoding (by, say, enabling Mule-UCS).

For a less Mule confused example, compare EBCDIC vs ASCII.  If the
pop3 server used a ASCII encoding of the password, the client must
also use ASCII.

> Something you should be aware of before applying any patches: pop3.el is
> maintained directly by the FSF since they forked it in 1999 and never
> bothered to tell me about it.

If you send a patch to the Emacs maintainers that modifies it to what
you want [as long as it works] I would hope that they will unfork it.

> It is part of Emacs, not Gnus.  All changes need to go through the
> FSF; but your XEmacs code will most probably be stripped.  XEmacs
> has a vastly different fork that incorporates much of Franklin Lee's
> work but will never be merged into the FSF's code tree.

Right, I wish that XEmacs pop3.el could be reverted to your version
(and that you would handle the bug reports ;)), and that people
wanting enhanced behaviour could use epop3 or another package
containing the current XEmacs pop3.el.  It is confusing to fork code
and not change the name.

> Wrapping APOP auth with reasonable advice may be the sanest thing to do.
> It certainly requires the least ammount of new code.

Defadvice isn't a good long term solution.

Re: pop3.el broken

[Stainless Steel Rat]

* Simon Josefsson <jas@extundo.com>  on Mon, 18 Mar 2002
| Huh?  A password is a text string, entered on the keyboard.  As such, it
| must be encoded in some character set in order to be input to a hash
| function.

No, it isn't.  A pass phrase could be an 8-bit string, not necessarilly
humanly readable text, and not necessarilly entered on a keyboard.  That is
an extreme situation, but if 8-bit pass phrases are to be accepted then
this extreme must be taken into consideration.  Such a string must never be
encoded as anything.  Or, to MULEtilate it, the coding system is binary
since no-conversion doesn't mean no-conversion some of the time.

| pop3 has implicitely encoded it as ASCII traditionally, assuming that
| (md5 "foobar") encodes the characters "foobar" as ASCII characters before
| applying md5 on it.

pop3.el has never made any such assumption.  MULE may make it seem to
appear that it does, but I have long maintained that MULE is broken in this
regard.  Emacs Lisp should work identically in any Emacs without having to
work around the environment's brain-damaged assumptions.

[...]
| If you send a patch to the Emacs maintainers that modifies it to what
| you want [as long as it works] I would hope that they will unfork it.

I did, and the FSF did not.

By the way, you cannot use the FSF's version with a VAX/VMS POP server.  I
fixed that bug in 1998, but the FSF has ignored the fix after repeated
submissions.

I gave up in disgust.

[...]
| Right, I wish that XEmacs pop3.el could be reverted to your version
| (and that you would handle the bug reports ;)), and that people
| wanting enhanced behaviour could use epop3 or another package
| containing the current XEmacs pop3.el.  It is confusing to fork code
| and not change the name.

Too little, too late.  I asked them repeatedly to change the name.  They
refused, or simply ignored the request.

I gave up in disgust.

[...]
| Defadvice isn't a good long term solution.

Yes, well, given that the real problem, MULE, will never be fixed, advice
is the only solution that has a reasonable chance of working everywhere.
There are three hopelessly forked versions of pop3.el out there.  Creating
a fourth is not going to help.
-- 
Rat <ratinox@peorth.gweep.net>    \ Happy Fun Ball contains a liquid core,
Minion of Nathan - Nathan says Hi! \ which, if exposed due to rupture, should
PGP Key: at a key server near you!  \ not be touched, inhaled, or looked at.
       That and five bucks will get you a small coffee at Starbucks.

Re: pop3.el broken

[Simon Josefsson]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Do you have any doubts?

Good analysis.

> If none, pop3.el can be modified something like follows.  It can be
> simplified if we don't have to handle to encode a password string.
>
>
> (defvar pop3-apop-passwd-coding-system nil
>   "*Coding system used to encode a non-ascii password for APOP.")

Can't we make it default to UTF-8, if available?

  (if (or (and (fboundp 'find-coding-system) (find-coding-system 'utf-8))
	  (and (fboundp 'coding-system-p) (coding-system-p 'utf-8)))
      'utf-8)

> (defmacro pop3-make-apop-auth (password)

I can't say that I tried all possible cases (or even any of them), but
this looks like the right approach.

Perhaps we can remove the pop3-md5-program stuff and use md5.el as
shipped with Gnus, but this isn't important. I think "md5sum" is more
common than "md5" (at least my redhat machine doesn't have md5).

Re: pop3.el broken

[Katsumi Yamaoka]

>>>>> In <iluadt5kckw.fsf@extundo.com>
>>>>>	Simon Josefsson <jas@extundo.com> wrote:

>> (defvar pop3-apop-passwd-coding-system nil
>>   "*Coding system used to encode a non-ascii password for APOP.")

> Can't we make it default to UTF-8, if available?

Yes, of course.

>> (defmacro pop3-make-apop-auth (password)

> I can't say that I tried all possible cases (or even any of them),
> but this looks like the right approach.

I have a lot of versions of Emacsen including XEmacs without
MULE and I've already tested that code totally. :-)

> Perhaps we can remove the pop3-md5-program stuff and use md5.el as
> shipped with Gnus, but this isn't important. I think "md5sum" is
> more common than "md5" (at least my redhat machine doesn't have md5).

Ok.

By the way, to improve the existent pop3.el is not really matter
of my own.  I can make an overall revision of pop3.el in the
right manner if I feel inclined to do that.  Maybe it will be
named pop.el.:-p  Otherwise, what I should do might be the use
of IMAP.  I simply understood that pop3.el is deadlocked.
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: pop3.el broken

[Stainless Steel Rat]

* Katsumi Yamaoka <yamaoka@jpl.org>  on Mon, 18 Mar 2002
| By the way, to improve the existent pop3.el is not really matter
| of my own.  I can make an overall revision of pop3.el in the
| right manner if I feel inclined to do that.  Maybe it will be
| named pop.el.:-p  Otherwise, what I should do might be the use
| of IMAP.  I simply understood that pop3.el is deadlocked.

All things considered, it may be time for someone to rewrite the library
from scratch for modern Emacsen.
-- 
Rat <ratinox@peorth.gweep.net>    \ When not in use, Happy Fun Ball should be
Minion of Nathan - Nathan says Hi! \ returned to its special container and
PGP Key: at a key server near you!  \ kept under refrigeration.
       That and five bucks will get you a small coffee at Starbucks.

Re: pop3.el broken

[Simon Josefsson]

On Tue, 19 Mar 2002, Katsumi Yamaoka wrote:

> By the way, to improve the existent pop3.el is not really matter
> of my own.  I can make an overall revision of pop3.el in the
> right manner if I feel inclined to do that.  Maybe it will be
> named pop.el.:-p  Otherwise, what I should do might be the use
> of IMAP.  I simply understood that pop3.el is deadlocked.

Yes, doing a rewrite of pop3.el and calling it pop.el would be a good
idea, I think.  Ideally it should incorporate all features from epop3 and
XEmacs pop3.el.  Hey, while we are at it, there could even be a manual.  
And active maintainers.  Or something.

Re: pop3.el broken

[Stainless Steel Rat]

* Katsumi Yamaoka <yamaoka@jpl.org>  on Fri, 15 Mar 2002
| The built-in function `md5' will encode a given string if it is a
| multibyte string.  The coding-system for encoding is decided by
| the following way:

Bad assumption.  md5 is not always a built-in function.

[...]
| (md5 (encode-coding-string (concat pop3-timestamp pass) 'utf-16-le))

This will break with any XEmacs built without MULE.

This is exactly why I say not to make changes in the guts of pop3.el.
-- 
Rat <ratinox@peorth.gweep.net>    \ Do not taunt Happy Fun Ball.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 
       That and five bucks will get you a small coffee at Starbucks.