* pop3.el broken @ 2002-03-13 10:58 Pavel Janík 2002-03-13 15:50 ` Stainless Steel Rat ` (3 more replies) 0 siblings, 4 replies; 28+ messages in thread From: Pavel Janík @ 2002-03-13 10:58 UTC (permalink / raw) Hi, this change 2002-03-12 Katsumi Yamaoka <yamaoka@jpl.org> * pop3.el (pop3-open-server): Set process buffer unibyte. is the cause of \202 appearing in e-mails fetched by POP. Reverting it fixed the problem for me. Please also consider applying something like this: --- message.el.~6.218.~ Wed Mar 13 07:04:53 2002 +++ message.el Wed Mar 13 11:54:35 2002 @@ -3062,8 +3062,8 @@ ;; free for -inject-arguments -- a big win for the user and for us ;; since we don't have to play that double-guessing game and the user ;; gets full control (no gestapo'ish -f's, for instance). --sj - (if (functionp 'message-qmail-inject-args) - (funcall 'message-qmail-inject-args) + (if (message-functionp message-qmail-inject-args) + (funcall message-qmail-inject-args) message-qmail-inject-args))) ;; qmail-inject doesn't say anything on it's stdout/stderr, ;; we have to look at the retval instead -- Pavel Janík Yours is the third report of this - it's definitely a bug in ext3. -- Andrew Morton in linux-kernel ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-13 10:58 pop3.el broken Pavel Janík @ 2002-03-13 15:50 ` Stainless Steel Rat 2002-03-13 16:20 ` Bill White ` (2 subsequent siblings) 3 siblings, 0 replies; 28+ messages in thread From: Stainless Steel Rat @ 2002-03-13 15:50 UTC (permalink / raw) * Pavel@Janik.cz (Pavel Janík) on Wed, 13 Mar 2002 | 2002-03-12 Katsumi Yamaoka <yamaoka@jpl.org> | * pop3.el (pop3-open-server): Set process buffer unibyte. | is the cause of \202 appearing in e-mails fetched by POP. Reverting it | fixed the problem for me. Yes, it is. pop3's process buffer must be multibyte and the coding system must be binary. Nothing else will work correctly on a MULE Emacs or XEmacs. -- Rat <ratinox@peorth.gweep.net> \ Happy Fun Ball may stick to certain types Minion of Nathan - Nathan says Hi! \ of skin. PGP Key: at a key server near you! \ That and five bucks will get you a small coffee at Starbucks. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-13 10:58 pop3.el broken Pavel Janík 2002-03-13 15:50 ` Stainless Steel Rat @ 2002-03-13 16:20 ` Bill White 2002-03-13 16:31 ` Bill White 2002-03-13 17:46 ` Bill White 2002-03-13 19:58 ` Simon Josefsson 3 siblings, 1 reply; 28+ messages in thread From: Bill White @ 2002-03-13 16:20 UTC (permalink / raw) On Wed Mar 13 2002 at 04:58, Pavel@Janik.cz (Pavel Janík) said: > Hi, > > this change > > 2002-03-12 Katsumi Yamaoka <yamaoka@jpl.org> > > * pop3.el (pop3-open-server): Set process buffer unibyte. > > is the cause of \202 appearing in e-mails fetched by POP. Reverting > it fixed the problem for me. Dunno about \202, but I'm suddenly seeing lots of \201s appearing in messages fetched via pop. They are always just before 8-bit characters. Cheers - bw -- Bill White . billw@wolfram.com . http://members.wri.com/billw "No ma'am, we're musicians." ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-13 16:20 ` Bill White @ 2002-03-13 16:31 ` Bill White 0 siblings, 0 replies; 28+ messages in thread From: Bill White @ 2002-03-13 16:31 UTC (permalink / raw) On Wed Mar 13 2002 at 10:20, Bill White <billw@wolfram.com> said: > On Wed Mar 13 2002 at 04:58, Pavel@Janik.cz (Pavel Janík) said: > >> Hi, >> >> this change >> >> 2002-03-12 Katsumi Yamaoka <yamaoka@jpl.org> >> >> * pop3.el (pop3-open-server): Set process buffer unibyte. >> >> is the cause of \202 appearing in e-mails fetched by POP. Reverting >> it fixed the problem for me. > > Dunno about \202, but I'm suddenly seeing lots of \201s appearing in > messages fetched via pop. They are always just before 8-bit > characters. See <url:http://members.wri.com/billw/201.jpg> Cheers - bw -- Bill White . billw@wolfram.com . http://members.wri.com/billw "No ma'am, we're musicians." ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-13 10:58 pop3.el broken Pavel Janík 2002-03-13 15:50 ` Stainless Steel Rat 2002-03-13 16:20 ` Bill White @ 2002-03-13 17:46 ` Bill White 2002-03-13 17:53 ` Paul Jarc 2002-03-13 19:58 ` Simon Josefsson 3 siblings, 1 reply; 28+ messages in thread From: Bill White @ 2002-03-13 17:46 UTC (permalink / raw) On Wed Mar 13 2002 at 04:58, Pavel@Janik.cz (Pavel Janík) said: > Hi, > > this change > > 2002-03-12 Katsumi Yamaoka <yamaoka@jpl.org> > > * pop3.el (pop3-open-server): Set process buffer unibyte. > > is the cause of \202 appearing in e-mails fetched by POP. Reverting > it fixed the problem for me. Reverting fixed the \201 problem for me, too. Will this be fixed soon? If not soon, how do I avoid picking up revision 6.11 of pop3.el when I 'cvs up'? Cheers - bw -- Bill White . billw@wolfram.com . http://members.wri.com/billw "No ma'am, we're musicians." ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-13 17:46 ` Bill White @ 2002-03-13 17:53 ` Paul Jarc 2002-03-13 20:21 ` Karl Kleinpaste 0 siblings, 1 reply; 28+ messages in thread From: Paul Jarc @ 2002-03-13 17:53 UTC (permalink / raw) Bill White <billw@wolfram.com> wrote: > how do I avoid picking up revision 6.11 of pop3.el when I 'cvs up'? "cvs -r 6.10 update lisp/pop3.el". Future invocations of "cvs update" will remember that pop3.el should stay at 6.10. When it's fixed later, do "cvs update -A lisp/pop3.el" to make it forget and get the latest version. paul ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-13 17:53 ` Paul Jarc @ 2002-03-13 20:21 ` Karl Kleinpaste 0 siblings, 0 replies; 28+ messages in thread From: Karl Kleinpaste @ 2002-03-13 20:21 UTC (permalink / raw) prj@po.cwru.edu (Paul Jarc) writes: > "cvs -r 6.10 update lisp/pop3.el" FTR, that should be "cvs update -r 6.10 lisp/pop3.el" because (at least in the latest CVS, as updated by RedHat) order is important, and the former version gives annoying diagnostics. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-13 10:58 pop3.el broken Pavel Janík ` (2 preceding siblings ...) 2002-03-13 17:46 ` Bill White @ 2002-03-13 19:58 ` Simon Josefsson 2002-03-13 22:39 ` Katsumi Yamaoka 3 siblings, 1 reply; 28+ messages in thread From: Simon Josefsson @ 2002-03-13 19:58 UTC (permalink / raw) Cc: Katsumi Yamaoka Pavel@Janik.cz (Pavel Janík) writes: > * pop3.el (pop3-open-server): Set process buffer unibyte. > > is the cause of \202 appearing in e-mails fetched by POP. Reverting it > fixed the problem for me. I reverted it in CVS. Katsumi, was the apop bug you reported the reason for the change? If so, I can't reproduce the following: (md5 "dingnusdingnusdingnusdingnusding") => "e071fa3c89ab48d83f55162fa456881d" (md5 (string-as-multibyte "dingnusdingnusdingnusdingnusding")) => "6486539a971f92213ffd3538f9ad8fea" My emacs seem to return e071f... for both cases, regardless of whether the buffer is multibyte or not. > Please also consider applying something like this: Thanks, done. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-13 19:58 ` Simon Josefsson @ 2002-03-13 22:39 ` Katsumi Yamaoka 2002-03-14 4:52 ` Stainless Steel Rat 0 siblings, 1 reply; 28+ messages in thread From: Katsumi Yamaoka @ 2002-03-13 22:39 UTC (permalink / raw) I'm sorry to all for my carelessness. >>>>> In <iluelioqlyq.fsf@extundo.com> >>>>> Simon Josefsson <jas@extundo.com> wrote: > Pavel@Janik.cz (Pavel Janík) writes: >> * pop3.el (pop3-open-server): Set process buffer unibyte. >> is the cause of \202 appearing in e-mails fetched by POP. Reverting >> it fixed the problem for me. > I reverted it in CVS. > Katsumi, was the apop bug you reported the reason for the change? > If so, I can't reproduce the following: > (md5 "dingnusdingnusdingnusdingnusding") > => "e071fa3c89ab48d83f55162fa456881d" > (md5 (string-as-multibyte "dingnusdingnusdingnusdingnusding")) > => "6486539a971f92213ffd3538f9ad8fea" > My emacs seem to return e071f... for both cases, regardless of > whether the buffer is multibyte or not. I found that the difference is caused by Mule-UCS. I'll consider how to fix the problem. -- Katsumi Yamaoka <yamaoka@jpl.org> ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-13 22:39 ` Katsumi Yamaoka @ 2002-03-14 4:52 ` Stainless Steel Rat 2002-03-14 7:01 ` Katsumi Yamaoka 2002-03-14 9:35 ` Simon Josefsson 0 siblings, 2 replies; 28+ messages in thread From: Stainless Steel Rat @ 2002-03-14 4:52 UTC (permalink / raw) * Katsumi Yamaoka <yamaoka@jpl.org> on Wed, 13 Mar 2002 | I found that the difference is caused by Mule-UCS. I'll | consider how to fix the problem. If you are looking at changing pop3.el because something is broken, stop. If it does not fit in the pop3-movemail function then the problem is somewhere not in pop3.el. I tried making the process buffer unibyte four years ago. It will not work. The MULE environment will force it multibyte anyway (see the recent batch of \201 sightings). The buffer must remain multibyte in a MULE environment, and must have coding-system 'binary. That is the only way it works. The bug you've found is in MULE-UCS, not pop3.el. -- Rat <ratinox@peorth.gweep.net> \ Caution: Happy Fun Ball may suddenly Minion of Nathan - Nathan says Hi! \ accelerate to dangerous speeds. PGP Key: at a key server near you! \ That and five bucks will get you a small coffee at Starbucks. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-14 4:52 ` Stainless Steel Rat @ 2002-03-14 7:01 ` Katsumi Yamaoka 2002-03-14 9:35 ` Simon Josefsson 1 sibling, 0 replies; 28+ messages in thread From: Katsumi Yamaoka @ 2002-03-14 7:01 UTC (permalink / raw) >>>>> In <m3sn73agyk.fsf@peorth.gweep.net> >>>>> Stainless Steel Rat <ratinox@peorth.gweep.net> wrote: > * Katsumi Yamaoka <yamaoka@jpl.org> on Wed, 13 Mar 2002 > | I found that the difference is caused by Mule-UCS. I'll > | consider how to fix the problem. > If you are looking at changing pop3.el because something is broken, stop. Ok, I agree. > If it does not fit in the pop3-movemail function then the problem is > somewhere not in pop3.el. I tried making the process buffer unibyte four > years ago. It will not work. The MULE environment will force it multibyte > anyway (see the recent batch of \201 sightings). > The buffer must remain multibyte in a MULE environment, and must have > coding-system 'binary. That is the only way it works. > The bug you've found is in MULE-UCS, not pop3.el. I'm afraid so it is. I've sent a question to the mule-ja list this morning (in Japan). Now I'm using the following advice for the temporary repairs. (defadvice md5 (before treat-7bit-chars-as-unibyte activate) "Treat 7bit chars as unibyte." (if (string-match "^[\000-\177]+" (ad-get-arg 0)) (ad-set-arg 0 (string-as-unibyte (ad-get-arg 0))))) -- Katsumi Yamaoka <yamaoka@jpl.org> ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-14 4:52 ` Stainless Steel Rat 2002-03-14 7:01 ` Katsumi Yamaoka @ 2002-03-14 9:35 ` Simon Josefsson 2002-03-14 22:11 ` Stainless Steel Rat 1 sibling, 1 reply; 28+ messages in thread From: Simon Josefsson @ 2002-03-14 9:35 UTC (permalink / raw) Cc: (ding) On 13 Mar 2002, Stainless Steel Rat wrote: > If it does not fit in the pop3-movemail function then the problem is > somewhere not in pop3.el. I tried making the process buffer unibyte four > years ago. It will not work. The MULE environment will force it multibyte > anyway (see the recent batch of \201 sightings). > > The buffer must remain multibyte in a MULE environment, and must have > coding-system 'binary. That is the only way it works. FWIW, imap.el uses unibyte buffers, so it can be made to work. OTOH, I'm somewhat inclined to change this, to reduce the amount of code that has anything to do with Mule. OTTH, don't fix what aint broken. (Nnimap uses string-as-multibyte on the data handed from imap.el to get it in a format acceptable by Gnus.) ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-14 9:35 ` Simon Josefsson @ 2002-03-14 22:11 ` Stainless Steel Rat 2002-03-15 13:18 ` Katsumi Yamaoka 0 siblings, 1 reply; 28+ messages in thread From: Stainless Steel Rat @ 2002-03-14 22:11 UTC (permalink / raw) * Simon Josefsson <jas@extundo.com> on Thu, 14 Mar 2002 | FWIW, imap.el uses unibyte buffers, so it can be made to work. The difference is that pop3.el uses only one buffer for the entire session. Everything happens in the process buffer. MULE becomes a liability when operating on data streams like that. -- Rat <ratinox@peorth.gweep.net> \ Ingredients of Happy Fun Ball include an Minion of Nathan - Nathan says Hi! \ unknown glowing substance which fell to PGP Key: at a key server near you! \ Earth, presumably from outer space. That and five bucks will get you a small coffee at Starbucks. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-14 22:11 ` Stainless Steel Rat @ 2002-03-15 13:18 ` Katsumi Yamaoka 2002-03-15 18:03 ` Simon Josefsson 2002-03-16 3:57 ` Stainless Steel Rat 0 siblings, 2 replies; 28+ messages in thread From: Katsumi Yamaoka @ 2002-03-15 13:18 UTC (permalink / raw) Cc: mule-ja [-- Attachment #1: Type: text/plain, Size: 1790 bytes --] The problem that the return value of `md5' for a multibyte string and a unibyte string differ under Emacs 21 is caused by Mule-UCS. However, it is not a bug of Mule-UCS. What we should do is to improve pop3.el after all. I will attach a new patch. The built-in function `md5' will encode a given string if it is a multibyte string. The coding-system for encoding is decided by the following way: (symbol-value (car coding-category-list)) You can find those procedures in fns.c. Mule-UCS will put the coding category `coding-category-utf-16-le' in the forefront of `coding-category-list' by default. The default value of `coding-category-utf-16-le' is `utf-16-le'. And a string extracted from a multibyte buffer will be a multibyte string even if it does not contain non-ascii characters. Thus the code (md5 (concat pop3-timestamp pass)) and (md5 (encode-coding-string (concat pop3-timestamp pass) 'utf-16-le)) are equivalent and a string will be altered to 16-bit characters as follows: (encode-coding-string "dingnusdingnus" 'utf-16-le) => "\377\376d^@i^@n^@g^@n^@u^@s^@d^@i^@n^@g^@n^@u^@s^@" Even if Mule-UCS is not used, a string as the first argument of `md5' is always encoded by some coding-system, but almost all coding-systems will not alter an ascii text. Exceptionally, it is a very rare case but it is possible that if a person has set the value of `coding-category-iso-7' with the coding-system `iso-2022-jp-1978-irv' (which is known as "old JIS") and push `coding-category-iso-7' into `coding-category-list', she or he would not be able to use the apop authentication with pop3.el. In my opinion that it should never be her or his fault. 2002-03-15 Katsumi Yamaoka <yamaoka@jpl.org> * pop3.el (pop3-string-as-unibyte): New macro. (pop3-apop): Use it. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: pop3.el.diff --] [-- Type: text/x-patch, Size: 1006 bytes --] --- pop3.el~ 2002-03-13 21:56:17.780672000 +0000 +++ pop3.el 2002-03-15 12:58:08.964942000 +0000 @@ -307,6 +307,13 @@ (if (not (and response (string-match "+OK" response))) (pop3-quit process)))) +(eval-when-compile + (defmacro pop3-string-as-unibyte (string) + "Return a unibyte string with the same individual bytes as STRING." + (if (fboundp 'string-as-unibyte) + (list 'string-as-unibyte string) + string))) + (defun pop3-apop (process user) "Send alternate authentication information to the server." (let ((pass pop3-password)) @@ -314,7 +321,8 @@ (setq pass (pop3-read-passwd (format "Password for %s: " pop3-maildrop)))) (if pass - (let ((hash (pop3-md5 (concat pop3-timestamp pass)))) + (let ((hash (pop3-md5 (pop3-string-as-unibyte + (concat pop3-timestamp pass))))) (pop3-send-command process (format "APOP %s %s" user hash)) (let ((response (pop3-read-response process t))) (if (not (and response (string-match "+OK" response))) ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-15 13:18 ` Katsumi Yamaoka @ 2002-03-15 18:03 ` Simon Josefsson 2002-03-16 4:03 ` Stainless Steel Rat 2002-03-16 3:57 ` Stainless Steel Rat 1 sibling, 1 reply; 28+ messages in thread From: Simon Josefsson @ 2002-03-15 18:03 UTC (permalink / raw) Cc: ding, mule-ja On Fri, 15 Mar 2002, Katsumi Yamaoka wrote: > The problem that the return value of `md5' for a multibyte string > and a unibyte string differ under Emacs 21 is caused by Mule-UCS. > However, it is not a bug of Mule-UCS. What we should do is to > improve pop3.el after all. I will attach a new patch. This patch looks OK. Out of curiosity, is it sufficient to, instead, pass 'binary as CODING-SYSTEM to `md5'? It seems as if the problem really is `md5' guessing incorrectly about a good coding system to use, using hints provided by Mule-UCS. Converting the string into unibyte before calling md5, as your patch does, is one solution, but maybe the string can be passed directly to `md5' if you provide it with a better coding system to encode the string as. > Even if Mule-UCS is not used, a string as the first argument of > `md5' is always encoded by some coding-system, but almost all > coding-systems will not alter an ascii text. Exceptionally, it > is a very rare case but it is possible that if a person has set > the value of `coding-category-iso-7' with the coding-system > `iso-2022-jp-1978-irv' (which is known as "old JIS") and push > `coding-category-iso-7' into `coding-category-list', she or he > would not be able to use the apop authentication with pop3.el. > In my opinion that it should never be her or his fault. The problem really is that POP3/APOP does not define in what encoding the password should be encoded into before inputting it to the hash function. It is left as a customization things, which means that only ASCII will interoperate unless you configure your client to what the server uses. The best would be if the protocol said "the password should be encoded into UTF-8" before hashed. Hm. Perhaps pop3.el should always encode the password as UTF-8 instead of using string-as-unibyte. At least it seems more configurable, if you make the 'utf-8 choice customizable. But I haven't tried it, so maybe passing 'utf-8 to `md5' does not work. (I remember that I really did not understand what I was doing when I wrote `md5'.) ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-15 18:03 ` Simon Josefsson @ 2002-03-16 4:03 ` Stainless Steel Rat 2002-03-16 9:59 ` Simon Josefsson 0 siblings, 1 reply; 28+ messages in thread From: Stainless Steel Rat @ 2002-03-16 4:03 UTC (permalink / raw) * Simon Josefsson <jas@extundo.com> on Fri, 15 Mar 2002 | The problem really is that POP3/APOP does not define in what encoding the | password should be encoded into before inputting it to the hash function. | It is left as a customization things, which means that only ASCII will | interoperate unless you configure your client to what the server uses. Actually, er... as I recall, POP3 passwords are not 8-bit clean (they are also Unix passwords). Simple ASCII is all that is allowed. Same goes for APOP and I think KPOP as well. You are doing something very, very wrong if you try to use 8-bit characters in passwords like that. -- Rat <ratinox@peorth.gweep.net> \ Do not use Happy Fun Ball on concrete. Minion of Nathan - Nathan says Hi! \ PGP Key: at a key server near you! \ That and five bucks will get you a small coffee at Starbucks. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-16 4:03 ` Stainless Steel Rat @ 2002-03-16 9:59 ` Simon Josefsson 2002-03-16 10:24 ` Simon Josefsson 0 siblings, 1 reply; 28+ messages in thread From: Simon Josefsson @ 2002-03-16 9:59 UTC (permalink / raw) Cc: (ding) Stainless Steel Rat <ratinox@peorth.gweep.net> writes: > * Simon Josefsson <jas@extundo.com> on Fri, 15 Mar 2002 > | The problem really is that POP3/APOP does not define in what encoding the > | password should be encoded into before inputting it to the hash function. > | It is left as a customization things, which means that only ASCII will > | interoperate unless you configure your client to what the server uses. > > Actually, er... as I recall, POP3 passwords are not 8-bit clean (they are > also Unix passwords). Simple ASCII is all that is allowed. Then passing 'ascii as the CODING-SYSTEM for `md5' seem like the right thing. If we want to extend the spec a little with the most likely politically correct coding system this week, we could change it into 'utf-8 (if the Emacs supports CODING-SYSTEM and the utf-8 coding system), which is compatible with ascii for ASCII-only characters, so it shouldn't break correct usage. The coding system to use should be customizable, of course. > Same goes for APOP and I think KPOP as well. You are doing > something very, very wrong if you try to use 8-bit characters in > passwords like that. APOP/KPOP doesn't transfer the password, so perhaps this area is underspecified. KPOP uses Kerberos, which "supports" non-ASCII characters, but only if the client and server uses the same charsets. AFAICR, they are moving towards UTF-8. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-16 9:59 ` Simon Josefsson @ 2002-03-16 10:24 ` Simon Josefsson 2002-03-16 15:19 ` Stainless Steel Rat 0 siblings, 1 reply; 28+ messages in thread From: Simon Josefsson @ 2002-03-16 10:24 UTC (permalink / raw) Cc: (ding) Simon Josefsson <jas@extundo.com> writes: >> * Simon Josefsson <jas@extundo.com> on Fri, 15 Mar 2002 >> | The problem really is that POP3/APOP does not define in what encoding the >> | password should be encoded into before inputting it to the hash function. >> | It is left as a customization things, which means that only ASCII will >> | interoperate unless you configure your client to what the server uses. >> >> Actually, er... as I recall, POP3 passwords are not 8-bit clean (they are >> also Unix passwords). Simple ASCII is all that is allowed. > > Then passing 'ascii as the CODING-SYSTEM for `md5' seem like the right > thing. ...except that there isn't an ascii CODING-SYSTEM. However, using raw-text gives the expected results under the emacsen I could test with (xemacs 21.4 nomule, xemacs 21.5 mule, emacs 21.1 native md5 and elisp md5 and external md5sum). ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-16 10:24 ` Simon Josefsson @ 2002-03-16 15:19 ` Stainless Steel Rat 2002-03-18 13:03 ` Katsumi Yamaoka 0 siblings, 1 reply; 28+ messages in thread From: Stainless Steel Rat @ 2002-03-16 15:19 UTC (permalink / raw) * Simon Josefsson <jas@extundo.com> on Sat, 16 Mar 2002 | ...except that there isn't an ascii CODING-SYSTEM. However, using | raw-text gives the expected results under the emacsen I could test | with (xemacs 21.4 nomule, xemacs 21.5 mule, emacs 21.1 native md5 and | elisp md5 and external md5sum). I believe raw-text (and raw?) do not exist in XEmacs 20. What happens if you use binary (which I know works for everything). -- Rat <ratinox@peorth.gweep.net> \ Do not taunt Happy Fun Ball. Minion of Nathan - Nathan says Hi! \ PGP Key: at a key server near you! \ That and five bucks will get you a small coffee at Starbucks. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-16 15:19 ` Stainless Steel Rat @ 2002-03-18 13:03 ` Katsumi Yamaoka 2002-03-18 15:53 ` Stainless Steel Rat 2002-03-18 17:30 ` Simon Josefsson 0 siblings, 2 replies; 28+ messages in thread From: Katsumi Yamaoka @ 2002-03-18 13:03 UTC (permalink / raw) Let's assume that the following premises are correct. 1. A POP3 server will use only ascii characters to generate a timestamp string in its banner greeting, which is described in RFC1460. There is no problem that pop3.el uses the coding-system binary and a multibyte buffer to transfer such a string from a server. 2. A password string may be either an ascii-only string or a string which includes non-ascii characters. The latter should be encoded with a reasonable coding-system before passing it to md5bin (which is mentioned below). I don't know how to set a non-ascii password using the popauth command, and I think the use of a non-ascii password is not popular, though. 3. Both a timestamp string and a password string do not contain newlines. 4. There is the imaginary function md5bin which will not encode a given string. It can be provided with one of the built-in md5, the lisp function md5 and the external process md5 which is called with the coding-system binary. Do you have any doubts? If none, pop3.el can be modified something like follows. It can be simplified if we don't have to handle to encode a password string. (defvar pop3-apop-passwd-coding-system nil "*Coding system used to encode a non-ascii password for APOP.") (defvar pop3-md5-binary nil) (defmacro pop3-make-apop-auth (password) "Return a string for the apop authentication corresponding to the value of `pop3-timestamp' and PASSWORD." (if (and nil;(fboundp 'md5) (subrp (symbol-function 'md5))) (if (and (featurep 'xemacs) (< (function-max-args 'md5) 4)) (if (featurep 'mule) ;; XEmacs 20 with MULE `(let ((pass ,password)) (md5 (concat pop3-timestamp (if pop3-apop-passwd-coding-system (encode-coding-string pass pop3-apop-passwd-coding-system) pass)))) ;; XEmacs 20 without MULE `(md5 (concat pop3-timestamp ,password))) (if (featurep 'mule) ;; Emacs 21 or XEmacs 21 with MULE `(let ((pass ,password)) (md5 (concat pop3-timestamp (if pop3-apop-passwd-coding-system (encode-coding-string pass pop3-apop-passwd-coding-system) pass)) nil nil 'binary)) ;; XEmacs 21 without MULE `(md5 (concat pop3-timestamp ,password) nil nil 'binary))) (let ((fn '(if (not pop3-md5-binary) (setq pop3-md5-binary (if (condition-case nil (require 'md5) (error nil)) 'md5 (lambda (string) (with-temp-buffer (insert string) (let ((coding-system-for-write 'binary)) (call-process-region (point-min) (point-max) pop3-md5-program t (current-buffer)) (buffer-substring 1 33))))))))) (if (featurep 'mule) `(let ((pass ,password)) ,fn (funcall pop3-md5-binary (concat pop3-timestamp (if pop3-apop-passwd-coding-system (encode-coding-string pass pop3-apop-passwd-coding-system) pass)))) `(progn ,fn (funcall pop3-md5-binary (concat pop3-timestamp ,password))))))) (defun pop3-apop (process user) [...] ;;(let ((hash (pop3-md5 (concat pop3-timestamp pass)))) (let ((hash (pop3-make-apop-auth pass))) -- Katsumi Yamaoka <yamaoka@jpl.org> ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-18 13:03 ` Katsumi Yamaoka @ 2002-03-18 15:53 ` Stainless Steel Rat 2002-03-18 17:12 ` Simon Josefsson 2002-03-18 17:30 ` Simon Josefsson 1 sibling, 1 reply; 28+ messages in thread From: Stainless Steel Rat @ 2002-03-18 15:53 UTC (permalink / raw) * Katsumi Yamaoka <yamaoka@jpl.org> on Mon, 18 Mar 2002 | The latter should be encoded with a reasonable coding-system before | passing it to md5bin (which is mentioned below). A pass phrase is raw data, not text. It should not be treated as something a human can read. It should -never- be smashed through any coding system. Something you should be aware of before applying any patches: pop3.el is maintained directly by the FSF since they forked it in 1999 and never bothered to tell me about it. It is part of Emacs, not Gnus. All changes need to go through the FSF; but your XEmacs code will most probably be stripped. XEmacs has a vastly different fork that incorporates much of Franklin Lee's work but will never be merged into the FSF's code tree. Wrapping APOP auth with reasonable advice may be the sanest thing to do. It certainly requires the least ammount of new code. -- Rat <ratinox@peorth.gweep.net> \ Do not taunt Happy Fun Ball. Minion of Nathan - Nathan says Hi! \ PGP Key: at a key server near you! \ That and five bucks will get you a small coffee at Starbucks. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-18 15:53 ` Stainless Steel Rat @ 2002-03-18 17:12 ` Simon Josefsson 2002-03-18 18:04 ` Stainless Steel Rat 0 siblings, 1 reply; 28+ messages in thread From: Simon Josefsson @ 2002-03-18 17:12 UTC (permalink / raw) Cc: (ding) Stainless Steel Rat <ratinox@peorth.gweep.net> writes: > | The latter should be encoded with a reasonable coding-system before > | passing it to md5bin (which is mentioned below). > > A pass phrase is raw data, not text. It should not be treated as something > a human can read. It should -never- be smashed through any coding system. Huh? A password is a text string, entered on the keyboard. As such, it must be encoded in some character set in order to be input to a hash function. pop3 has implicitely encoded it as ASCII traditionally, assuming that (md5 "foobar") encodes the characters "foobar" as ASCII characters before applying md5 on it. This is no longer the case, (md5 "foobar") can end up encoding the characters as UTF-16 before running md5 if you have configured Emacs to use that encoding (by, say, enabling Mule-UCS). For a less Mule confused example, compare EBCDIC vs ASCII. If the pop3 server used a ASCII encoding of the password, the client must also use ASCII. > Something you should be aware of before applying any patches: pop3.el is > maintained directly by the FSF since they forked it in 1999 and never > bothered to tell me about it. If you send a patch to the Emacs maintainers that modifies it to what you want [as long as it works] I would hope that they will unfork it. > It is part of Emacs, not Gnus. All changes need to go through the > FSF; but your XEmacs code will most probably be stripped. XEmacs > has a vastly different fork that incorporates much of Franklin Lee's > work but will never be merged into the FSF's code tree. Right, I wish that XEmacs pop3.el could be reverted to your version (and that you would handle the bug reports ;)), and that people wanting enhanced behaviour could use epop3 or another package containing the current XEmacs pop3.el. It is confusing to fork code and not change the name. > Wrapping APOP auth with reasonable advice may be the sanest thing to do. > It certainly requires the least ammount of new code. Defadvice isn't a good long term solution. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-18 17:12 ` Simon Josefsson @ 2002-03-18 18:04 ` Stainless Steel Rat 0 siblings, 0 replies; 28+ messages in thread From: Stainless Steel Rat @ 2002-03-18 18:04 UTC (permalink / raw) * Simon Josefsson <jas@extundo.com> on Mon, 18 Mar 2002 | Huh? A password is a text string, entered on the keyboard. As such, it | must be encoded in some character set in order to be input to a hash | function. No, it isn't. A pass phrase could be an 8-bit string, not necessarilly humanly readable text, and not necessarilly entered on a keyboard. That is an extreme situation, but if 8-bit pass phrases are to be accepted then this extreme must be taken into consideration. Such a string must never be encoded as anything. Or, to MULEtilate it, the coding system is binary since no-conversion doesn't mean no-conversion some of the time. | pop3 has implicitely encoded it as ASCII traditionally, assuming that | (md5 "foobar") encodes the characters "foobar" as ASCII characters before | applying md5 on it. pop3.el has never made any such assumption. MULE may make it seem to appear that it does, but I have long maintained that MULE is broken in this regard. Emacs Lisp should work identically in any Emacs without having to work around the environment's brain-damaged assumptions. [...] | If you send a patch to the Emacs maintainers that modifies it to what | you want [as long as it works] I would hope that they will unfork it. I did, and the FSF did not. By the way, you cannot use the FSF's version with a VAX/VMS POP server. I fixed that bug in 1998, but the FSF has ignored the fix after repeated submissions. I gave up in disgust. [...] | Right, I wish that XEmacs pop3.el could be reverted to your version | (and that you would handle the bug reports ;)), and that people | wanting enhanced behaviour could use epop3 or another package | containing the current XEmacs pop3.el. It is confusing to fork code | and not change the name. Too little, too late. I asked them repeatedly to change the name. They refused, or simply ignored the request. I gave up in disgust. [...] | Defadvice isn't a good long term solution. Yes, well, given that the real problem, MULE, will never be fixed, advice is the only solution that has a reasonable chance of working everywhere. There are three hopelessly forked versions of pop3.el out there. Creating a fourth is not going to help. -- Rat <ratinox@peorth.gweep.net> \ Happy Fun Ball contains a liquid core, Minion of Nathan - Nathan says Hi! \ which, if exposed due to rupture, should PGP Key: at a key server near you! \ not be touched, inhaled, or looked at. That and five bucks will get you a small coffee at Starbucks. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-18 13:03 ` Katsumi Yamaoka 2002-03-18 15:53 ` Stainless Steel Rat @ 2002-03-18 17:30 ` Simon Josefsson 2002-03-18 23:43 ` Katsumi Yamaoka 1 sibling, 1 reply; 28+ messages in thread From: Simon Josefsson @ 2002-03-18 17:30 UTC (permalink / raw) Cc: ding Katsumi Yamaoka <yamaoka@jpl.org> writes: > Do you have any doubts? Good analysis. > If none, pop3.el can be modified something like follows. It can be > simplified if we don't have to handle to encode a password string. > > > (defvar pop3-apop-passwd-coding-system nil > "*Coding system used to encode a non-ascii password for APOP.") Can't we make it default to UTF-8, if available? (if (or (and (fboundp 'find-coding-system) (find-coding-system 'utf-8)) (and (fboundp 'coding-system-p) (coding-system-p 'utf-8))) 'utf-8) > (defmacro pop3-make-apop-auth (password) I can't say that I tried all possible cases (or even any of them), but this looks like the right approach. Perhaps we can remove the pop3-md5-program stuff and use md5.el as shipped with Gnus, but this isn't important. I think "md5sum" is more common than "md5" (at least my redhat machine doesn't have md5). ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-18 17:30 ` Simon Josefsson @ 2002-03-18 23:43 ` Katsumi Yamaoka 2002-03-19 1:13 ` Stainless Steel Rat 2002-03-19 9:54 ` Simon Josefsson 0 siblings, 2 replies; 28+ messages in thread From: Katsumi Yamaoka @ 2002-03-18 23:43 UTC (permalink / raw) >>>>> In <iluadt5kckw.fsf@extundo.com> >>>>> Simon Josefsson <jas@extundo.com> wrote: >> (defvar pop3-apop-passwd-coding-system nil >> "*Coding system used to encode a non-ascii password for APOP.") > Can't we make it default to UTF-8, if available? Yes, of course. >> (defmacro pop3-make-apop-auth (password) > I can't say that I tried all possible cases (or even any of them), > but this looks like the right approach. I have a lot of versions of Emacsen including XEmacs without MULE and I've already tested that code totally. :-) > Perhaps we can remove the pop3-md5-program stuff and use md5.el as > shipped with Gnus, but this isn't important. I think "md5sum" is > more common than "md5" (at least my redhat machine doesn't have md5). Ok. By the way, to improve the existent pop3.el is not really matter of my own. I can make an overall revision of pop3.el in the right manner if I feel inclined to do that. Maybe it will be named pop.el.:-p Otherwise, what I should do might be the use of IMAP. I simply understood that pop3.el is deadlocked. -- Katsumi Yamaoka <yamaoka@jpl.org> ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-18 23:43 ` Katsumi Yamaoka @ 2002-03-19 1:13 ` Stainless Steel Rat 2002-03-19 9:54 ` Simon Josefsson 1 sibling, 0 replies; 28+ messages in thread From: Stainless Steel Rat @ 2002-03-19 1:13 UTC (permalink / raw) * Katsumi Yamaoka <yamaoka@jpl.org> on Mon, 18 Mar 2002 | By the way, to improve the existent pop3.el is not really matter | of my own. I can make an overall revision of pop3.el in the | right manner if I feel inclined to do that. Maybe it will be | named pop.el.:-p Otherwise, what I should do might be the use | of IMAP. I simply understood that pop3.el is deadlocked. All things considered, it may be time for someone to rewrite the library from scratch for modern Emacsen. -- Rat <ratinox@peorth.gweep.net> \ When not in use, Happy Fun Ball should be Minion of Nathan - Nathan says Hi! \ returned to its special container and PGP Key: at a key server near you! \ kept under refrigeration. That and five bucks will get you a small coffee at Starbucks. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-18 23:43 ` Katsumi Yamaoka 2002-03-19 1:13 ` Stainless Steel Rat @ 2002-03-19 9:54 ` Simon Josefsson 1 sibling, 0 replies; 28+ messages in thread From: Simon Josefsson @ 2002-03-19 9:54 UTC (permalink / raw) Cc: ding On Tue, 19 Mar 2002, Katsumi Yamaoka wrote: > By the way, to improve the existent pop3.el is not really matter > of my own. I can make an overall revision of pop3.el in the > right manner if I feel inclined to do that. Maybe it will be > named pop.el.:-p Otherwise, what I should do might be the use > of IMAP. I simply understood that pop3.el is deadlocked. Yes, doing a rewrite of pop3.el and calling it pop.el would be a good idea, I think. Ideally it should incorporate all features from epop3 and XEmacs pop3.el. Hey, while we are at it, there could even be a manual. And active maintainers. Or something. ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: pop3.el broken 2002-03-15 13:18 ` Katsumi Yamaoka 2002-03-15 18:03 ` Simon Josefsson @ 2002-03-16 3:57 ` Stainless Steel Rat 1 sibling, 0 replies; 28+ messages in thread From: Stainless Steel Rat @ 2002-03-16 3:57 UTC (permalink / raw) * Katsumi Yamaoka <yamaoka@jpl.org> on Fri, 15 Mar 2002 | The built-in function `md5' will encode a given string if it is a | multibyte string. The coding-system for encoding is decided by | the following way: Bad assumption. md5 is not always a built-in function. [...] | (md5 (encode-coding-string (concat pop3-timestamp pass) 'utf-16-le)) This will break with any XEmacs built without MULE. This is exactly why I say not to make changes in the guts of pop3.el. -- Rat <ratinox@peorth.gweep.net> \ Do not taunt Happy Fun Ball. Minion of Nathan - Nathan says Hi! \ PGP Key: at a key server near you! \ That and five bucks will get you a small coffee at Starbucks. ^ permalink raw reply [flat|nested] 28+ messages in thread
end of thread, other threads:[~2002-03-19 9:54 UTC | newest] Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2002-03-13 10:58 pop3.el broken Pavel Janík 2002-03-13 15:50 ` Stainless Steel Rat 2002-03-13 16:20 ` Bill White 2002-03-13 16:31 ` Bill White 2002-03-13 17:46 ` Bill White 2002-03-13 17:53 ` Paul Jarc 2002-03-13 20:21 ` Karl Kleinpaste 2002-03-13 19:58 ` Simon Josefsson 2002-03-13 22:39 ` Katsumi Yamaoka 2002-03-14 4:52 ` Stainless Steel Rat 2002-03-14 7:01 ` Katsumi Yamaoka 2002-03-14 9:35 ` Simon Josefsson 2002-03-14 22:11 ` Stainless Steel Rat 2002-03-15 13:18 ` Katsumi Yamaoka 2002-03-15 18:03 ` Simon Josefsson 2002-03-16 4:03 ` Stainless Steel Rat 2002-03-16 9:59 ` Simon Josefsson 2002-03-16 10:24 ` Simon Josefsson 2002-03-16 15:19 ` Stainless Steel Rat 2002-03-18 13:03 ` Katsumi Yamaoka 2002-03-18 15:53 ` Stainless Steel Rat 2002-03-18 17:12 ` Simon Josefsson 2002-03-18 18:04 ` Stainless Steel Rat 2002-03-18 17:30 ` Simon Josefsson 2002-03-18 23:43 ` Katsumi Yamaoka 2002-03-19 1:13 ` Stainless Steel Rat 2002-03-19 9:54 ` Simon Josefsson 2002-03-16 3:57 ` Stainless Steel Rat
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).