From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/67835 Path: news.gmane.org!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.gnus.general Subject: Re: bug#1174: 23.0.60; Some UTF-8 mails displaying wrongly in Emacs 23 Date: Mon, 01 Dec 2008 16:04:01 -0500 Message-ID: References: <871vyf6q40.fsf@marauder.physik.uni-ulm.de> <87k5ampwej.fsf@marauder.physik.uni-ulm.de> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1228165577 13842 80.91.229.12 (1 Dec 2008 21:06:17 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 1 Dec 2008 21:06:17 +0000 (UTC) Cc: Frank Schmitt , ding@gnus.org, 1174@emacsbugs.donarmstrong.com To: Simon Josefsson Original-X-From: ding-owner+M16282@lists.math.uh.edu Mon Dec 01 22:07:22 2008 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.50) id 1L7Fz0-0007tw-TP for ding-account@gmane.org; Mon, 01 Dec 2008 22:07:07 +0100 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1L7FwY-0002xU-52; Mon, 01 Dec 2008 15:04:34 -0600 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1L7FwW-0002xI-Bv for ding@lists.math.uh.edu; Mon, 01 Dec 2008 15:04:32 -0600 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtp (Exim 4.69) (envelope-from ) id 1L7FwS-0000PR-2W for ding@lists.math.uh.edu; Mon, 01 Dec 2008 15:04:32 -0600 Original-Received: from pruche.dit.umontreal.ca ([132.204.246.22]) by quimby.gnus.org with esmtp (Exim 3.36 #1 (Debian)) id 1L7Fwc-0004sF-00 for ; Mon, 01 Dec 2008 22:04:38 +0100 Original-Received: from alfajor.home (vpn-132-204-232-120.acd.umontreal.ca [132.204.232.120]) by pruche.dit.umontreal.ca (8.14.1/8.14.1) with ESMTP id mB1L4cxf018284; Mon, 1 Dec 2008 16:04:38 -0500 Original-Received: by alfajor.home (Postfix, from userid 20848) id 0A4831C1EF; Mon, 1 Dec 2008 16:04:01 -0500 (EST) In-Reply-To: <87k5ampwej.fsf@marauder.physik.uni-ulm.de> (Reiner Steib's message of "Sat, 29 Nov 2008 13:08:20 +0100") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (gnu/linux) X-NAI-Spam-Score: 0 X-NAI-Spam-Rules: 1 Rules triggered RV3159=0 X-Spam-Score: -2.6 (--) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:67835 Archived-At: > Simon, could you please clarify why you reverted Stefan's change in > `nnimap-demule'? It breaks reading UTF-8 articles via nnimap.el in > Emacs 23. Having looked at the code again, I'm more than ever confident that string-to-unibyte is the right thing to use. Maybe the code I installed back then failed to fallback to string-as-unibyte when string-to-unibyte was not available, which caused a bug for Simon? In any case the newly committed code has a prenthesis typo that makes it still use the old code and ignore the new config var nnimap-demule-use-string-to-multibyte. Also I recommend to just use the patch below instead. The first hunk removes an unnecessary use of nnimap-demule since the output will be inserted into a unibyte buffer. Stefan --- nnimap.el.~1.50.~ 2008-12-01 15:38:55.000000000 -0500 +++ nnimap.el 2008-12-01 15:49:53.000000000 -0500 @@ -608,12 +608,11 @@ (with-current-buffer nnimap-server-buffer (setq uid imap-current-message mbx imap-current-mailbox - headers (nnimap-demule - (if (imap-capability 'IMAP4rev1) + headers (if (imap-capability 'IMAP4rev1) ;; xxx don't just use car? alist doesn't contain ;; anything else now, but it might... (nth 2 (car (imap-message-get uid 'BODYDETAIL))) - (imap-message-get uid 'RFC822.HEADER))) + (imap-message-get uid 'RFC822.HEADER)) lines (imap-body-lines (imap-message-body imap-current-message)) chars (imap-message-get imap-current-message 'RFC822.SIZE))) (nnheader-insert-nov @@ -901,40 +900,17 @@ (when (nnimap-possibly-change-server server) (nnoo-status-message 'nnimap server))) -(defvar nnimap-demule-use-string-to-multibyte (fboundp 'string-to-multibyte) - "Temporary internal debug variable. -If you have problems (UTF-8 not decoded correctly on IMAP) with -the default value, please report it as a bug!") -;; FIXME: Clarify if we need to make this variable conditional on the Emacs -;; version (Emacs 22 vs. Emacs 23;Emacs 21 doesn't have `string-to-multibyte' -;; anyhow). --rsteib -;; -;; http://thread.gmane.org/gmane.emacs.gnus.general/67112 -;; (bug#464, reported by James Cloos) -;; http://thread.gmane.org/gmane.emacs.bugs/21524 -;; (bug#1174, reported by Frank Schmitt) - -(defun nnimap-demule (string) - ;; BEWARE: we used to use string-as-multibyte here which is braindead - ;; because it will turn accidental emacs-mule-valid byte sequences - ;; into multibyte chars. --Stef - ;; Reverted, braindead got 7.5 out of 10 on imdb, so it can't be - ;; that bad. --Simon - (gnus-message 9 "nnimap-demule-use-string-to-multibyte: %s" - nnimap-demule-use-string-to-multibyte) - (if nnimap-demule-use-string-to-multibyte - ;; Stefan - (funcall (if (and (fboundp 'string-to-multibyte) - (subrp (symbol-function 'string-to-multibyte))) - 'string-to-multibyte - 'identity) - (or string ""))) - ;; Simon - (funcall (if (and (fboundp 'string-as-multibyte) - (subrp (symbol-function 'string-as-multibyte))) - 'string-as-multibyte - 'identity) - (or string ""))) +;; We used to use a string-as-multibyte here, but it is really incorrect. +;; This function is used when we're about to insert a unibyte string +;; into a potentially multibyte buffer. The string is either an article +;; header or body (or both?), undecoded. When Emacs is asked to convert +;; a unibyte string to multibyte, it may either use the equivalent of +;; nothing (e.g. non-Mule XEmacs), string-make-unibyte (i.e. decode using +;; locale), string-as-multibyte (decode using emacs-internal coding system) +;; or string-to-multibyte (keep the data undecoded as a sequence of bytes). +;; Only the last one preserves the data such that we can reliably later on +;; decode the text using the mime info. +(defalias 'nnimap-demule 'mm-string-to-multibyte) (defun nnimap-make-callback (article gnus-callback buffer) "Return a callback function."