coding problem in article display with gnus 5.13 incl. in emacs cvs

[Patrick Drechsler <patrick@pdrechsler.de>]

Hi,

after updating GNU/Emacs CVS a few days ago I noticed that the article
buffer sometimes does not display umlaut characters correctly anymore.

Symptoms: Either the article buffer displays (A) the character's control
sequence (ie "ä" -> "\344") or (B) it skips the character it can not
display. This did not happen in previous versions of Gnus v5.13 shipped
with Emacs CVS.

To reproduce (A):

Step 1/2: Send yourself a mail with another mail client (for example
Thunderbird) with the following content (do not copy the umlauts from
the example below: Type them in i.e. Thunderbird!):

--8<---------------cut here---------------start------------->8---
here are some german umlauts:

ae: ä
oe: ö
ue: ü
ss: ß

end of test
--8<---------------cut here---------------end--------------->8---

Step 2/2: This message is now in my "Sent" folder. It is displayed
incorrectly in Gnus (Below I replaced the control characters by the
string displayed):

--8<---------------cut here---------------start------------->8---
From: Patrick Drechsler <patrick@XXX.XX>
Subject: test umlauts
To: Patrick Drechsler <patrickXdrechsler@XXX.XXX>
User-Agent: Thunderbird 1.5.0.14ubu (X11/20080306)

here are some german umlauts:

ae: \344
oe: \366
ue: \374
ss: \337

end of test
--8<---------------cut here---------------end--------------->8---

Placing the cursor on "\344" and `M-x describe-char' results in:

,----
|         character: \344 (4194276, #o17777744, #x3fffe4)
| preferred charset: eight-bit (Raw bytes 0-255)
|        code point: 0xE4
|            syntax: w 	which means: word
|       buffer code: #xE4
|         file code: not encodable by coding system undecided-unix
|           display: no font available
| 
| Character code properties are not shown: customize what to show
| 
| There are text properties here:
|   auto-composed        t
`---- 

Here is the output of `M-x describe-char' from the ä character in second
sentence:

,----
|         character: ä (228, #o344, #xe4)
| preferred charset: iso-8859-1 (Latin-1 (ISO/IEC 8859-1))
|        code point: 0xE4
|            syntax: w 	which means: word
|          category: j:Japanese l:Latin
|       buffer code: #xC3 #xA4
|         file code: #xC3 #xA4 (encoded by coding system utf-8-emacs-unix)
|           display: by this font (glyph code)
|      bitstream vera sans mono:pixelsize=13:foundry=bitstream:weight=regular:slant=roman:width=normal (#x6C)
`----

(Don't ask me why the category indicates "j:Japanese")


Concerning (B):

On other occations I noticed that displayed umlauts are deleted or
replaced by a space.

Here is an example:

,----
| List-Id: Linux <linux.listserv.heise.de>
| To: patrickXXX@XXX.XX
| From: Linux <newsletter@listserv.heise.de>
| Date: Mon, 07 Apr 2008 04:02:02 +0200
| Content-Type: multipart/alternative; boundary="-----b007da1ee42"
| Content-Transfer-Encoding: 8bit
| MIME-Version: 1.0
| Subject: heise online News Linux 07.04.2008
| Message-Id: <E1JiiJi-00083q-2g.05@web.heise.de>
| [...] 
|
| *  Nachrichten berblick zum Thema Linux der vergangenen sieben Tage          *
| *
|    Benchmark-Suite f r Linux
| 
|    Phoronix hat eine umfassende Benchmark-Suite f r Linux ver ffentlicht.
| [...] 
`----


Can other people using Gnus (not "No Gnus") with current Gnu Emacs CVS
confirm this behaviour?

Going through the Gnus settings below, the only thing that catches my
eye is "mm-charset-synonym-alist". Searching the Gnus manual for the
terms "charset" results in section 3.20 Charsets. But I have trouble
understanding it.

Can anybody give me a pointer which variable(s) need to be adopted?

Previous versions of Gnus v5.13 included with Emacs CVS displayed the
article buffer correctly.

TIA,

Patrick

,----[ Emacs version and settings: ]
| In GNU Emacs 23.0.60.2 (i686-pc-linux-gnu, GTK+ Version 2.8.20)
|  of 2008-04-04 on golem
| Windowing system distributor `The X.Org Foundation', version 11.0.70000000
| configured using `configure  '--prefix=/home/patrick/prg/stow/emacs-devel' '--with-xft' '--with-x-toolkit=gtk''
| 
| Important settings:
|   value of $LC_ALL: nil
|   value of $LC_COLLATE: nil
|   value of $LC_CTYPE: nil
|   value of $LC_MESSAGES: nil
|   value of $LC_MONETARY: nil
|   value of $LC_NUMERIC: nil
|   value of $LC_TIME: nil
|   value of $LANG: de_DE.UTF-8
|   value of $XMODIFIERS: nil
|   locale-coding-system: utf-8-unix
|   default-enable-multibyte-characters: t
| 
| Major mode: Message
| 
| Minor modes in effect:
|   gnus-message-citation-mode: t
|   mml-mode: t
|   delete-selection-mode: t
|   show-paren-mode: t
|   savehist-mode: t
|   pc-selection-mode: t
|   iswitchb-mode: t
|   display-time-mode: t
|   which-function-mode: t
|   shell-dirtrack-mode: t
|   tooltip-mode: t
|   mouse-wheel-mode: t
|   menu-bar-mode: t
|   file-name-shadow-mode: t
|   global-font-lock-mode: t
|   font-lock-mode: t
|   blink-cursor-mode: t
|   global-auto-composition-mode: t
|   auto-composition-mode: t
|   auto-compression-mode: t
|   column-number-mode: t
|   line-number-mode: t
|   transient-mark-mode: t
|   abbrev-mode: t
`----

,----[ Gnus settings ]
| (setq gnus-default-nntp-server "")
| (setq gnus-select-method
|       '(nntp "news.myserver.foo"))
| (setq gnus-message-archive-group
|       '((if
| 	    (message-news-p)
| 	    "nnimap+mail.mymailserver.bla:INBOX.sent.news" "nnimap+mail.mymailserver.bla:INBOX.Sent")))
| (setq gnus-secondary-select-methods
|       '((nnml "private")
| 	(nntp "news.gmane.org")
| 	(nnimap "mail.mymailserver.bla"
| 		(nnimap-list-pattern
| 		 ("*")))))
| (setq gnus-group-change-level-function 'gnus-topic-change-level)
| (setq gnus-summary-expunge-below -999)
| (setq gnus-summary-mode-hook
|       '(gnus-agent-mode))
| (setq gnus-exit-group-hook
|       '(gnus-uu-delete-work-dir gnus-uu-clean-up))
| (setq gnus-group-line-format "%M%S%p%P%5y:%B%(%G%)%l %O\n")
| (setq gnus-group-catchup-group-hook
|       '(gnus-topic-update-topic))
| (setq gnus-exit-gnus-hook
|       '(mm-destroy-postponed-undisplay-list))
| (setq gnus-visible-headers "^From:\\|^Newsgroups:\\|^Subject:\\|^Date:\\|^Followup-To:\\|^Reply-To:\\|^Organization:\\|^Summary:\\|^Keywords:\\|^To:\\|^[BGF]?Cc:\\|^Posted-To:\\|^Mail-Copies-To:\\|^Mail-Followup-To:\\|^Apparently-To:\\|^Gnus-Warning:\\|^Resent-From:\\|^X-Sent:\\|^X-Newsreader:\\|^X-Mailer:\\|^User-Agent:\\|^X-User-Agent:\\|^X-Injected-Via-Gmane:\\|^Message-ID:")
| (setq gnus-saved-headers "^From:\\|^Newsgroups:\\|^Subject:\\|^Date:\\|^Followup-To:\\|^Reply-To:\\|^Organization:\\|^Summary:\\|^Keywords:\\|^To:\\|^[BGF]?Cc:\\|^Posted-To:\\|^Mail-Copies-To:\\|^Mail-Followup-To:\\|^Apparently-To:\\|^Gnus-Warning:\\|^Resent-From:\\|^X-Sent:")
| (setq gnus-article-prepare-hook
|       '(bbdb/gnus-pop-up-bbdb-buffer))
| (setq gnus-init-file "~/.emacs.d/.gnus")
| (setq gnus-startup-hook
|       '(bbdb-insinuate-gnus))
| (setq gnus-setup-news-hook
|       '(gnus-agent-queue-setup gnus-fixup-nnimap-unread-after-getting-new-news))
| (setq gnus-save-newsrc-hook
|       '(bbdb-offer-save))
| (setq gnus-group-mode-hook
|       '(gnus-agent-mode gnus-topic-mode))
| (setq gnus-posting-styles
|       '((".*"
| 	 (organization "none")
| 	 (Face
| 	  '(gnus-convert-png-to-face "~/.emacs.d/faces/ant.png")))
| 	(".*gmane.comp.lang.r.general"
| 	 (signature-file "~/.emacs.d/mailsignature/sig-workcam-r")
| 	 (Face
| 	  '(gnus-convert-png-to-face "~/.emacs.d/faces/ant.png")))))
| (setq gnus-home-score-file "~/.emacs.d/total-score-file.SCORE")
| (setq gnus-server-mode-hook
|       '(gnus-agent-mode))
| (setq mm-charset-synonym-alist
|       '((ibm866 . cp866)
| 	(unicode . utf-16-le)
| 	(ks_c_5601-1987 . cp949)
| 	(windows-31j . cp932)
| 	(iso8859-1 . iso-8859-1)))
| (setq message-send-mail-function 'smtpmail-send-it)
| (setq message-post-method
|       '(nntp "news"))
| (setq message-setup-hook
|       '(bbdb-define-all-aliases bbdb-insinuate-message))
| (setq message-mode-hook
|       '(#[nil "\302\030\303	!)\207"
| 	      [gnus-article-copy gnus-setup-message-group nil gnus-configure-posting-styles]
| 	      2]
| 	#[nil "\302 \211\020\211\021\207"
| 	      [message-mailer message-newsreader gnus-extended-version]
| 	      2]))
| (setq message-header-setup-hook
|       '(gnus-inews-insert-archive-gcc gnus-inews-insert-gcc))
| (setq message-signature 'randomsig-signature)
`----


-- 
Linus: I'll throw the ball, see? 
       Then you go bounding after it and bring it back!
Snoopy: Maybe we should think about this a little more.