Gnus, BBDB and MULE-UCS don't like one another

Gnus, BBDB and MULE-UCS don't like one another

[Roland Mas]

Hello people,

This mail is addressed to the following people:

- Manoj Srivastava (Debian Gnus maintainer);
- Takuo Kitame (Debian BBDB and MULE-UCS maintainer);
- Miyashita Hisashi, author of MULE-UCS;
- the Gnus development mailing-list;
- the BBDB development mailing-list;
- the Debian bug tracking system report for the bug #88289

Please excuse the multiple addressees, but I think they all are
concerned (or part of them are, in case of the mailing-lists).
Justification is at the end.

  There is this story about a French guy (that would be me) who tries
to learn Japanese, or at least not to forget it too fast now that he's
out of school.  The guy is a big Gnus fan, the guy loves Unicode (and
needs it anyway), the guy uses BBDB a lot.  And the guy uses Debian.
So the guy installs the Debian packages for Gnus, BBDB and MULE-UCS
(along with GNU Emacs 20.7).  He reads and posts in
fr.lettres.langue.japonaise, a French-speaking newsgroup about the
Japanese language.  French needs some non-ASCII characters, so does
Japanese, UTF-8 is the encoding of choice, and all is well.  Except...

  Except one day all ceases to be well: the UTF-8 messages are not
decoded anymore.  Which means the "small E with acute" character
displays as "capital A with tilde" + "copyright sign" (which is its
UTF-8 encoding), and the Japanese characters are completely
unreadable.  That is a Bad Situation, and we obviously want to go back
toe the Good One.

  To make a long story short, I spent a few nights tracking the
problem, and here's how it happens.  Know that it happens when
upgrading from Gnus 5.8.7 to 5.8.8, whatever (recent) versions of BBDB
and MULE-UCS are installed.  

  Let's say some user wants to start Gnus.  emacs -f gnus is how I do
it, but M-x gnus in an already running Emacs works the same.

- Emacs starts and reads the startup.el file.  On Debian, that startup
  file loads all *.elc? files in /etc/emacs20/, in their order of
  appearance.
- Emacs loads 00debian-vars.elc (some variables and function
  definitions).
- Then 20gnus-init.el (append the right dir to the load-path).
- Then 50bbdb-init.el (require message, then set some variables).
- Then 50mule-ucs-init.el (define the Unicode character sets and
  encodings). 
- Then Gnus starts.

  That's the outline (I'm not mentioning the non-relevant files).

  Then the user enters some group, and tries to read an UTF-8 encoded
message.  Some checks are done, amongst which one that checks that the
charset is known.  For optimisation puropses (I guess), the list of
available charsets is cached in a variable, mm-coding-system-list.
That variable is only accessed through the mm-get-coding-system-list
function, that does all of the caching: the first time it is called,
it copies the value of coding-system-list into mm-coding-system-list,
and the following times it just returns the cached value.

  Now what happens?  Well, it appears that under Gnus 5.8.7 the
variable is nil at that moment, which means it gets computed right now
and yields the correct list, whereas under Gnus 5.8.8 the value is
already present, so the cached value is used.  Unfortunately, in this
last case, the cached value is wrong, in that it doesn't contain any
reference to the encodings defined by MULE-UCS.  Aha, and why does it
not?  Because the value was calculated too early, before MULE-UCS has
initialised.  Here comes the real story:

- Emacs loads 00debian-vars.elc.  No problem here.
- Emacs loads 20gnus-init.el.  Still okay.
- Emacs load 50bbdb-init.el.  Note that the MULE-UCS init step has not
  happened yet.
- The 50bbdb-init.el has a (require 'message) clause, so message.el is
  loaded. 
- message.el requires 'nnheader.
- nnheader.el requires 'mm-util, which defines the functions
  mm-get-coding-system-list and mm-coding-system-p, and
  the mm-charset-synonym-alist variable.
- only *afterwards* is MULE-UCS initialised.

  It so happens that the mm-util.el in Gnus 5.8.7 defines the
mm-charset-synonym-alist variable with a static content, whereas the
one in Gnus 5.8.8 uses a backquote form that calls the
mm-coding-system-p predicate, which in turn calls the
mm-get-coding-system-list function.  Which means, Gnus 5.8.8
initialises that mm-coding-system cache earlier than Gnus 5.8.7.  Only
it happens because BBDB triggers it indirectly.  And unfortunately, B
is before M in the latin alphabet, so the 50bbdb-init.el is lesser
than 50mule-ucs-init.el.  The final result is what looks like a bug.

  Hairy, eh?

  So, now come the reasons I sent this email to so many people.

- Manoj Srivastava: I nagged you about this bug, it's not your fault,
  you can now sleep in peace.

- Takuo Kitame: since you happen to maintain both the BBDB package and
  the MULE-UCS package for Debian, I'd suggest changing the init file
  names: simply renaming 50mule-ucs-init.el to 40mule-ucs-init.el
  fixes the problem here, and that's much less of a hack than the
  other solutions I have come with to fix that particular problem in
  Gnus.  It basically does the right thing, since it defines the
  Unicode coding systems before doing something that uses the coding
  systems list.

- Miyashita Hisashi and the BBDB folks: I think that maybe you could
  include a warning somewhere in the docs of your softwares, so that
  people using them would be aware of the implications of using BBDB
  and MULE-UCS.  They just have to make sure MULE-UCS is initialised
  before BBDB is.

- the Gnus development mailing-list: this might be useful to you and
  to some users...  If they complain that Gnus broke UTF-8 reading you
  now could ask them if they use BBDB and MULE-UCS, and in what order
  these are initialized.

- the Debian bug tracking system report for the bug #88289: well, that
  one seems to be either closable or reassignable to bdbb or (more
  probably) to mule-ucs.

  That's about it.  I hope this will help people, although of course I
hope for them they haven't spent too much time and energy into looking
for that bug.

  If I may coin in a side note, I highly recommend the reading of
"Dirk Gently's Holistic Detective Agency", by Douglas Adams.  To quote
the back cover: "[...] Dirk Gently, self-styled private investigator,
sets out to prove the fundamental interconnectedness of all things by
solving a mysterious murder, [...] unravelling a mysterious mystery
and eating a lot of pizza."  Although I personnally prefer sushi,
that's rather close to the way I spent these last few nights.  There's
also one by Isaac Asimov where a succession of quite harmless acts end
up as a crime, but I don't know the English title.

  Have a nice day, and send me your comments.

Roland.
-- 
Roland Mas

'And what would humans be without love?'
RARE, said Death.  -- in Sourcery (Terry Pratchett)

Re: Gnus, BBDB and MULE-UCS don't like one another

[Dm. Paduchih]

>>>>> "RM" == Roland Mas <lolando@debian.org> writes:

[...]

RM>   That's about it.  I hope this will help people, although of course I
RM> hope for them they haven't spent too much time and energy into looking
RM> for that bug.

[...]

Thank You, Roland.  Your message is very helpful for me.  I had a
trouble with charset 'windows-1251' which i define in my ~/.emacs as

(codepage-setup 1251)
(define-coding-system-alias 'windows-1251 'cp1251)

It is that the Gnus documentation suggests.

I have added
(setq mm-coding-system-list nil)
in ~/.emacs and the trouble has gone away!

Best Regards,
Dmitry