MULE primer

MULE primer

[Lars Magne Ingebrigtsen]

What is *the* way to learn about all this MULE stuff?  I have a
distinct feeling I must be misunderstanding something basic here.  I
mean, here's (日本語) some Japanese text, so I must be doing something 
right, but I still don't understand it.

And the XEmacs and Emacs MULEs are subtly and not-so-subtly
different, in that the XEmacs stuff works and the Emacs stuff
doesn't.  (Well.)

The Emacs 20.3 manual seems kinda slight on this subject...  Is there
a new Emacs Lisp manual out?  Should I read the XEmacs Lisp manual?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: MULE primer

[Alan Shutko]

>>>>> "L" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

L> The Emacs 20.3 manual seems kinda slight on this subject...  Is
L> there a new Emacs Lisp manual out?  Should I read the XEmacs Lisp
L> manual?

There was one in pretesting, but I didn't see it on the ftp site last
I looked.  Email RMS about it?

-- 
Alan Shutko <ats@acm.org> - By consent of the corrupted
You will have a long and unpleasant discussion with your supervisor.

Re: MULE primer

[Hrvoje Niksic]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> The Emacs 20.3 manual seems kinda slight on this subject...  Is
> there a new Emacs Lisp manual out?  Should I read the XEmacs Lisp
> manual?

Mule is completely undocumented in XEmacs.  I don't know how much FSF
Emacs manuals explain, though.

I've never figured out the MULE stuff myself.  This is why I don't use 
it, although I supposedly should.  :-(

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
Then...  his face does a complete change of expression.  It goes from
a "Vengeance is mine" expression, to a "What the fuck" blank look.

Re: MULE primer

[Michael Welsh Duggan]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> What is *the* way to learn about all this MULE stuff?  I have a
> distinct feeling I must be misunderstanding something basic here.  I
> mean, here's (日本語) some Japanese text, so I must be doing something 
> right, but I still don't understand it.

Well, you must be doing something right, because I too can receive and
read the Japanese txt.

> And the XEmacs and Emacs MULEs are subtly and not-so-subtly
> different, in that the XEmacs stuff works and the Emacs stuff
> doesn't.  (Well.)

Can you be more specific on what doesn't "work"?

> The Emacs 20.3 manual seems kinda slight on this subject...  Is there
> a new Emacs Lisp manual out?  Should I read the XEmacs Lisp manual?

Hmm...  The prereaders version that Alan Shutko mentioned is still at:
etlport.etl.go.jp:/pub/mule/.notready/elisp.xtar.gz

-- 
Michael Duggan
(md5i@cs.cmu.edu)
.

Re: MULE primer

[SL Baur]

[cc'ed to xemacs-mule]
Lars Magne Ingebrigtsen <larsi@gnus.org> writes in ding@gnus.org:

> What is *the* way to learn about all this MULE stuff?

For detailed technical documentation about charsets, JIS, etc. there's 
the O'Reilly Blowfish book -- Understanding Japanese Information
Processing.  It makes very dry reading except perhaps for Chapter 5 -- 
"Japanese Input" which does give some clues how to make things work.

Unfortunately, unless you can read Japanese there doesn't appear to be 
any documentation.  I learned how to get around by trying things out
and asking questions.  It also helped that I had to build and install
the required extra packages -- I have Canna, Wnn4.2, Wnn6 (courtesy of 
OMRON), SJ3, and Kinput2 (an XIM server that simultaneously can
connect to all of the above backends) installed on my development
machine.  They all work slightly differently, of course. :-P

English documentation for XEmacs is/was being worked on, but the
person who was doing it got as far as a very close draft then dropped
away :-(.

A cheap pocket tourist's dictionary is good enough to get some examples
to type in.

> I have a distinct feeling I must be misunderstanding something basic
> here.  I mean, here's (日本語) some Japanese text, so I must be
> doing something right, but I still don't understand it.

I'm not sure where to begin.  Assuming some basic knowledge of
Japanese ...

For example, the word 日本語 can be represented three different ways.
日本語 is the Kanji form and what you will see in writing.  It can
represented phonetically in hiragana as にほんご.  That's actually
what I see first entering the word in canna (C-\ japanese-canna).
Hiragana is a phonetic alphabet, so what you see is what you type to
get it, in this case `nihongo'.

Re: MULE primer

[Alan Shutko]

>>>>> "M" == Michael Welsh Duggan <md5i@cs.cmu.edu> writes:

M> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> I mean, here's (^[$(BF|K\8l^[(B) some Japanese text, so I must be doing
>> something right, but I still don't understand it.

M> Well, you must be doing something right, because I too can receive
M> and read the Japanese txt.

Not necessarily.  I can see it fine, and all I have is the
decode-quoted-unreadable on an old gnus!  

-- 
Alan Shutko <ats@acm.org> - By consent of the corrupted
Chess tonight.

Re: MULE primer

[Lars Magne Ingebrigtsen]

Michael Welsh Duggan <md5i@cs.cmu.edu> writes:

> > And the XEmacs and Emacs MULEs are subtly and not-so-subtly
> > different, in that the XEmacs stuff works and the Emacs stuff
> > doesn't.  (Well.)
> 
> Can you be more specific on what doesn't "work"?

I just can't seem to get a handle on how things work.  If I eval
`(rfc2047-decode-string "=?ISO-2022-JP?B?GyRCTVUybBsoQik=?=")' with
point in a unibyte buffer, then I get garbage, even if I don't intend
to insert the string.  If I'm in a multibyte buffer, things work.  So
it seems that if you want to have multibyte chars one place, you
almost have to force all buffers to be multibyte, just to make sure.
(Which is what those `set-buffer-multibyte' things that are littered
over Gnus now is all about.)

These things just *work* under XEmacs without having to think about
them. 

> Hmm...  The prereaders version that Alan Shutko mentioned is still at:
> etlport.etl.go.jp:/pub/mule/.notready/elisp.xtar.gz

Thanks; I'll have a look at it.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: MULE primer

[Lars Magne Ingebrigtsen]

Alan Shutko <ats@acm.org> writes:

> >> I mean, here's (^[$(BF|K\8l^[(B) some Japanese text, so I must be doing
> >> something right, but I still don't understand it.
> 
> M> Well, you must be doing something right, because I too can receive
> M> and read the Japanese txt.
> 
> Not necessarily.  I can see it fine, and all I have is the
> decode-quoted-unreadable on an old gnus!  

See?  I don't understand these things.

(This one should not have any japanese characters in it, since it's an 
us-ascii message that has some wacky escape sequences.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: MULE primer

[Stephen J. Turnbull]

>>>>> "sb" == SL Baur <steve@xemacs.org> writes:

    sb> [cc'ed to xemacs-mule]

Which is where I picked it up; I imagine it will continue on both
lists, I don't (and don't have time to :( ) subscribe to ding.

    sb> Lars Magne Ingebrigtsen <larsi@gnus.org> writes in
    sb> ding@gnus.org:

    >> What is *the* way to learn about all this MULE stuff?

    sb> For detailed technical documentation about charsets, JIS,
    sb> etc. there's the O'Reilly Blowfish book -- Understanding
    sb> Japanese Information Processing.  It makes very dry reading
    sb> except perhaps for Chapter 5 -- "Japanese Input" which does
    sb> give some clues how to make things work.

I agree with the reviewer's opinion about "dry," and it's primarily
relevant (to people who aren't implementing core Mule functionality)
as background.  BTW, "Son-of-Blowfish" is supposed to go to press RSN.
It has the advantage of being much more internationally-oriented,
supporting Korean, Chinese, and Vietnamese as well as Japanese.  A
partial and old version is available on-line at

      file://ftp.uu.net/vendor/oreilly/nutshell/ujip/doc/cjk.inf

(about 165k, and now over two years old).

    sb> Unfortunately, unless you can read Japanese there doesn't
    sb> appear to be any documentation.

I must not understand what the question was, because in XEmacs Info
there are useful discussions of Mule under both Internals and Lispref.
The extensive comments in the C code implementing XEmacs's Mule (there
was nothing like them in Mule 2.3, I don't know if the commenting has
increased in recent ETL/FSF Mules) are also useful.  I think those are
mostly in mule-coding.c and mule-charset.c.  All the functions I have
ever needed to use are documented there or via `describe-function'.

The latter is only useful if you know what function you want, of
course.  The resources listed are not a programmer's guide,
unfortunately.  A very coarse approximation to a programmer's guide
would be O'Reilly's "Xlib Programmer's Manual" (vol 1 of the series,
the name may be inexact, you need the R5 (separate volume) or R6
(revised edition, much better)), in the chapters on internationalizing
applications.  (I did write "_very_ coarse", OK?)  At least it gives
you some idea of how to use an internationalized programming system to
create applications that can use many languages.

Mule is somewhat more complicated, because it is multilingual, and the
X internationalization model doesn't really address that (viz the
clumsy contortions you need to go through to use multiple X Input
Methods).

For internal changes to XEmacs to support new functionality, Hrvoje
Niksic has written a Mule-izing guide for the C code (sorry, dunno the
URL offhand), but I don't think that's what you have in mind.

The ISO 2022 and Unicode v2 standards documents have some rationale
about language handling, but that should be considered deep background 
(sort of an appendix to "Blowfish").  Don't bother with ISO 10646, it
contains no rationale.

_Before_ you read the FSF's preprint elisp manual (the section
entitled, informatively enough, "non-ASCII characters") read the
corresponding XEmacs documentation.  The XEmacs documentation gets the 
abstractions and semantics right, the FSF's docs are unclear, if not
misleading, in several places.  The preprint elisp manual is also not
a programmer's guide (zero examples and little sense of the context in
which functions might be useful). :-(

    sb> A cheap pocket tourist's dictionary is good enough to get some
    sb> examples to type in.

There is also Jim Breen's edict package

	 file://ftp.monash.edu.au/pub/nihongo/edict.{gz,doc}

(a flat file, about 1MB; to read it directly in an XEmacs buffer you
may need to set the coding system to euc-japanese manually for reasons
I don't understand), which can be accessed via the XEmacs package
edict (which does not contain the dictionary itself at the
maintainer's request).  The edict package is also at least somewhat
compatible with Emacs 20.2, but I found working in that environment
really painful (the differences from Emacs 19.34/Mule 2.3 were, ah,
poorly documented) so I can't vouch for more than "it will load and
look up a couple of words".

    >> I have a distinct feeling I must be misunderstanding something
    >> basic here.  I mean, here's (日本語) some Japanese text, so I
    >> must be doing something right, but I still don't understand it.

Most things (such as handling buffers containing characters from
various codesets) are done fairly automatically by Mule, and aren't
your problem.  Some other things (such as inputting the characters)
are handled by external subsystems that interface directly to Mule,
and aren't your problem.

It's only when you want to do things like set the MIME Content-Type
header correctly, encode/decode non-ASCII message headers, encode
multilingual buffers to Postscript, or handle right to left languages
and bidirectional text (don't bother; nobody knows how to do this
right yet) that "knowing how to program Mule" becomes an issue.

    sb> I'm not sure where to begin.

Move to Hong Kong? :-)  I would suggest with the Xlib documentation on
writing internationalizable applications.  That gives a feeling for
the problems and the abstractions involved in internationalizating an
app.  There's also some discussion in the Motif Programming Guide
(same series, vol 6A I think).  If "Son-of-Blowfish" should happen to
appear on a bookshelf near you, I would recommend that.  I haven't
seen a draft, but an acquaintance at O'Reilly says that this book is
going to be more oriented to programming than its predecessor,
although still heavily theoretical.

One thing about the differences between Mule/XEmacs and Mule/FSFの
Emacs: they're not going to get smaller any time soon, and the FSF has
been making many changes to the interface, consistently over the 20.x
releases.  If you want the Emacs with the latest and greatest in
multilingual features, you'll probably want to track the changes the
ETL/FSF people are making.  Lots of pain for not much gain, IMHO.

XEmacs's Mule interface will probably be rather stable, at least for
as long as it takes you to learn how to do it.  I can predict with
fair confidence that Emacs's Mule will be UNstable for at least that
long, and in any case you will probably have to support 19.x, 20.0,
20.1, 20.2, and 20.3 Emacs separately in multilingual code.

Caveat: this last dismal assessment only applies if you mean to (a)
implement MIME standards for language handling directly and fully
internally to Gnus, or (b) get handling of buffers with multilingual
content absolutely pedantically precisely correct.  As mentioned
above, most basic text manipulations are handled automatically by all
the Mules.

-- 
University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Institute of Policy and Planning Sciences        Tel/fax: +1 (298) 53-5091

Re: MULE primer

[Michael Welsh Duggan]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Michael Welsh Duggan <md5i@cs.cmu.edu> writes:
> 
> > > And the XEmacs and Emacs MULEs are subtly and not-so-subtly
> > > different, in that the XEmacs stuff works and the Emacs stuff
> > > doesn't.  (Well.)
> > 
> > Can you be more specific on what doesn't "work"?
> 
> I just can't seem to get a handle on how things work.  If I eval
> `(rfc2047-decode-string "=?ISO-2022-JP?B?GyRCTVUybBsoQik=?=")' with
> point in a unibyte buffer, then I get garbage, even if I don't intend
> to insert the string.  If I'm in a multibyte buffer, things work.  So
> it seems that if you want to have multibyte chars one place, you
> almost have to force all buffers to be multibyte, just to make sure.
> (Which is what those `set-buffer-multibyte' things that are littered
> over Gnus now is all about.)
> 
> These things just *work* under XEmacs without having to think about
> them. 

If you think that this is a bug, you should report it.  This is an
odd-numbered Emacs release, and as such each and every bug you
encounter should be reported as soon as possible, so that a fix can
make it into the subsequent even-numbered release.

-- 
Michael Duggan
(md5i@cs.cmu.edu)
.

Re: MULE primer

[William M. Perry]

Michael Welsh Duggan <md5i@cs.cmu.edu> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Michael Welsh Duggan <md5i@cs.cmu.edu> writes:
> > 
> > > > And the XEmacs and Emacs MULEs are subtly and not-so-subtly
> > > > different, in that the XEmacs stuff works and the Emacs stuff
> > > > doesn't.  (Well.)
> > > 
> > > Can you be more specific on what doesn't "work"?
> > 
> > I just can't seem to get a handle on how things work.  If I eval
> > `(rfc2047-decode-string "=?ISO-2022-JP?B?GyRCTVUybBsoQik=?=")' with
> > point in a unibyte buffer, then I get garbage, even if I don't intend
> > to insert the string.  If I'm in a multibyte buffer, things work.  So
> > it seems that if you want to have multibyte chars one place, you
> > almost have to force all buffers to be multibyte, just to make sure.
> > (Which is what those `set-buffer-multibyte' things that are littered
> > over Gnus now is all about.)
> > 
> > These things just *work* under XEmacs without having to think about
> > them. 
> 
> If you think that this is a bug, you should report it.  This is an
> odd-numbered Emacs release, and as such each and every bug you
> encounter should be reported as soon as possible, so that a fix can
> make it into the subsequent even-numbered release.

  Especially as RMS is on the road this month, and plans to do 20.4 while
he is gone of just bugfixes, etc.

-bp

font suggestions for GNU Emacs? (was Re: MULE primer)

[John H Palmieri]

GNU Emacs 20.3.1 (sparc-sun-solaris2.5.1, X toolkit) of Thu Aug 27 1998 on darwin

I am looking for suggestions about what fontset to use, or what fonts
to use to make a fontset, or something like that, so that I can
pleasantly read MULE stuff.  I think I have these choices:


1. Do the default, which seems to use the font

 -Misc-Fixed-Medium-R-Normal--15-140-75-75-C-90-ISO8859-1

This isn't able to handle many international fonts.  (Well, it can do
accents and such in French, German, and Spanish, but that's about it.)


2. I could use "standard fontset", which can handle a lot more
international fonts (e.g., it lets me see Japanese characters in the
passage below), but is butt-ugly (in my opinion).  This option does
not let me *pleasantly* read MULE stuff.

     -*-fixed-medium-r-normal-*-16-*-*-*-*-*-fontset-standard


3. Construct another fontset by hand in, say, my .Xdefaults file.


4. Download some other fonts from somewhere.  (I have intlfonts-1.1
already.)


Does anyone have suggestions for either 3 or 4?  Or is there something
easier I could do?  (Or harder: should I build GNU Emacs again,
telling it to use some fancy new fontset as the default?)

> What is *the* way to learn about all this MULE stuff?  I have a
> distinct feeling I must be misunderstanding something basic here.  I
> mean, here's (日本語) some Japanese text, so I must be doing something 
> right, but I still don't understand it.

-- 
John H. Palmieri
e-mail: palmieri@member.ams.org        205 Computing/Mathematics Building
URL: http://www.nd.edu/~jpalmier/      University of Notre Dame
(219) 631-8846                         Notre Dame, IN 46556

Re: MULE primer

[Lars Magne Ingebrigtsen]

Michael Welsh Duggan <md5i@cs.cmu.edu> writes:

> If you think that this is a bug, you should report it.  This is an
> odd-numbered Emacs release, and as such each and every bug you
> encounter should be reported as soon as possible, so that a fix can
> make it into the subsequent even-numbered release.

I don't think these are bugs -- they are just the result of Emacs'
unibyte/multibyte model.  XEmacs doesn't have that problem.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: MULE primer

[Alan Shutko]

>>>>> "L" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

L> Alan Shutko <ats@acm.org> writes:

>>  Not necessarily.  I can see it fine, and all I have is the
>> decode-quoted-unreadable on an old gnus!

L> See?  I don't understand these things.

L> (This one should not have any japanese characters in it, since it's
L> an us-ascii message that has some wacky escape sequences.)

Actually, it did, for me.  Seems that mule just recognized there was
japanese stuff in there and treated it appropriately.  Probably would
have caused problems if there really was some wacky escape characters
in it.

(Remember, this is with an old Gnus, not pgnus.)

-- 
Alan Shutko <ats@acm.org> - By consent of the corrupted
It's not the fall that kills you, it's the landing.

Re: MULE primer

[Hallvard B Furuseth]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> I don't think these are bugs -- they are just the result of Emacs'
> unibyte/multibyte model.  XEmacs doesn't have that problem.

Well, it's at least a documentation problem, since you needed to ask.

-- 
Hallvard

Re: font suggestions for GNU Emacs? (was Re: MULE primer)

[John H Palmieri]

John H Palmieri <John.H.Palmieri.2@nd.edu> writes:

> GNU Emacs 20.3.1 (sparc-sun-solaris2.5.1, X toolkit) of Thu Aug 27 1998 on darwin
> 
> I am looking for suggestions about what fontset to use, or what fonts
> to use to make a fontset, or something like that, so that I can
> pleasantly read MULE stuff.  I think I have these choices:

   [snip]
 
> 2. I could use "standard fontset", which can handle a lot more
> international fonts (e.g., it lets me see Japanese characters in the
> passage below), but is butt-ugly (in my opinion).  This option does
> not let me *pleasantly* read MULE stuff.

   [snip]

> 4. Download some other fonts from somewhere.  (I have intlfonts-1.1
> already.)

Wait, it's looking better now.  When I first installed intlfonts-1.1,
somehow I missed the instructions about running mkfontdir (which
didn't get run because I didn't do a full installation, perhaps?).
I've now run that, and the standard fontset looks much better for some
reason.  Hitting C-h h gives me a screen full of all sorts of pretty
fonts, too.

-- 
John H. Palmieri
e-mail: palmieri@member.ams.org        205 Computing/Mathematics Building
URL: http://www.nd.edu/~jpalmier/      University of Notre Dame
(219) 631-8846                         Notre Dame, IN 46556