Problem using some Japanese characters on Windows

Problem using some Japanese characters on Windows

[Rajesh Jangam]

[-- Attachment #1.1: Type: text/plain, Size: 730 bytes --]

Hi All,

I am a zsh user and using zsh-4.3.4 on a Japanese windows box.
I compiled it with multi-byte support enabled.

However, for some Japanese strings, it gives a "bad pattern" error.
Attached is a screen shot with an example.

After debugging a bit, I found that we have used some characters in the
extended ascii range
for marking special characters like: Star(*), Quest(?) etc.

However these seem to clash with the CP932 character set which is the
default on Windows
CP932 defines characters in the extended Ascii character range and beyond.
(> 0x81)

Please see:
http://www.microsoft.com/globaldev/reference/dbcs/932.mspx

Please do let me know if this is a valid bug and if so, how can we get
around it?

Thanks,
Rajesh

[-- Attachment #1.2: Type: text/html, Size: 883 bytes --]

[-- Attachment #2: zsh-bug.JPG --]
[-- Type: image/jpeg, Size: 31373 bytes --]

Re: Problem using some Japanese characters on Windows

[Peter Stephenson]

On Mon, 4 Feb 2008 19:30:59 +0530
"Rajesh Jangam" <rajeshjangam@gmail.com> wrote:
> I am a zsh user and using zsh-4.3.4 on a Japanese windows box.
> I compiled it with multi-byte support enabled.
> 
> However, for some Japanese strings, it gives a "bad pattern" error.
> Attached is a screen shot with an example.

I presume you're using cygwin.  I haven't investigated the multibyte
support there in any detail, but my impression last time I looked was that
it was quite limited at the level of system libraries etc.  (Windows has
its own ways of doing this that are of course much more complete but aren't
directly accessible.)

> After debugging a bit, I found that we have used some characters in the
> extended ascii range
> for marking special characters like: Star(*), Quest(?) etc.

This (probably) isn't the problem: the special characters are distinguished
from native characters by appropriate internal quoting (which should be
transparent to the user).  I say "probably" because there may be local
bugs, but you'll have to give details of what you're doing.  (Try the new
version 4.3.5 first.)

> However these seem to clash with the CP932 character set which is the
> default on Windows
> CP932 defines characters in the extended Ascii character range and beyond.
> (> 0x81)

You certainly need to make sure the shell has been told about the character
set, typically by setting the environment variable LANG to the appropriate
locale, otherwise the shell won't know what to do with characters in that
range.  The point of multi-byte support is essentially that the shell does
rely on the system, instead of guessing.

This is where I'm unsure how good the support currently is in Cygwin.  If
it doesn't make your locale available then you're out of luck and you might
be better off configuring with --disable-multibyte.  You might get more
information on that score from Cygwin people.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070