[OSX] weird UTF-8 behavior with cd autocompletion

[OSX] weird UTF-8 behavior with cd autocompletion

[Xavier Guérin]

[-- Attachment #1: Type: text/plain, Size: 581 bytes --]

Dear list,

I'm encountering a small issue with my ZSHs, both 4.3.9 and 4.3.10 devel.
Although UTF-8 works fine, autocompletion with the CD command produces a
weird output.

For instance, let's take this directory structure:

~/Documents/
~/Documents/Conférence

And the following command :

$ cd ~/Documents/Conf + <TAB>

I get the following output:

$ cd ~/Documents/Confe<0301>rences/

My terminal.app is configured with UTF-8 encoding, and both LANG and LC_ALL
are set as "fr_FR.UTF-8".

Do you have any idea how I could fixe that ?

Regards,
Xavier

Re: [OSX] weird UTF-8 behavior with cd autocompletion

[Andrey Borzenkov]

[-- Attachment #1: Type: Text/Plain, Size: 760 bytes --]

On Thursday 11 of February 2010 21:04:18 Xavier Guérin wrote:
> Dear list,
> 
> I'm encountering a small issue with my ZSHs, both 4.3.9 and 4.3.10
> devel. Although UTF-8 works fine, autocompletion with the CD command
> produces a weird output.
> 
> For instance, let's take this directory structure:
> 
> ~/Documents/
> ~/Documents/Conférence
> 
> And the following command :
> 
> $ cd ~/Documents/Conf + <TAB>
> 
> I get the following output:
> 
> $ cd ~/Documents/Confe<0301>rences/
> 
> My terminal.app is configured with UTF-8 encoding, and both LANG and
> LC_ALL are set as "fr_FR.UTF-8".
> 

I cannot reproduce it here running Linux (Mandriva cooker), locale 
ru_RU.UTF-8 (exactly your example). May be it is system-dependent?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [OSX] weird UTF-8 behavior with cd autocompletion

[Benjamin R. Haskell]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 956 bytes --]

On Thu, 11 Feb 2010, Xavier Guérin wrote:

> Dear list,
> 
> I'm encountering a small issue with my ZSHs, both 4.3.9 and 4.3.10 
> devel.  Although UTF-8 works fine, autocompletion with the CD command 
> produces a weird output.
> 
> For instance, let's take this directory structure:
> 
> ~/Documents/
> ~/Documents/Conférence
> 
> And the following command :
> 
> $ cd ~/Documents/Conf + <TAB>
> 
> I get the following output:
> 
> $ cd ~/Documents/Confe<0301>rences/
> 
> My terminal.app is configured with UTF-8 encoding, and both LANG and 
> LC_ALL are set as "fr_FR.UTF-8".
> 
> Do you have any idea how I could fixe that ?

The problem is that HFS+ stores filenames in Unicode Normalization Form 
D (D = Decomposed).  So, even though this character exists:

é  00e9  LATIN SMALL LETTER E WITH ACUTE

On disk, it's stored as:
e  0065  LATIN SMALL LETTER E
   0301  COMBINING ACUTE ACCENT

That's the diagnosis.  Don't know the cure.

-- 
Best,
Ben

Re: [OSX] weird UTF-8 behavior with cd autocompletion

[Mikael Magnusson]

2010/2/11 Xavier Guérin <guerinx@gmail.com>:
> Dear list,
>
> I'm encountering a small issue with my ZSHs, both 4.3.9 and 4.3.10 devel.
> Although UTF-8 works fine, autocompletion with the CD command produces a
> weird output.
>
> For instance, let's take this directory structure:
>
> ~/Documents/
> ~/Documents/Conférence
>
> And the following command :
>
> $ cd ~/Documents/Conf + <TAB>
>
> I get the following output:
>
> $ cd ~/Documents/Confe<0301>rences/
>
> My terminal.app is configured with UTF-8 encoding, and both LANG and LC_ALL
> are set as "fr_FR.UTF-8".
>
> Do you have any idea how I could fixe that ?

setopt combiningchars

-- 
Mikael Magnusson

Re: [OSX] weird UTF-8 behavior with cd autocompletion

[David Evans]

On Thu, Feb 11, 2010 at 07:59:49PM +0100, Mikael Magnusson wrote:
> 
> setopt combiningchars
> 

  Has this made it into the docs?  It doesn't seem to be in the man
pages I have for 4.3.10.

Re: [OSX] weird UTF-8 behavior with cd autocompletion

[Mikael Magnusson]

On 11 February 2010 20:14, David Evans <David.Evans@cl.cam.ac.uk> wrote:
> On Thu, Feb 11, 2010 at 07:59:49PM +0100, Mikael Magnusson wrote:
>>
>> setopt combiningchars
>>
>
>  Has this made it into the docs?  It doesn't seem to be in the man
> pages I have for 4.3.10.

Yes, look for COMBINING_CHARS

-- 
Mikael Magnusson

Re: [OSX] weird UTF-8 behavior with cd autocompletion

[Xavier Guérin]

[-- Attachment #1: Type: text/plain, Size: 985 bytes --]

I have one more question:

With this option set and "%c" set in PS1, the UTF8 characters present in the
current directory name are printed before PS1.
Example with PS1="[%c]$" :

[Thèse]$ ls
...
è%
[Thèse]

Or:

[tété]$ ls
...
éé$
[tété]

Reading the documentation, it looks like print does not handle UTF-8
properly. Is is linked or is there another magic setopt ? :)

Regards,
Xavier

Le 11 févr. 2010 à 20:14, David Evans a écrit :

> On Thu, Feb 11, 2010 at 07:59:49PM +0100, Mikael Magnusson wrote:
>>
>> setopt combiningchars
>>
>
>  Has this made it into the docs?  It doesn't seem to be in the man
> pages I have for 4.3.10.
>


-----
Xavier Guérin - Doctoral student
TIMA Laboratory, SLS Group
46, av. Félix Viallet, 38000 GRENOBLE, FR
Mail : xavier.guerin@imag.fr<javascript:webmailto('xavier.guerin@imag.fr');>

Phone : +33(0)4 76 57 47 59
Ad Astra Per Aspera
Triste e' quel discepolo che non avanza il suo maestro