Phil's prompt is not working when LANG is set to UTF-8

Phil's prompt is not working when LANG is set to UTF-8

[Wael Nasreddine]

[-- Attachment #1: Type: text/plain, Size: 836 bytes --]

Hello,

I'm having a problem with my zsh prompt which uses phil's prompt[1],
when I export LANG=en_US.utf8 the prompt is, as you can see in the
screenshot[2], all messed up, anyone knows why this is happening??
I've tried it on both Linux and Unix and I have had the same results.
Also please take a look at the discussion[3] that happend on IRC a
while ago.

Thank you

[1]: http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.zsh/prompts/phils?view=markup
[2]: http://img482.imageshack.us/img482/3899/wrongrprompt3su1.png
[3]: http://nopaste.nasreddine.com/18ca877db3.html

-- 
Wael Nasreddine
http://wael.nasreddine.com
PGP: 1024D/C8DD18A2 06F6 1622 4BC8 4CEB D724  DE12 5565 3945 C8DD 18A2

.: An infinite number of monkeys typing into GNU emacs,
   would never make a good program. (L. Torvalds 1995) :.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Mikael Magnusson]

On 11/02/2008, Wael Nasreddine <mla@nasreddine.com> wrote:
> Hello,
>
> I'm having a problem with my zsh prompt which uses phil's prompt[1],
> when I export LANG=en_US.utf8 the prompt is, as you can see in the
> screenshot[2], all messed up, anyone knows why this is happening??
> I've tried it on both Linux and Unix and I have had the same results.
> Also please take a look at the discussion[3] that happend on IRC a
> while ago.
>
> Thank you

What version of zsh are you using? It works fine here with 4.3.5 and
rxvt-unicode both set to UTF-8 before sourcing the prompt.

-- 
Mikael Magnusson

Re: Phil's prompt is not working when LANG is set to UTF-8

[Wael Nasreddine]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

On Mon, Feb 11, 2008 at 08:01:36AM +0100, Mikael Magnusson <mikachu@gmail.com> wrote:
> On 11/02/2008, Wael Nasreddine <mla@nasreddine.com> wrote:
> > Hello,

> > I'm having a problem with my zsh prompt which uses phil's prompt[1],
> > when I export LANG=en_US.utf8 the prompt is, as you can see in the
> > screenshot[2], all messed up, anyone knows why this is happening??
> > I've tried it on both Linux and Unix and I have had the same results.
> > Also please take a look at the discussion[3] that happend on IRC a
> > while ago.

> > Thank you

> What version of zsh are you using? It works fine here with 4.3.5 and
> rxvt-unicode both set to UTF-8 before sourcing the prompt.

Oh I'm sorry I forgot to mention that this only happens on consoles not in X
emulators ( /dev/tty1 etc... ) I have also the combination of
rxvt-unicode/zsh/UTF-8 and it works fine but on the console it's too messy so
I'm deprived of consoles ATM.

Anyway my zsh version is 4.3.5

-- 
Wael Nasreddine
http://wael.nasreddine.com
PGP: 1024D/C8DD18A2 06F6 1622 4BC8 4CEB D724  DE12 5565 3945 C8DD 18A2

.: An infinite number of monkeys typing into GNU emacs,
   would never make a good program. (L. Torvalds 1995) :.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Peter Stephenson]

On Mon, 11 Feb 2008 10:20:21 +0100
Wael Nasreddine <mla@nasreddine.com> wrote:
> Oh I'm sorry I forgot to mention that this only happens on consoles not in X
> emulators ( /dev/tty1 etc... ) I have also the combination of
> rxvt-unicode/zsh/UTF-8 and it works fine but on the console it's too messy so
> I'm deprived of consoles ATM.
> 
> Anyway my zsh version is 4.3.5

I suspect the right thing to do here might be either to "unsetopt
multibyte" or "export LC_ALL=C".  It sounds like the combination of
the console and the C library aren't doing the stuff the shell needs
to support multibyte characters.  If that's not the problem, please send
more detail about your settings and environment and what works and what
doesn't work.

(That last sentence applies to everybody.  Zsh runs in some very different
environments.  We could *always* do with knowing what that is, even
if it turns out not to be relevant.  If it applies in lots of
different environments, please say so---as many people already do.)

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: Phil's prompt is not working when LANG is set to UTF-8

[Wael Nasreddine]

[-- Attachment #1: Type: text/plain, Size: 3148 bytes --]

On Mon, Feb 11, 2008 at 01:07:07PM +0000, Peter Stephenson <pws@csr.com> wrote:
> On Mon, 11 Feb 2008 10:20:21 +0100
> Wael Nasreddine <mla@nasreddine.com> wrote:
> > Oh I'm sorry I forgot to mention that this only happens on consoles not in X
> > emulators ( /dev/tty1 etc... ) I have also the combination of
> > rxvt-unicode/zsh/UTF-8 and it works fine but on the console it's too messy so
> > I'm deprived of consoles ATM.

> > Anyway my zsh version is 4.3.5

> I suspect the right thing to do here might be either to "unsetopt
> multibyte" or "export LC_ALL=C".  It sounds like the combination of
> the console and the C library aren't doing the stuff the shell needs
> to support multibyte characters.  If that's not the problem, please send
> more detail about your settings and environment and what works and what
> doesn't work.

> (That last sentence applies to everybody.  Zsh runs in some very different
> environments.  We could *always* do with knowing what that is, even
> if it turns out not to be relevant.  If it applies in lots of
> different environments, please say so---as many people already do.)

Thank you for your reply, but, set/unsetting multibyte opt has no
effect but as said before when LANG is set to utf8 it doesn't work,
for now I use LANG=en_US (iso-8859-1) it does work this way though I
don't want to stick with this encoding, anyway here's some more
info for you.

My current setup is Gentoo, 2.6.24, I thought it was irrelevant
because as I said before I tried it on both Linux and Unix, I tried
Arch Linux, Ubuntu, Gentoo, Frugalware, FreeBSD etc.. All with the
same result, so I know that the problem comes from somewhere in my
profile, an incompatibility or something I'm not sure.

.zshrc and all the files that it includes:
    http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.zshrc?view=co
    http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.zsh/alias?view=co
    http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.zsh/bindkey?view=co
    http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.zsh/opts?view=co  <-- where all opts are set.
    http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.zsh/prompts/phils?view=log
    http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.shells/alias?view=log
    http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.shells/autolaunch?view=log
    http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.shells/colors?view=log
    http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.shells/exports?view=log <-- where I define the LANG.
    http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.shells/functions?view=log

Just to try and save you some time reading all those files here's the
output of:
setopt: http://nopaste.nasreddine.com/fed365b5cb.html
env: http://nopaste.nasreddine.com/4d004fdb39.html


Thank you.

-- 
Wael Nasreddine
http://wael.nasreddine.com
PGP: 1024D/C8DD18A2 06F6 1622 4BC8 4CEB D724  DE12 5565 3945 C8DD 18A2

.: An infinite number of monkeys typing into GNU emacs,
   would never make a good program. (L. Torvalds 1995) :.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Wael Nasreddine]

[-- Attachment #1: Type: text/plain, Size: 4512 bytes --]

On Mon, Feb 11, 2008 at 02:37:43PM +0100, Wael Nasreddine <mla@nasreddine.com> wrote:
> On Mon, Feb 11, 2008 at 01:07:07PM +0000, Peter Stephenson <pws@csr.com> wrote:
> > On Mon, 11 Feb 2008 10:20:21 +0100
> > Wael Nasreddine <mla@nasreddine.com> wrote:
> > > Oh I'm sorry I forgot to mention that this only happens on consoles not in X
> > > emulators ( /dev/tty1 etc... ) I have also the combination of
> > > rxvt-unicode/zsh/UTF-8 and it works fine but on the console it's too messy so
> > > I'm deprived of consoles ATM.

> > > Anyway my zsh version is 4.3.5

> > I suspect the right thing to do here might be either to "unsetopt
> > multibyte" or "export LC_ALL=C".  It sounds like the combination of
> > the console and the C library aren't doing the stuff the shell needs
> > to support multibyte characters.  If that's not the problem, please send
> > more detail about your settings and environment and what works and what
> > doesn't work.

> > (That last sentence applies to everybody.  Zsh runs in some very different
> > environments.  We could *always* do with knowing what that is, even
> > if it turns out not to be relevant.  If it applies in lots of
> > different environments, please say so---as many people already do.)

> Thank you for your reply, but, set/unsetting multibyte opt has no
> effect but as said before when LANG is set to utf8 it doesn't work,
> for now I use LANG=en_US (iso-8859-1) it does work this way though I
> don't want to stick with this encoding, anyway here's some more
> info for you.

> My current setup is Gentoo, 2.6.24, I thought it was irrelevant
> because as I said before I tried it on both Linux and Unix, I tried
> Arch Linux, Ubuntu, Gentoo, Frugalware, FreeBSD etc.. All with the
> same result, so I know that the problem comes from somewhere in my
> profile, an incompatibility or something I'm not sure.

> .zshrc and all the files that it includes:
>     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.zshrc?view=co
>     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.zsh/alias?view=co
>     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.zsh/bindkey?view=co
>     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.zsh/opts?view=co  <-- where all opts are set.
>     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.zsh/prompts/phils?view=log
>     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.shells/alias?view=log
>     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.shells/autolaunch?view=log
>     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.shells/colors?view=log
>     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.shells/exports?view=log <-- where I define the LANG.
>     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/trunk/etc/.shells/functions?view=log

> Just to try and save you some time reading all those files here's the
> output of:
> setopt: http://nopaste.nasreddine.com/fed365b5cb.html
> env: http://nopaste.nasreddine.com/4d004fdb39.html

> Thank you.

Please use the below URLs instead, I just migrated my servers to
FreeBSD and I'm still fixing some stuff, google knows my repo this way
not the one above, sorry for any inconvenience.
.zshrc and all the files that it includes:
     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/wael/trunk/etc/.zshrc?view=co
     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/wael/trunk/etc/.zsh/alias?view=co
     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/wael/trunk/etc/.zsh/bindkey?view=co
     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/wael/trunk/etc/.zsh/opts?view=co  <-- where all opts are set.
     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/wael/trunk/etc/.zsh/prompts/phils?view=log
     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/wael/trunk/etc/.shells/alias?view=log
     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/wael/trunk/etc/.shells/autolaunch?view=log
     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/wael/trunk/etc/.shells/colors?view=log
     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/wael/trunk/etc/.shells/exports?view=log <-- where I define the LANG.
     http://wael.nasreddine.com/cgi-bin/viewvc.cgi/wael/trunk/etc/.shells/functions?view=log


-- 
Wael Nasreddine
http://wael.nasreddine.com
PGP: 1024D/C8DD18A2 06F6 1622 4BC8 4CEB D724  DE12 5565 3945 C8DD 18A2

.: An infinite number of monkeys typing into GNU emacs,
   would never make a good program. (L. Torvalds 1995) :.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Peter Stephenson]

Wael Nasreddine wrote:
> Thank you for your reply, but, set/unsetting multibyte opt has no
> effect but as said before when LANG is set to utf8 it doesn't work,
> for now I use LANG=en_US (iso-8859-1) it does work this way though I
> don't want to stick with this encoding, anyway here's some more
> info for you.

This is probably the key information: it suggests, unless you happen to
know otherwise, that the console simply doesn't support UTF-8 in its
current configuration.

There are two ingredients to being able to use an encoding:  the
hardware is set up to display characters using it, and the software has
been told about it.  LANG etc. deal with the latter, but have no effect
on the former.  Smart terminal emulators will try to find the encoding
when they start up.  You don't really have this option with a console.
My system (Fedora 8) has a unicode_start command; have you tried that?

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: Phil's prompt is not working when LANG is set to UTF-8

[Wael Nasreddine]

[-- Attachment #1: Type: text/plain, Size: 1522 bytes --]

On Mon, Feb 11, 2008 at 02:07:07PM +0000, Peter Stephenson <pws@csr.com> wrote:
> Wael Nasreddine wrote:
> > Thank you for your reply, but, set/unsetting multibyte opt has no
> > effect but as said before when LANG is set to utf8 it doesn't work,
> > for now I use LANG=en_US (iso-8859-1) it does work this way though I
> > don't want to stick with this encoding, anyway here's some more
> > info for you.

> This is probably the key information: it suggests, unless you happen to
> know otherwise, that the console simply doesn't support UTF-8 in its
> current configuration.

> There are two ingredients to being able to use an encoding:  the
> hardware is set up to display characters using it, and the software has
> been told about it.  LANG etc. deal with the latter, but have no effect
> on the former.  Smart terminal emulators will try to find the encoding
> when they start up.  You don't really have this option with a console.
> My system (Fedora 8) has a unicode_start command; have you tried that?

I tried both unicode_start and manually, I tried this whole HOWTO[1]
as well without any change :( does it work on your PC using the same
prompt and LANG, LC_ALL are utf8??

[1]: http://gentoo-wiki.com/HOWTO_Make_your_system_use_unicode/utf-8

-- 
Wael Nasreddine
http://wael.nasreddine.com
PGP: 1024D/C8DD18A2 06F6 1622 4BC8 4CEB D724  DE12 5565 3945 C8DD 18A2

.: An infinite number of monkeys typing into GNU emacs,
   would never make a good program. (L. Torvalds 1995) :.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[antho.charles]

[-- Attachment #1: Type: text/plain, Size: 1077 bytes --]

Hi,

I didn't have this problem on Debian etch, but it appears when I
upgraded to lenny. It's the same problem: if I set LANG to a non utf8
encoding (fr_FR instead of fr_FR.UTF-8), RPROMPT is good, otherwise
it's partially on another line and the cusor is after RPROMPT.
(cf. http://tinyurl.com/3xjeqt)
I "solve" this problem with:

-- 8< -- 8< --
 ###
 # See if we can use extended characters to look nicer.
 
 if [[ $TERM != linux ]]; then
    typeset -A altchar
    set -A altchar ${(s..)terminfo[acsc]}
    PR_SET_CHARSET="%{$terminfo[enacs]%}"
    PR_SHIFT_IN="%{$terminfo[smacs]%}"
    PR_SHIFT_OUT="%{$terminfo[rmacs]%}"
    PR_HBAR=${altchar[q]:--}
    PR_ULCORNER=${altchar[l]:--}
    PR_LLCORNER=${altchar[m]:--}
    PR_LRCORNER=${altchar[j]:--}
    PR_URCORNER=${altchar[k]:--}
fi
-- 8< -- 8< --

I don't define those variables if I'm in a console, then the prompt
looks good but without the fancy stuff, so it's not the real answer.
(cf. http://tinyurl.com/2bjl6x)
And my consoles are already in unicode mode.

-- 
Anthony CHARLES

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Peter Stephenson]

On Mon, 11 Feb 2008 16:07:09 +0100
antho.charles@gmail.com wrote:
> I didn't have this problem on Debian etch, but it appears when I
> upgraded to lenny. It's the same problem: if I set LANG to a non utf8
> encoding (fr_FR instead of fr_FR.UTF-8), RPROMPT is good, otherwise
> it's partially on another line and the cusor is after RPROMPT.
> (cf. http://tinyurl.com/3xjeqt)

Sounds like it ought to be fairly reproducible, but it's not happening on
Fedora 8 (and it still sounds suspiciously like the shell is getting duff
information about the environment, though that's certainly not the only
possibility).  I've tried adding multibyte characters to the command line
and complicating the RPROMPT and even adding multibyte characters to that,
but it still works OK.

Are there any particular things on the command line, forms of RPROMPT etc.
etc. that show this up?  (We really need something that narrows this down.)

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: Phil's prompt is not working when LANG is set to UTF-8

[Anthony Charles]

[-- Attachment #1: Type: text/plain, Size: 318 bytes --]

On Mon, Feb 11, 2008 at 03:52:00PM +0000, Peter Stephenson wrote:
> Are there any particular things on the command line, forms of RPROMPT etc.
> etc. that show this up?  (We really need something that narrows this down.)

I'll be off this week, but I will have time to test this week-end.

-- 
Anthony CHARLES

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Wael Nasreddine]

[-- Attachment #1: Type: text/plain, Size: 1610 bytes --]

On Mon, Feb 11, 2008 at 04:07:09PM +0100, antho.charles@gmail.com wrote:
> Hi,

> I didn't have this problem on Debian etch, but it appears when I
> upgraded to lenny. It's the same problem: if I set LANG to a non utf8
> encoding (fr_FR instead of fr_FR.UTF-8), RPROMPT is good, otherwise
> it's partially on another line and the cusor is after RPROMPT.
> (cf. http://tinyurl.com/3xjeqt)
> I "solve" this problem with:

> -- 8< -- 8< --

>  # See if we can use extended characters to look nicer.

>  if [[ $TERM != linux ]]; then
>     typeset -A altchar
>     set -A altchar ${(s..)terminfo[acsc]}
>     PR_SET_CHARSET="%{$terminfo[enacs]%}"
>     PR_SHIFT_IN="%{$terminfo[smacs]%}"
>     PR_SHIFT_OUT="%{$terminfo[rmacs]%}"
>     PR_HBAR=${altchar[q]:--}
>     PR_ULCORNER=${altchar[l]:--}
>     PR_LLCORNER=${altchar[m]:--}
>     PR_LRCORNER=${altchar[j]:--}
>     PR_URCORNER=${altchar[k]:--}
> fi
> -- 8< -- 8< --

> I don't define those variables if I'm in a console, then the prompt
> looks good but without the fancy stuff, so it's not the real answer.
> (cf. http://tinyurl.com/2bjl6x)
> And my consoles are already in unicode mode.

This could be a nice solution for the time being, at least when I call
the auto completion via double tabs I won't go blind :) though we
really need a real one.

> -- 
> Anthony CHARLES

-- 
Wael Nasreddine
http://wael.nasreddine.com
PGP: 1024D/C8DD18A2 06F6 1622 4BC8 4CEB D724  DE12 5565 3945 C8DD 18A2

.: An infinite number of monkeys typing into GNU emacs,
   would never make a good program. (L. Torvalds 1995) :.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Wael Nasreddine]

[-- Attachment #1: Type: text/plain, Size: 2171 bytes --]

On Mon, Feb 11, 2008 at 03:52:00PM +0000, Peter Stephenson <pws@csr.com> wrote:
> On Mon, 11 Feb 2008 16:07:09 +0100
> antho.charles@gmail.com wrote:
> > I didn't have this problem on Debian etch, but it appears when I
> > upgraded to lenny. It's the same problem: if I set LANG to a non utf8
> > encoding (fr_FR instead of fr_FR.UTF-8), RPROMPT is good, otherwise
> > it's partially on another line and the cusor is after RPROMPT.
> > (cf. http://tinyurl.com/3xjeqt)

> Sounds like it ought to be fairly reproducible, but it's not happening on
> Fedora 8 (and it still sounds suspiciously like the shell is getting duff
> information about the environment, though that's certainly not the only
> possibility).  I've tried adding multibyte characters to the command line
> and complicating the RPROMPT and even adding multibyte characters to that,
> but it still works OK.

> Are there any particular things on the command line, forms of RPROMPT etc.
> etc. that show this up?  (We really need something that narrows this down.)

I haven't wrote this PROMPT I simply found it on google[1], it's an
exact copy of it though I have disabled APM/IBAM support beause
sometimes ACPI hangs on this computer and so all my shells hang.

Anyway I'll try it on Fedora and see how it goes, If I had the same
result then it's probably something in my environment, in that case
could you please try it with my whole environment?? Just add a new
user (test user) and do the following:

$ cd ~
$ svn co svn://wael.nasreddine.com/wael/trunk/co-base .

This should check out my basic configurations, everything related to
the shells/environment...

Then create all the symlinks..

$ ~/bin/svnfix

This would put all files where they should have been..

Now you could test with my environment and this way we can narrow it
down to System/Environment problems...


[1]: http://aperiodic.net/phil/prompt/

-- 
Wael Nasreddine
http://wael.nasreddine.com
PGP: 1024D/C8DD18A2 06F6 1622 4BC8 4CEB D724  DE12 5565 3945 C8DD 18A2

.: An infinite number of monkeys typing into GNU emacs,
   would never make a good program. (L. Torvalds 1995) :.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 1365 bytes --]

On Monday 11 February 2008, Peter Stephenson wrote:
> 
> On Mon, 11 Feb 2008 16:07:09 +0100
> antho.charles@gmail.com wrote:
> > I didn't have this problem on Debian etch, but it appears when I
> > upgraded to lenny. It's the same problem: if I set LANG to a non utf8
> > encoding (fr_FR instead of fr_FR.UTF-8), RPROMPT is good, otherwise
> > it's partially on another line and the cusor is after RPROMPT.
> > (cf. http://tinyurl.com/3xjeqt)
> 
> Sounds like it ought to be fairly reproducible, but it's not happening on
> Fedora 8 (and it still sounds suspiciously like the shell is getting duff
> information about the environment, though that's certainly not the only
> possibility).  I've tried adding multibyte characters to the command line
> and complicating the RPROMPT and even adding multibyte characters to that,
> but it still works OK.
> 
> Are there any particular things on the command line, forms of RPROMPT etc.
> etc. that show this up?  (We really need something that narrows this down.)
> 

I can reproduce it on Mandriva cooker with locale en_US.UTF-8 or ru_RU.UTF-8. It 
does not matter whether there are UTF-8 characters on the screen (at least, I am 
not sure whether there are - en_US.UTF-8 should not emit any non-ASCII strings 
as far as I can tell).

I will try to find time on weekend to check more closely.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 2184 bytes --]

On Wednesday 13 February 2008, Andrey Borzenkov wrote:
> On Monday 11 February 2008, Peter Stephenson wrote:
> > 
> > On Mon, 11 Feb 2008 16:07:09 +0100
> > antho.charles@gmail.com wrote:
> > > I didn't have this problem on Debian etch, but it appears when I
> > > upgraded to lenny. It's the same problem: if I set LANG to a non utf8
> > > encoding (fr_FR instead of fr_FR.UTF-8), RPROMPT is good, otherwise
> > > it's partially on another line and the cusor is after RPROMPT.
> > > (cf. http://tinyurl.com/3xjeqt)
> > 
> > Sounds like it ought to be fairly reproducible, but it's not happening on
> > Fedora 8 (and it still sounds suspiciously like the shell is getting duff
> > information about the environment, though that's certainly not the only
> > possibility).  I've tried adding multibyte characters to the command line
> > and complicating the RPROMPT and even adding multibyte characters to that,
> > but it still works OK.
> > 
> > Are there any particular things on the command line, forms of RPROMPT etc.
> > etc. that show this up?  (We really need something that narrows this down.)
> > 
> 
> I can reproduce it on Mandriva cooker with locale en_US.UTF-8 or ru_RU.UTF-8. 
It 
> does not matter whether there are UTF-8 characters on the screen (at least, I 
am 
> not sure whether there are - en_US.UTF-8 should not emit any non-ASCII strings 
> as far as I can tell).
> 

Actually the problem seems to be that terminal description contains non-UTF-8 
conform characters.

{pts/0}% infocmp | grep acsc
        acsc=``aaffggiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~,
{pts/0}% infocmp linux | grep acsc
        acsc=+\020\,
\021-\030.^Y0\333`\004a\261f\370g\361h\260i\316j\331k\277l\332m\300n\305o~p\304q\304r\304s_t\303u\264v\301w\302x\263y\363z\362
{\343|\330}\234~\376,

the first one is for dtterm, the second - for linux console.

The prompt is using q, l, m, j, k - at least some of them have high bit set.

So something gets confused computing prompt width. I am not really sure
how to fix it except teaching zsh about ACS mode. Is there any curses function
that would return screen width of character string?


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Wael Nasreddine]

[-- Attachment #1: Type: text/plain, Size: 3028 bytes --]

This One Time, at Band Camp, Andrey Borzenkov <arvidjaar@newmail.ru> said, On Wed, Feb 13, 2008 at 10:01:21PM +0300:
> On Wednesday 13 February 2008, Andrey Borzenkov wrote:
> > On Monday 11 February 2008, Peter Stephenson wrote:

> > > On Mon, 11 Feb 2008 16:07:09 +0100
> > > antho.charles@gmail.com wrote:
> > > > I didn't have this problem on Debian etch, but it appears when I
> > > > upgraded to lenny. It's the same problem: if I set LANG to a non utf8
> > > > encoding (fr_FR instead of fr_FR.UTF-8), RPROMPT is good, otherwise
> > > > it's partially on another line and the cusor is after RPROMPT.
> > > > (cf. http://tinyurl.com/3xjeqt)

> > > Sounds like it ought to be fairly reproducible, but it's not happening on
> > > Fedora 8 (and it still sounds suspiciously like the shell is getting duff
> > > information about the environment, though that's certainly not the only
> > > possibility).  I've tried adding multibyte characters to the command line
> > > and complicating the RPROMPT and even adding multibyte characters to that,
> > > but it still works OK.

> > > Are there any particular things on the command line, forms of RPROMPT etc.
> > > etc. that show this up?  (We really need something that narrows this down.)


> > I can reproduce it on Mandriva cooker with locale en_US.UTF-8 or ru_RU.UTF-8. 
> It 
> > does not matter whether there are UTF-8 characters on the screen (at least, I 
> am 
> > not sure whether there are - en_US.UTF-8 should not emit any non-ASCII strings 
> > as far as I can tell).


> Actually the problem seems to be that terminal description contains non-UTF-8 
> conform characters.

> {pts/0}% infocmp | grep acsc
>         acsc=``aaffggiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~,
> {pts/0}% infocmp linux | grep acsc
>         acsc=+\020\,
> \021-\030.^Y0\333`\004a\261f\370g\361h\260i\316j\331k\277l\332m\300n\305o~p\304q\304r\304s_t\303u\264v\301w\302x\263y\363z\362
> {\343|\330}\234~\376,

> the first one is for dtterm, the second - for linux console.

> The prompt is using q, l, m, j, k - at least some of them have high bit set.

> So something gets confused computing prompt width. I am not really sure
> how to fix it except teaching zsh about ACS mode. Is there any curses function
> that would return screen width of character string?

Maybe that's the reason, but changing TERM has no effect whatsoever,
see, whatever TERM is it just never works, but strangely, GNU Screen
works perfectly, New screen sessions and attached ones, My TERM in GNU
Screen is 'screen-256color'...

@Peter have you tried my envirenment on your Fedora install? I tried
installing fedora but it kickstart doesn't support LVM over DM-CRYPT
setup, that'll be fixed in Fedora 9 AFAIK...

-- 
Wael Nasreddine
http://wael.nasreddine.com
PGP: 1024D/C8DD18A2 06F6 1622 4BC8 4CEB D724  DE12 5565 3945 C8DD 18A2

.: An infinite number of monkeys typing into GNU emacs,
   would never make a good program. (L. Torvalds 1995) :.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 2907 bytes --]

On Thursday 14 February 2008, Wael Nasreddine wrote:
> This One Time, at Band Camp, Andrey Borzenkov <arvidjaar@newmail.ru> said, On 
Wed, Feb 13, 2008 at 10:01:21PM +0300:
> > On Wednesday 13 February 2008, Andrey Borzenkov wrote:
> > > On Monday 11 February 2008, Peter Stephenson wrote:
> 
> > > > On Mon, 11 Feb 2008 16:07:09 +0100
> > > > antho.charles@gmail.com wrote:
> > > > > I didn't have this problem on Debian etch, but it appears when I
> > > > > upgraded to lenny. It's the same problem: if I set LANG to a non utf8
> > > > > encoding (fr_FR instead of fr_FR.UTF-8), RPROMPT is good, otherwise
> > > > > it's partially on another line and the cusor is after RPROMPT.
> > > > > (cf. http://tinyurl.com/3xjeqt)
> 
> > > > Sounds like it ought to be fairly reproducible, but it's not happening 
on
> > > > Fedora 8 (and it still sounds suspiciously like the shell is getting 
duff
> > > > information about the environment, though that's certainly not the only
> > > > possibility).  I've tried adding multibyte characters to the command 
line
> > > > and complicating the RPROMPT and even adding multibyte characters to 
that,
> > > > but it still works OK.
> 
> > > > Are there any particular things on the command line, forms of RPROMPT 
etc.
> > > > etc. that show this up?  (We really need something that narrows this 
down.)
> 
> 
> > > I can reproduce it on Mandriva cooker with locale en_US.UTF-8 or 
ru_RU.UTF-8. 
> > It 
> > > does not matter whether there are UTF-8 characters on the screen (at 
least, I 
> > am 
> > > not sure whether there are - en_US.UTF-8 should not emit any non-ASCII 
strings 
> > > as far as I can tell).
> 
> 
> > Actually the problem seems to be that terminal description contains 
non-UTF-8 
> > conform characters.
> 
> > {pts/0}% infocmp | grep acsc
> >         acsc=``aaffggiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~,
> > {pts/0}% infocmp linux | grep acsc
> >         acsc=+\020\,
> > 
\021-\030.^Y0\333`\004a\261f\370g\361h\260i\316j\331k\277l\332m\300n\305o~p\304q\304r\304s_t\303u\264v\301w\302x\263y\363z\362
> > {\343|\330}\234~\376,
> 
> > the first one is for dtterm, the second - for linux console.
> 
> > The prompt is using q, l, m, j, k - at least some of them have high bit set.
> 
> > So something gets confused computing prompt width. I am not really sure
> > how to fix it except teaching zsh about ACS mode. Is there any curses 
function
> > that would return screen width of character string?
> 
> Maybe that's the reason, but changing TERM has no effect whatsoever,

But replacing all ACS characters with space has:

    PR_SET_CHARSET=""
    PR_SHIFT_IN=""
    PR_SHIFT_OUT=""
    PR_HBAR=' '
    PR_ULCORNER=' '
    PR_LLCORNER=' '
    PR_LRCORNER=' '
    PR_URCORNER=' '

If you have time you could try them one by one to see which one(s) break prompt.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Wael Nasreddine]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

This One Time, at Band Camp, Andrey Borzenkov <arvidjaar@newmail.ru> said, On Thu, Feb 14, 2008 at 06:54:36AM +0300:
> > Maybe that's the reason, but changing TERM has no effect whatsoever,

> But replacing all ACS characters with space has:

>     PR_SET_CHARSET=""
>     PR_SHIFT_IN=""
>     PR_SHIFT_OUT=""
>     PR_HBAR=' '
>     PR_ULCORNER=' '
>     PR_LLCORNER=' '
>     PR_LRCORNER=' '
>     PR_URCORNER=' '

> If you have time you could try them one by one to see which one(s) break prompt.
I tried them one by one, only PR_HBAR makes it a little better, here's
a screenshot[1] of the prompt with PR_HBAR=' '

[1]: http://img184.imageshack.us/img184/4779/zshutf8qz9.png


-- 
Wael Nasreddine
http://wael.nasreddine.com
PGP: 1024D/C8DD18A2 06F6 1622 4BC8 4CEB D724  DE12 5565 3945 C8DD 18A2

.: An infinite number of monkeys typing into GNU emacs,
   would never make a good program. (L. Torvalds 1995) :.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Wael Nasreddine]

[-- Attachment #1: Type: text/plain, Size: 2469 bytes --]

This One Time, at Band Camp, Wael Nasreddine <mla@nasreddine.com> said, On Mon, Feb 11, 2008 at 09:57:21PM +0100:
> On Mon, Feb 11, 2008 at 03:52:00PM +0000, Peter Stephenson <pws@csr.com> wrote:
> > On Mon, 11 Feb 2008 16:07:09 +0100
> > antho.charles@gmail.com wrote:
> > > I didn't have this problem on Debian etch, but it appears when I
> > > upgraded to lenny. It's the same problem: if I set LANG to a non utf8
> > > encoding (fr_FR instead of fr_FR.UTF-8), RPROMPT is good, otherwise
> > > it's partially on another line and the cusor is after RPROMPT.
> > > (cf. http://tinyurl.com/3xjeqt)

> > Sounds like it ought to be fairly reproducible, but it's not happening on
> > Fedora 8 (and it still sounds suspiciously like the shell is getting duff
> > information about the environment, though that's certainly not the only
> > possibility).  I've tried adding multibyte characters to the command line
> > and complicating the RPROMPT and even adding multibyte characters to that,
> > but it still works OK.

> > Are there any particular things on the command line, forms of RPROMPT etc.
> > etc. that show this up?  (We really need something that narrows this down.)

> I haven't wrote this PROMPT I simply found it on google[1], it's an
> exact copy of it though I have disabled APM/IBAM support beause
> sometimes ACPI hangs on this computer and so all my shells hang.

> Anyway I'll try it on Fedora and see how it goes, If I had the same
> result then it's probably something in my environment, in that case
> could you please try it with my whole environment?? Just add a new
> user (test user) and do the following:

> $ cd ~
> $ svn co svn://wael.nasreddine.com/wael/trunk/co-base .

> This should check out my basic configurations, everything related to
> the shells/environment...

> Then create all the symlinks..

> $ ~/bin/svnfix

> This would put all files where they should have been..

> Now you could test with my environment and this way we can narrow it
> down to System/Environment problems...

Peter I couldn't install Fedora because it doesn't work with LVM over
DM-Crypt, have you tried my environment ??


> [1]: http://aperiodic.net/phil/prompt/

-- 
Wael Nasreddine
http://wael.nasreddine.com
PGP: 1024D/C8DD18A2 06F6 1622 4BC8 4CEB D724  DE12 5565 3945 C8DD 18A2

.: An infinite number of monkeys typing into GNU emacs,
   would never make a good program. (L. Torvalds 1995) :.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Peter Stephenson]

Wael Nasreddine wrote:
> Peter I couldn't install Fedora because it doesn't work with LVM over
> DM-Crypt, have you tried my environment ??

No, it seems unlikely I'm going to have time for that sort of
time-consuming procedure which is any case speculative.  It seems like
the next step is understanding the implications of Andrei's findings
since he's already narrowed it down.  I don't currently know anything
about the system he's talking about.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: Phil's prompt is not working when LANG is set to UTF-8

[Wael Nasreddine]

[-- Attachment #1: Type: text/plain, Size: 1449 bytes --]

This One Time, at Band Camp, Peter Stephenson <pws@csr.com> said, On Thu, Feb 14, 2008 at 01:00:52PM +0000:
> Wael Nasreddine wrote:
> > Peter I couldn't install Fedora because it doesn't work with LVM over
> > DM-Crypt, have you tried my environment ??

> No, it seems unlikely I'm going to have time for that sort of
> time-consuming procedure which is any case speculative.  It seems like
> the next step is understanding the implications of Andrei's findings
> since he's already narrowed it down.  I don't currently know anything
> about the system he's talking about.

Maybe this problem I am having is actually related to another, very
irritating, problem and like the problem previously mentioned I don't
understand why it's happening... The below screenshot[1] is when I run
my Gnome session from GDM it doesn't happen when I ran it from usual
startx command, I really don't get it, what would be different when I
start my Gnome session from GDM or from startx ??

Take a look at this screenshot[1], it speaks for itself...

I tried what Andrei has found and Actually the terminfo is returning
weird values...

[1]: http://img529.imageshack.us/img529/7752/screenshotiy1.png

-- 
Wael Nasreddine
http://wael.nasreddine.com
PGP: 1024D/C8DD18A2 06F6 1622 4BC8 4CEB D724  DE12 5565 3945 C8DD 18A2

.: An infinite number of monkeys typing into GNU emacs,
   would never make a good program. (L. Torvalds 1995) :.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Phil's prompt is not working when LANG is set to UTF-8

[Peter Stephenson]

Wael Nasreddine wrote:
> Maybe this problem I am having is actually related to another, very
> irritating, problem and like the problem previously mentioned I don't
> understand why it's happening... The below screenshot[1] is when I run
> my Gnome session from GDM it doesn't happen when I ran it from usual
> startx command, I really don't get it, what would be different when I
> start my Gnome session from GDM or from startx ??
> 
> Take a look at this screenshot[1], it speaks for itself...

Presumably you mean one of the non-ASCII characters is showing up
correctly in one window but different ways in other windows.

Assuming they really are the same character, which you can check e.g. by
looking at the characters in your prompts:

print -P $PS1 | xxd

then there's no obvious sign here this is due to the shell, since everything
appears to be lined up correctly.  You probably need to check the
settings for the terminals in the different windows, in particular that
they're using the same character sets.  Different emulators handle this
differently: in xterm, you need to tell it the enconding as part of the
font, while gnome-terminal and others with menu-driven configuration
will try to combine the locale with the typeface you select.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070