printf %<n>s in UTF-8 is not always POSIX-compliant

printf %<n>s in UTF-8 is not always POSIX-compliant

[Vincent Lefevre]

Hi,

I've reported the following bug:

  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=659932

In UTF-8 locales:

xvii% printf ".%2s.\n" é
. é.
xvii% emulate sh
xvii% printf ".%2s.\n" é
.é.
xvii% emulate ksh       
xvii% printf ".%2s.\n" é
. é.

It is correct in sh mode (according to POSIX[*]), but not in ksh mode,
which should also follow the POSIX behavior. What about zsh mode?

[*] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html
and
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap05.html#tag_05
for %<n>s.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: printf %<n>s in UTF-8 is not always POSIX-compliant

[Bart Schaefer]

On Feb 15,  3:15am, Vincent Lefevre wrote:
}
} In UTF-8 locales:
} 
} xvii% printf ".%2s.\n" é
} .é.

Am I understanding correctly that the intent here is that é is a two-
byte character so %2s should print the two literal bytes, rather than
print the single logical character in a field two logical characters
wide?

The reason it's different for "emulate sh" is that sh emulation turns
off all support for multibyte characters (unsetopt multibyte).  If you
were to do
	emulate sh -c 'setopt multibyte; printf ".%2s.\n" é'
then I believe you'd see the same behavior as with "emulate ksh".

As to whether it's correct ... I think I'd prefer the logical rather
than literal interpretation, but it'll be difficult [or a hack that
requires looking at the global emulation state, so it won't be possible
to reproduce it with plain setopts] to turn off multibyte processing in
printf for ksh emulation but not native zsh.

Re: printf %<n>s in UTF-8 is not always POSIX-compliant

[Vincent Lefevre]

On 2012-02-15 00:14:12 -0800, Bart Schaefer wrote:
> On Feb 15,  3:15am, Vincent Lefevre wrote:
> }
> } In UTF-8 locales:
> } 
> } xvii% printf ".%2s.\n" é
> } .é.
> 
> Am I understanding correctly that the intent here is that é is a two-
> byte character so %2s should print the two literal bytes, rather than
> print the single logical character in a field two logical characters
> wide?

Yes, the number is the size in bytes, not in characters. I think
that the intent is to deal with internal structures (e.g. with
file formats where some fields have a fixed or limited size, and
the same syntax can be used in C to avoid buffer overflows).
Note that there's the same problem with:

xvii% printf ".%.3s.\n" éabcd
.éab.
xvii% emulate ksh
xvii% printf ".%.3s.\n" éabcd
.éab.
xvii% emulate sh             
xvii% printf ".%.3s.\n" éabcd
.éa.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: printf %<n>s in UTF-8 is not always POSIX-compliant

[Peter Stephenson]

On Wed, 15 Feb 2012 00:14:12 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> The reason it's different for "emulate sh" is that sh emulation turns
> off all support for multibyte characters (unsetopt multibyte).  If you
> were to do
> 	emulate sh -c 'setopt multibyte; printf ".%2s.\n" é'
> then I believe you'd see the same behavior as with "emulate ksh".
> 
> As to whether it's correct ... I think I'd prefer the logical rather
> than literal interpretation, but it'll be difficult [or a hack that
> requires looking at the global emulation state, so it won't be possible
> to reproduce it with plain setopts] to turn off multibyte processing in
> printf for ksh emulation but not native zsh.

This sounds correct... We've never promised ksh mode would be a complete
representation of ksh anyway.  I realise that, for historical reasons
related to standards rather than zsh, you'd expect ksh mode to be POSIX
compatible, but actually we don't tend to bother because ksh mode isn't
that widely used and so doesn't get a lot of attention (I certainly
never use it).  If you really want compatibility native zsh mode or sh
mode are the sensible choices.

So probably the fix is to spread fear, uncertainty and doubt about ksh
mode.  I'll start right now.

If there is a hard-core ksh mode user who'd like to maintain it, of
course, that's another story.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

Re: printf %<n>s in UTF-8 is not always POSIX-compliant

[Vincent Lefevre]

On 2012-02-15 11:05:19 +0000, Peter Stephenson wrote:
> This sounds correct... We've never promised ksh mode would be a complete
> representation of ksh anyway.  I realise that, for historical reasons
> related to standards rather than zsh, you'd expect ksh mode to be POSIX
> compatible, but actually we don't tend to bother because ksh mode isn't
> that widely used and so doesn't get a lot of attention (I certainly
> never use it).  If you really want compatibility native zsh mode or sh
> mode are the sensible choices.

The problem is that on some machines, one has a symlink ksh -> zsh.
If I type ksh or run a script with #!/usr/bin/ksh, I expect this to
behave as a real ksh.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: printf %<n>s in UTF-8 is not always POSIX-compliant

[Frank Terbeck]

Vincent Lefevre wrote:
> On 2012-02-15 11:05:19 +0000, Peter Stephenson wrote:
>> This sounds correct... We've never promised ksh mode would be a complete
>> representation of ksh anyway.
[...]
> The problem is that on some machines, one has a symlink ksh -> zsh.
> If I type ksh or run a script with #!/usr/bin/ksh, I expect this to
> behave as a real ksh.

Frankly, that would be the vendor's fault then. There are many *MANY*
ksh implementations, that make for a reasonable link target (ksh93,
pdksh or mksh - to name just a few). Zsh is not one of them.

IMHO, ksh-emulation is a little bit like csh emulation: It's meant to
make users with ksh background feel more "at home", not as a strict
bug-for-bug emulation.

Regards, Frank

Re: printf %<n>s in UTF-8 is not always POSIX-compliant

[Peter Stephenson]

On Wed, 15 Feb 2012 13:09:17 +0100
Frank Terbeck <ft@bewatermyfriend.org> wrote:
> IMHO, ksh-emulation is a little bit like csh emulation: It's meant to
> make users with ksh background feel more "at home", not as a strict
> bug-for-bug emulation.

That's basically how I see it.  It doesn't mean we can't do better --- but
I don't think we can do better by people who don't really use the mode
initiating random tweaks in the hope that the world becomes a better
place.  We really would need someone who is in a position to take a
global view of how changes to the mode affect the emulation.

This is a rather different case from POSIX emulation, where there's (i)
a standard (ii) quite a lot of visibility of what the effect of changes
are.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

Re: printf %<n>s in UTF-8 is not always POSIX-compliant

[Vincent Lefevre]

On 2012-02-15 13:09:17 +0100, Frank Terbeck wrote:
> Frankly, that would be the vendor's fault then. There are many *MANY*
> ksh implementations, that make for a reasonable link target (ksh93,
> pdksh or mksh - to name just a few). Zsh is not one of them.

OK, bug reported.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: printf %<n>s in UTF-8 is not always POSIX-compliant

[Oliver Kiddle]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

Bart wrote:
> 
> Am I understanding correctly that the intent here is that é is a two-
> byte character so %2s should print the two literal bytes, rather than
> print the single logical character in a field two logical characters
> wide?

That's correct. The POSIX definition uses bytes. For multibyte
behaviour, there is an L modifier. I don't really see the sense in it
myself: I don't want to write low-level stuff in the shell.

Frank Terbeck wrote:
> Frankly, that would be the vendor's fault then. There are many *MANY*
> ksh implementations, that make for a reasonable link target (ksh93,
> pdksh or mksh - to name just a few). Zsh is not one of them.

The fact that zsh is far from a perfect emulation doesn't stop it from
being useful. I don't necessarily want to install a separate ksh package
and zsh will run ksh scripts at least as well as pdksh.

Oliver

Re: printf %<n>s in UTF-8 is not always POSIX-compliant

[Vincent Lefevre]

On 2012-02-15 15:42:15 +0100, Oliver Kiddle wrote:
> Bart wrote:
> > Am I understanding correctly that the intent here is that ?? is a two-
> > byte character so %2s should print the two literal bytes, rather than
> > print the single logical character in a field two logical characters
> > wide?
> 
> That's correct. The POSIX definition uses bytes. For multibyte
> behaviour, there is an L modifier. I don't really see the sense in it
> myself: I don't want to write low-level stuff in the shell.

I think that's for consistency with C. Also, the shell could then
be used as a front-end to test string-related things.

> Frank Terbeck wrote:
> > Frankly, that would be the vendor's fault then. There are many *MANY*
> > ksh implementations, that make for a reasonable link target (ksh93,
> > pdksh or mksh - to name just a few). Zsh is not one of them.
> 
> The fact that zsh is far from a perfect emulation doesn't stop it from
> being useful. I don't necessarily want to install a separate ksh package
> and zsh will run ksh scripts at least as well as pdksh.

But then the emulation should be correct.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)