globbing bug, 3.0.6

globbing bug, 3.0.6

[Trond Eivind Glomsrød]

hoser% rpm -q zsh
zsh-3.0.7-4
hoser% echo $ZSH_VERSION 
3.0.7
hoser% cat /etc/redhat-release 
Red Hat Linux release 6.2 (Zoot)
hoser% ls 
AA  AC	AE  BB	BD  CA	CC  CE	ab  ad	ba  bc	be  cb	cd  da	dc  de
AB  AD	BA  BC	BE  CB	CD  aa	ac  ae	bb  bd	ca  cc	ce  db	dd
hoser% ls [A-C]*
AA  AC	AE  BB	BD  CA	CC  CE	ab  ad	ba  bc	be
AB  AD	BA  BC	BE  CB	CD  aa	ac  ae	bb  bd
hoser%

-- 
Trond Eivind Glomsrød
Red Hat, Inc.

Re: globbing bug, 3.0.6

[Zefram]

Trond Eivind=?iso-8859-1?q?_Glomsr=F8d?= wrote:
>hoser% ls [A-C]*
>AA  AC	AE  BB	BD  CA	CC  CE	ab  ad	ba  bc	be
>AB  AD	BA  BC	BE  CB	CD  aa	ac  ae	bb  bd

What do you have $LC_CTYPE, $LC_ALL and $LANG set to?  This is the usual
sign of having a locale set inappropriately -- [A-C] is being treated as
[AaBbC].

-zefram

Re: globbing bug, 3.0.6

[Trond Eivind Glomsrød]

Zefram <zefram@fysh.org> writes:

> Trond Eivind=?iso-8859-1?q?_Glomsr=F8d?= wrote:
> >hoser% ls [A-C]*
> >AA  AC	AE  BB	BD  CA	CC  CE	ab  ad	ba  bc	be
> >AB  AD	BA  BC	BE  CB	CD  aa	ac  ae	bb  bd
> 
> What do you have $LC_CTYPE, $LC_ALL and $LANG set to? 

hoser% env |grep -E "LC|LANG"
GDM_LANG=no_NO
LANG=no_NO
hoser%

> This is the usual sign of having a locale set inappropriately --
> [A-C] is being treated as [AaBbC].

It seems to do that - and when sorting sentences, it does make some
sense. I don't think it does here, though. Checking, it seems to
affect de_DE, en_US as well...


-- 
Trond Eivind Glomsrød
Red Hat, Inc.

Re: globbing bug, 3.0.6

[Chmouel Boudjnah]

teg@redhat.com (Trond Eivind Glomsrød) writes:

> hoser% rpm -q zsh
> zsh-3.0.7-4
> hoser% echo $ZSH_VERSION 
> 3.0.7
> hoser% cat /etc/redhat-release 
> Red Hat Linux release 6.2 (Zoot)
> hoser% ls 
> AA  AC	AE  BB	BD  CA	CC  CE	ab  ad	ba  bc	be  cb	cd  da	dc  de
> AB  AD	BA  BC	BE  CB	CD  aa	ac  ae	bb  bd	ca  cc	ce  db	dd
> hoser% ls [A-C]*
> AA  AC	AE  BB	BD  CA	CC  CE	ab  ad	ba  bc	be
> AB  AD	BA  BC	BE  CB	CD  aa	ac  ae	bb  bd
> hoser%

it's the POSIX standard behavior when locale setting is set. it's the
same thing for bash2 by default.

-- 
MandrakeSoft Inc                http://www.mandrakesoft.com
Pasadena, CA USA.                                 --Chmouel

Re: globbing bug, 3.0.6

[Trond Eivind Glomsrød]

Chmouel Boudjnah <chmouel@mandrakesoft.com> writes:

> teg@redhat.com (Trond Eivind Glomsrød) writes:
> 
> > hoser% rpm -q zsh
> > zsh-3.0.7-4
> > hoser% echo $ZSH_VERSION 
> > 3.0.7
> > hoser% cat /etc/redhat-release 
> > Red Hat Linux release 6.2 (Zoot)
> > hoser% ls 
> > AA  AC	AE  BB	BD  CA	CC  CE	ab  ad	ba  bc	be  cb	cd  da	dc  de
> > AB  AD	BA  BC	BE  CB	CD  aa	ac  ae	bb  bd	ca  cc	ce  db	dd
> > hoser% ls [A-C]*
> > AA  AC	AE  BB	BD  CA	CC  CE	ab  ad	ba  bc	be
> > AB  AD	BA  BC	BE  CB	CD  aa	ac  ae	bb  bd
> > hoser%
> 
> it's the POSIX standard behavior when locale setting is set.

Yes. http://wwwold.dkuug.dk/jtc1/sc22/wg20/docs/n634.txt:

************************************************************************
(1)     Once special characters (punctuation) have been removed
        from original strings, the ordering is determined by
        scanning forward (left to right) [disregarding case and
        diacriticals].
************************************************************************

And that's horribly broken IMHO. At least, I think it's broken in
shell expansion (translates to: "it's not what I'm used to, it's
different in different locales (POSIX vs. the rest, including en_US),
it makes deleting just uppercase filenames in a range hard")


-- 
Trond Eivind Glomsrød
Red Hat, Inc.

Re: globbing bug, 3.0.6

[Bart Schaefer]

On Apr 27,  3:11pm, Trond Eivind=?iso-8859-1?q?_Glomsr=F8d?= wrote:
} Subject: Re: globbing bug, 3.0.6
}
} Chmouel Boudjnah <chmouel@mandrakesoft.com> writes:
} 
} > it's the POSIX standard behavior when locale setting is set.
} 
} And that's horribly broken IMHO. At least, I think it's broken in
} shell expansion

I don't want to change this in 3.0.8 unless it's also changing in 3.1.7
or soon thereafter.  What's the verdict on this one, folks?  Apparently
bash behaves as zsh does now, but that's not a conclusive argument.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

RE: globbing bug, 3.0.6

[Andrej Borsenkow]

> On Apr 27,  3:11pm, Trond Eivind=?iso-8859-1?q?_Glomsr=F8d?= wrote:
> } Subject: Re: globbing bug, 3.0.6
> }
> } Chmouel Boudjnah <chmouel@mandrakesoft.com> writes:
> }
> } > it's the POSIX standard behavior when locale setting is set.
> }
> } And that's horribly broken IMHO. At least, I think it's broken in
> } shell expansion
>
> I don't want to change this in 3.0.8 unless it's also
> changing in 3.1.7
> or soon thereafter.  What's the verdict on this one, folks?
> Apparently
> bash behaves as zsh does now, but that's not a conclusive argument.
>


That's general question - how far is compatibility with current Unix
standards important? For better or for worse - current behaviour is one
of the requirements. The problem is (I can go into more details if
needed) Zsh in current form does not and *can* not support full i18n as
specified in docs (this applies to such obscure things as "collate
elements" e.g.). So, the ultimate question is - should we aim at full
POSIX/SUS/... compatibility, at least as an option (that may need
substantial rewrite), or drop it in favour of more "user friendliness".

Now, with much more powerful regular expressions, I even agree - for
interactive shell plain ASCII collating may be more appropriate - and
scripts should use [[:class:]] form anyway.

-andrej

Re: globbing bug, 3.0.6

[Peter Stephenson]

> I don't want to change this in 3.0.8 unless it's also changing in 3.1.7
> or soon thereafter.  What's the verdict on this one, folks?  Apparently
> bash behaves as zsh does now, but that's not a conclusive argument.

I hesitate to reply, since I seem to be missing something, but as far as
I'm concerned, zsh 3.1.7 just uses straightforward character range
comparisons (not POSIX collation) with ordinary ranges such as [A-Z], but
uses whatever the system tells it with special classes such as [[:upper:]]
(this comes straight from ctype, not the zsh type macros), and this has
been the case for the past couple of versions.  Does anyone have evidence
to the contrary?  Barring a new option, is that the best default behaviour?
Does anyone want an option POSIX_RANGES to restore full POSIX behaviour,
despite the drawbacks?

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070

Re: globbing bug, 3.0.6

[Sven Wischnowsky]

Bart Schaefer wrote:

> On Apr 27,  3:11pm, Trond Eivind=?iso-8859-1?q?_Glomsr=F8d?= wrote:
> } Subject: Re: globbing bug, 3.0.6
> }
> } Chmouel Boudjnah <chmouel@mandrakesoft.com> writes:
> } 
> } > it's the POSIX standard behavior when locale setting is set.
> } 
> } And that's horribly broken IMHO. At least, I think it's broken in
> } shell expansion
> 
> I don't want to change this in 3.0.8 unless it's also changing in 3.1.7
> or soon thereafter.  What's the verdict on this one, folks?  Apparently
> bash behaves as zsh does now, but that's not a conclusive argument.

Right, and they get their own bug reports about this (for example on
gnu.bash.bug).


Andrej Borsenkow wrote:

> That's general question - how far is compatibility with current Unix
> standards important? For better or for worse - current behaviour is one
> of the requirements. The problem is (I can go into more details if
> needed) Zsh in current form does not and *can* not support full i18n as
> specified in docs (this applies to such obscure things as "collate
> elements" e.g.). So, the ultimate question is - should we aim at full
> POSIX/SUS/... compatibility, at least as an option (that may need
> substantial rewrite), or drop it in favour of more "user friendliness".
> 
> Now, with much more powerful regular expressions, I even agree - for
> interactive shell plain ASCII collating may be more appropriate - and
> scripts should use [[:class:]] form anyway.

I'd say we should add an option that turns on POSIX/i18n behaviour and 
defaults to `off' and documented like `currently this only
affects...'. I think its a case where not conforming to standards is
the right thing, reducing the number of (probably horrible)
surprises. Then we could think about adding the match-flags to turn
on/off the behaviour determined by the option for single patterns (I
haven't had a look at this match-flag code yet so I don't know how
hard to implement this is).

And later we can make the option control more things.

This also reminds me that the classes in match specs don't care about
the locale at all...

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

RE: globbing bug, 3.0.6

[Andrej Borsenkow]

>
> This also reminds me that the classes in match specs don't care about
> the locale at all...
>

This is a bug IMHO. Just consider case insensitive completion of
localised file names ... I hope, that sometimes it won't be that
uncommon.

-andrej

Re: globbing bug, 3.0.6

[Sven Wischnowsky]

Peter Stephenson wrote:

> > I don't want to change this in 3.0.8 unless it's also changing in 3.1.7
> > or soon thereafter.  What's the verdict on this one, folks?  Apparently
> > bash behaves as zsh does now, but that's not a conclusive argument.
> 
> I hesitate to reply, since I seem to be missing something, but as far as
> I'm concerned, zsh 3.1.7 just uses straightforward character range
> comparisons (not POSIX collation) with ordinary ranges such as [A-Z], but
> uses whatever the system tells it with special classes such as [[:upper:]]
> (this comes straight from ctype, not the zsh type macros), and this has
> been the case for the past couple of versions.  Does anyone have evidence
> to the contrary? 

;-) I didn't reply to Andrej's reply to my mail because I had a look
at pattern.c after that and found the same.

And no, I've never seen that behaviour myself, but I don't use locales 
anyway, so... The only thing that's really still there is the call to
strcoll() in the sorting function in glob.c.

> Barring a new option, is that the best default behaviour?
> Does anyone want an option POSIX_RANGES to restore full POSIX behaviour,
> despite the drawbacks?

Personally I don't care about conformance to any standard (in this
respect, at least) or compatibility to other shells. I just wanted say 
that /if/ we do it, we should make it configurable and shouldn't make
it the default.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de