BUG? - 4.0.2 - parameter substitution won't double backslashes in values

BUG? - 4.0.2 - parameter substitution won't double backslashes in values

[Derek Peschel]

I want to write the elements of $dirstack out to a file, separated by
newlines.  If an element in $dirstack contains a newline, I want to write
a backslash before the newline in the file.  Parameter substitution
managed that:

print ${dirstack[0]/
/\\\\
}

But I also want to double any backslashes in $dirstack, and I haven't
managed to do that yet with parameter substitution.  The backslash
sequences in the parameter seem to be interpreted before substitution
happens.

Suppose I have a subdirectory "a\bc" under my home directory.

print $dirstack[0]
/usr/home/dpeschel/ac
["\b" is a backspace]

print ${dirstack[0]/\\/\\\\}
/usr/home/dpeschel/ac
[no change]

print ${dirstack[0]/\\\\/\\\\\\\\}
/usr/home/dpeschel/ac
[no change]

print ${dirstack[0]/b/t}
/usr/home/dpeschel/a	c
[the "\b" gets changed to "\t" which is a tab]

Then I thought of using a single backslash -- given that escape sequences
happen "at a lower level" than parameter substitution, and parameter-
substitution backslashes must be quoted, it makes a kind of sense that
an unquoted backslash would affect the version of the value with un-
processed escape sequences.

The result was really weird:
print ${dirstack[0]/\/t}
t/usr/home/dpeschel/ac

If there's another way to do this, aside from parameter substitution,
that would be OK.

I haven't yet tackled reading the items back from the file, but obviously
writing has to work first.

-- Derek

Re: BUG? - 4.0.2 - parameter substitution won't double backslashes in values

[Peter Stephenson]

Derek Peschel wrote:
> I want to write the elements of $dirstack out to a file, separated by
> newlines.  If an element in $dirstack contains a newline, I want to write
> a backslash before the newline in the file.  Parameter substitution
> managed that:
> 
> print ${dirstack[0]/
> /\\\\
> }
>...

I haven't looked at this in detail, but you might be coming a bit unstuck
by using `print' instead of `print -r'.  The latter will avoid interpreting
the backslashes at that point, which is almost certainly closer to what you
want.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 392070


**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************

Re: BUG? - 4.0.2 - parameter substitution won't double backslashes in values

[Derek Peschel]

On Thu, Feb 07, 2002 at 10:33:55AM +0000, Peter Stephenson wrote:
> Derek Peschel wrote:

> > I want to write the elements of $dirstack out to a file, separated by
> > newlines.  If an element in $dirstack contains a newline, I want to write
> > a backslash before the newline in the file.  Parameter substitution
> > managed that:
> > 
> > print ${dirstack[0]/
> > /\\\\
> > }
> >...
> 
> I haven't looked at this in detail, but you might be coming a bit unstuck
> by using `print' instead of `print -r'.  The latter will avoid interpreting
> the backslashes at that point, which is almost certainly closer to what you
> want.

Good advice.  I did have to read the sentence a couple of times to be
absolutely sure you were telling me to use "print -r".

But I would still expect some change (a backspace should change to "\b" when
using "print", or "\b" should change to "\\b" when using "print -r").
And there is absolutely none as far as I can tell.  And I think that's odd.

-- Derek

Re: BUG? - 4.0.2 - parameter substitution won't double backslashes in values

[Bart Schaefer]

On Thu, 7 Feb 2002, Derek Peschel wrote:

> But I would still expect some change (a backspace should change to "\b" when
> using "print", or "\b" should change to "\\b" when using "print -r").

Um, no.  With "print" a "\b" changes to backspace; with "print -r", "\b"
remains "\b".  In either case a literal backspace remains a backspace.

If you want to convert backspace and other "control characters" to a
visible representation, you can use the (V) parameter flag:

zsh% bs=$(print -n '\b')
zsh% print ${(V)bs}
^H

Note that the conversion from '\b' to backspace was done by print, and the
conversion from backspace to ^H was done by parameter expansion.  There is
no built-in mechanism to convert from backspace to '\b'.

Re: BUG? - 4.0.2 - parameter substitution won't double backslashes in values

[Derek Peschel]

On Thu, Feb 07, 2002 at 07:20:38PM +0000, Bart Schaefer wrote:
> On Thu, 7 Feb 2002, Derek Peschel wrote:

> > But I would still expect some change (a backspace should change to "\b" when
> > using "print", or "\b" should change to "\\b" when using "print -r").
> 
> Um, no.  With "print" a "\b" changes to backspace; with "print -r", "\b"
> remains "\b".  In either case a literal backspace remains a backspace.
> 
> If you want to convert backspace and other "control characters" to a
> visible representation, you can use the (V) parameter flag:
> 
> zsh% bs=$(print -n '\b')
> zsh% print ${(V)bs}
> ^H
> 
> Note that the conversion from '\b' to backspace was done by print, and the
> conversion from backspace to ^H was done by parameter expansion.  There is
> no built-in mechanism to convert from backspace to '\b'.

Maybe my explanation was too complicated, or probably you missed the
beginning of the thread.

I have a string containing the characters "a", backslash, "b", "c".
When I print it using "print", it appears as "ac" (because the backslash,
"b" gets converted to backspace).  When I print it using "print -r",
it appears as "a\bc".

I want to use parameter substitution to convert the backslash to two
backslashes.  I haven't managed it yet -- that's what I was referring
to when I wrote "but I would still expect some change".

Assuming I did manage it, I would have the five characters "a", backslash,
backslash, "b", "c".  If I printed _them_ using "print", I would expect
to see "a\bc" and if I printed them using "print -r" I would expect to
see "a\\bc".

My first post has all the examples in it.  Please see that one.

-- Derek

Re: BUG? - 4.0.2 - parameter substitution won't double backslashes in values

[Bart Schaefer]

On Thu, 7 Feb 2002, Derek Peschel wrote:

> Maybe my explanation was too complicated, or probably you missed the
> beginning of the thread.

I saw the beginning of the thread, and I saw Sven's answer, which didn't
seem to bear repeating, so I was responding only to the parenthetical
comment about backspace changing to "\b".  As Sven's answer apparently
does bear repeating:

> I have a string containing the characters "a", backslash, "b", "c".
[...]
> I want to use parameter substitution to convert the backslash to two
> backslashes.

You *probably* want the (q) parameter flag:

zsh% x='a\bc'
zsh% print ${(q)x}
a\bc
zsh% print -r ${(q)x}
a\\bc

However, (q) will also insert a backslash in front of any other character
that is special to the shell parser.  If you want *only* to double all the
backslashes, you need one of:

zsh% print -r ${x//\\\/\\\\}
a\\bc
zsh% print -r ${x:gs/\\/\\\\\\\\}
a\\bc

The reason you need three backslashes as the pattern in the first case is
rather complicated and could possibly be considered a bug; it has to do
with using glob-pattern interpretation in ${x//...}.  The reason you need
eight backslashes as the replacement in the second case is a lot easier to
explain; the eight are reduced to four by the initial parse of the shell
command line, and then reduced again to two when the :gs replacement
occurs.

The second one is probably more reliable, as it works the same even if the
expansion is enclosed in double quotes.

Re: parameter substitution won't double backslashes in values

[Derek Peschel]

On Thu, Feb 07, 2002 at 09:00:05PM +0000, Bart Schaefer wrote:
> On Thu, 7 Feb 2002, Derek Peschel wrote:

> > Maybe my explanation was too complicated, or probably you missed the
> > beginning of the thread.
> 
> I saw the beginning of the thread, and I saw Sven's answer, which didn't
> seem to bear repeating, so I was responding only to the parenthetical
> comment about backspace changing to "\b".  As Sven's answer apparently
> does bear repeating:

Thanks for clearing that up.   I apologize if I was a bit rude.  Sven's
answer is the most practical for my needs.

> However, (q) will also insert a backslash in front of any other character
> that is special to the shell parser.  If you want *only* to double all the
> backslashes, you need one of:
> 
> zsh% print -r ${x//\\\/\\\\}
> a\\bc
> zsh% print -r ${x:gs/\\/\\\\\\\\}
> a\\bc
> 
> The reason you need three backslashes as the pattern in the first case is
> rather complicated and could possibly be considered a bug; it has to do
> with using glob-pattern interpretation in ${x//...}.  The reason you need

I was afraid of this... parameter expansion is obviously a complicated
operation.  (I took "BUG? - " out of the subject because I haven't found
a problem that I know how to identify or fix.  But not being able to
understand the syntax is somewhat of a bug in itself.)

For example, I only have a guess as to why these examples do what they do.

> x="/usr/home/dpeschel"
> print ${(q)x//t}
/usr/home/dpeschel
> print ${(q)x/\/t}
t/usr/home/dpeschel

The first example has "//" (as in global search-and-replace) followed by
a search pattern of "t", without the "/" to delimit the replacement pattern
or the pattern itself.

The second example has "/" (as in non-global search-and-replace) followed
by an empty search pattern, then "\/" which ends the search pattern,
then a replacement pattern of "t".

Is this true?

-- Derek

Re: parameter substitution won't double backslashes in values

[Bart Schaefer]

On Fri, 8 Feb 2002, Derek Peschel wrote:

> I was afraid of this... parameter expansion is obviously a complicated
> operation.

Zsh's very own 12-step program.

> For example, I only have a guess as to why these examples do what they do.
>
> > x="/usr/home/dpeschel"
> > print ${(q)x//t}
> /usr/home/dpeschel
> > print ${(q)x/\/t}
> t/usr/home/dpeschel
>
> The first example has "//" (as in global search-and-replace) followed by
> a search pattern of "t", without the "/" to delimit the replacement pattern
> or the pattern itself.

Right.  Try it with "e" instead, you get `/usr/hom/dpschl'.  By the way,
the (q) there isn't doing anything useful (in this specific example).

> The second example has "/" (as in non-global search-and-replace) followed
> by an empty search pattern, then "\/" which ends the search pattern,
> then a replacement pattern of "t".
>
> Is this true?

Yes.  To match an actual slash, you need ${x/\\//t} or ${x//\\//t} to do
it globally.

In ${x/\/t} the first parse of the outer ${...} gives something like "x
slash escaped-slash t".  The escaped-slash ends the pattern because, if
you'd put it in double quotes like "${x/a/b}" the double-quotes will also
cause the second slash to appear to be an escaped-slash, so the parser
compensates by ending the pattern at any escaped-slash.  This is where the
fun begins.

In ${x/\\//t} the parse is "x slash backslash slash slash t" (the double
backslash becomes a single backslash) so *now* when the pattern is parsed
the backslash+slash become a slash and the third slash ends the pattern.

So in ${x//\\\/t}, the triple backslash makes it through as a single
backslash, but in ${x//\\\\/t} the third slash gets escaped and becomes
part of the pattern, which is completely not what you expect.  Further, if
you take *either* the triple-backslash form *or* the quadruple, and put it
in double quotes, the third slash is again part of the pattern -- which
means that in double quotes there's no way to cause a backslash to be part
of the pattern (which is what I think is a bug), except to do this trick:

y='\'
print "${x/$y/t}"

That works because the $y is not expanded until after the end of the
pattern have been identified.

It gets even stranger if we start discussing how backslashes in the
replacement affect the parsing of the closing curly-brace, but I don't
have time to go there.

Re: BUG? - 4.0.2 - parameter substitution won't double backslashes in values

[Sven Wischnowsky]

Derek Peschel wrote:

> I want to write the elements of $dirstack out to a file, separated by
> newlines.  If an element in $dirstack contains a newline, I want to write
> a backslash before the newline in the file.  Parameter substitution
> managed that:

If what you're really after is being able to dump the value of
$dirstack to a file to be able to restore it from that file later, you
could just use some trickery:

  dump_dirstack() {
    print -lr 'dirstack=(' "${(q@)dirstack}" ')' > ~/.dirstack
  }

  # pretty useless wrapper function...

  restore_dirstack() {
    . ~/.dirstack
  }

Or some such.

A generic function to store an array (1st arg) to a file (2nd arg)
would be:

  dump_array() {
    print -lr "$1=(" "${(P@q)1}" ')' > $2
  }

Hope that helps...


Bye
  Sven

-- 
Sven Wischnowsky                          wischnow@berkom.de