backreferences

backreferences

[Ray Andrews]

I'm not sure if this is even a good question:

     test2 ()

    {
    sstring="before inside after"
    if [[ "$sstring" = (#b)([^i]#inside)(*) ]];

    then
       echo "\n\ndo nothing\n\n"
    fi

    echo one $match[1]
    echo two $match[2]
    }


... all good.  But is is possible to populate 'match'
without the 'if' test? Point being that I don't really
want to 'do anything' except populate 'match'.  There's
no problem, I'm just wondering if it can be expressed
more simply and I'll bet it can.

Better question:

In this construction " [^i] " is it possible to use a
string rather than a character in the exclusion test?
I've found convoluted ways to do it, but is there anything
simple?

Re: backreferences

[Bart Schaefer]

On Oct 15, 11:28am, Ray Andrews wrote:
} Subject: backreferences
}
}     sstring="before inside after"
}     if [[ "$sstring" = (#b)([^i]#inside)(*) ]];
} 
} ... all good.  But is is possible to populate 'match'
} without the 'if' test?

The [[ ]] syntax is not part of the "if" syntax; "if" is followed by
a command whose exit status is tested, so [[ ]] is a command.  Thus
you can simply write

    [[ "$sstring" = (#b)([^i]#inside)(*) ]]

by itself.  You do need the [[ ]] context to invoke pattern matching.

} Better question:
} 
} In this construction " [^i] " is it possible to use a
} string rather than a character in the exclusion test?

I'm not sure what you're asking.  In a pattern, [string] is a character
class (match any of s,t,r,i,n,g) and [^string] is the inverse of that
character class, but that whole subexpression always matches only one
character in the tested string.

So "use a string rather than a character" might mean that you want to
construct a character class by writing something similar to [^$class] 
where the parameter $class needs to be expanded, or it might mean that
you're trying to not-match multiple characters in the tested string in
a certain order.  In the latter case you want ^(string), or more often
(^(string)), but you also must setopt EXTENDED_GLOB.

Re: backreferences

[Ray Andrews]

On 10/15/2015 04:16 PM, Bart Schaefer wrote:

>      [[ "$sstring" = (#b)([^i]#inside)(*) ]]
Very good, I tried a few things and missed the one that works. But I'm 
finding out that
it's dangerous w.o. the test, since 'match' will remain silently 
unchanged if the
comparison fails.
> ... or it might mean that
> you're trying to not-match multiple characters in the tested string in
> a certain order.
Yes.

> In the latter case you want ^(string), or more often
> (^(string)), but you also must setopt EXTENDED_GLOB.
Sorry for the ambiguity.  Clarity in the mind of the sender and clarity 
in the mind
of the receiver are not the same thing.  I mean that the match should 
fail not on
meeting one character (or selection of characters), but it should fail 
on meeting
a specific sequence of characters:

    test ()
    {

    sstring="abcdeedcbaabcde"
    if [[ "$sstring" = (#b)([(^(edcba))]*)(edcba)(*) ]];

    then
           echo "you have a match"
    else
         match=
    fi

    echo "one   $match[1]"
    echo "two   $match[2]"
    echo "three $match[3]"
    }


     one   abcde
     two   edcba
     three abcde


I tried to learn how to do that with sed and never did get it figured out.
zsh can give us just about most of what we want anyway.  Pretty cool.

Thanks.

Re: backreferences

[Bart Schaefer]

On Oct 15,  6:16pm, Ray Andrews wrote:
} Subject: Re: backreferences
}
}     if [[ "$sstring" = (#b)([(^(edcba))]*)(edcba)(*) ]];

Umm, no. [(^(edcba))] is still a character class (open paren, caret,
e,d,c,b,a, close paren).  Just (^(edcba)) without the square brackets.
And you have more parens then, so your $match[] indexes are wrong.

	if [[ "$sstring" = (#b)((^(edcba))*)(edcba)(*) ]]

There are 5 sets of parens, and you care about $match[1], $match[4],
and $match[5].  $match[2] is the prefix of $match[1] that was not
consumed by the middle *, and $match[3] is an empty substring of
$match[2] (because it was excluded from matching).  Count off the
open parens left to right to see this.

In fact you don't even need the middle * because (^edcba) will eat
an arbitrarily long string as long as it is not literally "edcba".
So you can reduce this to

	if [[ "$sstring" = (#b)(^edcba)(edcba)(*) ]]

and then you're back to only needing $match[1,3].

Re: backreferences

[Mikael Magnusson]

On Fri, Oct 16, 2015 at 4:30 AM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> On Oct 15,  6:16pm, Ray Andrews wrote:
> } Subject: Re: backreferences
> }
> }     if [[ "$sstring" = (#b)([(^(edcba))]*)(edcba)(*) ]];
>
> Umm, no. [(^(edcba))] is still a character class (open paren, caret,
> e,d,c,b,a, close paren).  Just (^(edcba)) without the square brackets.
> And you have more parens then, so your $match[] indexes are wrong.
>
>         if [[ "$sstring" = (#b)((^(edcba))*)(edcba)(*) ]]
>
> There are 5 sets of parens, and you care about $match[1], $match[4],
> and $match[5].  $match[2] is the prefix of $match[1] that was not
> consumed by the middle *, and $match[3] is an empty substring of
> $match[2] (because it was excluded from matching).  Count off the
> open parens left to right to see this.
>
> In fact you don't even need the middle * because (^edcba) will eat
> an arbitrarily long string as long as it is not literally "edcba".
> So you can reduce this to
>
>         if [[ "$sstring" = (#b)(^edcba)(edcba)(*) ]]
>
> and then you're back to only needing $match[1,3].

As a sidenote, (^foo)* is always useless to write, since (^foo) will
expand to the empty string, and then the * will consume anything else.
A useful way to think of (^foo) is a * that will exclude any matches
that don't match the pattern foo.

-- 
Mikael Magnusson

Re: backreferences

[Kurtis Rader]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

On Thu, Oct 15, 2015 at 9:11 PM, Mikael Magnusson <mikachu@gmail.com> wrote:
>
> As a sidenote, (^foo)* is always useless to write, since (^foo) will
> expand to the empty string, and then the * will consume anything else.
> A useful way to think of (^foo) is a * that will exclude any matches
> that don't match the pattern foo.


As a recovering Perl addict I cannot +1 this comment enough. Regular
expressions are highly addictive and dangerous. But as Mikael points out it
is extremely easy to write a regexp that is worse than worthless. Google
"regular expression negative lookahead".

Also, Google "now you have two problems". You'll find numerous articles
talking about Jamie Zawinski's observation:

"Some people, when confronted with a problem, think "I know, I'll use
regular expressions." Now they have two problems."

I wholeheartedly agree with that sentiment. Notwithstanding the fact I
still employ regular expressions every single day. The important thing
being that I avoid them outside of ad-hoc interactive searches unless I
have expended considerable thought about their correctness and failure
modes if handed malformed input.

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

Re: backreferences

[Ray Andrews]

On 10/15/2015 09:27 PM, Kurtis Rader wrote:
> On Thu, Oct 15, 2015 at 9:11 PM, Mikael Magnusson <mikachu@gmail.com> wrote:
> As a recovering Perl addict I cannot +1 this comment enough. Regular 
> expressions are highly addictive and dangerous. But as Mikael points 
> out it is extremely easy to write a regexp that is worse than worthless. 

Indeed.  When it goes wrong it can be a spectacular mess.  I could never 
abide the fact that sed doesn't have a confirm mode.  There's 'regexxer' 
that does.  Still, what else than regex is there?  Is there any avoiding it?

Re: backreferences

[Bart Schaefer]

On Oct 16,  6:11am, Mikael Magnusson wrote:
}
} As a sidenote, (^foo)* is always useless to write, since (^foo) will
} expand to the empty string, and then the * will consume anything else.

Minor correction ... (^foo) will be the empty string only if the tested
string begins with "foo".  But then * will consume the "foo", making
the (^foo) useless.  Conversely if the string does not contain "foo" at
all, (^foo) will consume all of it and * will match the empty string.

However, (^foo)(*) could be very useful with backreferences, because
(^foo) puts everything up to "foo" into $match[1], and then (*) puts
from "foo" to the end into $match[2].  Which happens to be a lot like
what Ray was trying to accomplish.

} A useful way to think of (^foo) is a * that will exclude any matches
} that don't match the pattern foo.

Yes.  There's a really long explanation of this in Etc/FAQ 3.27.

Re: backreferences

[Bart Schaefer]

On Oct 15, 10:05pm, Bart Schaefer wrote:
}
} However, (^foo)(*) could be very useful with backreferences, because
} (^foo) puts everything up to "foo" into $match[1], and then (*) puts
} from "foo" to the end into $match[2].

And now I've got two problems instead of one.

Because of course (^foo) matches "foobar" even though the latter has
"foo" as a prefix.  You need (^foo*) to do what I was describing, or
(|^foo) which makes sense only if you either know very little about
regular expressions or you realize that zsh prefers the leftmost of
"|" alternatives over the longest of alternatives.

Re: backreferences

[Ray Andrews]

On 10/15/2015 10:28 PM, Bart Schaefer wrote:
> On Oct 15, 10:05pm, Bart Schaefer wrote:
> }
> } However, (^foo)(*) could be very useful with backreferences, because
> } (^foo) puts everything up to "foo" into $match[1], and then (*) puts
> } from "foo" to the end into $match[2].
>
> And now I've got two problems instead of one.
>
> Because of course (^foo) matches "foobar" even though the latter has
> "foo" as a prefix.  You need (^foo*) to do what I was describing, or
> (|^foo) which makes sense only if you either know very little about
> regular expressions or you realize that zsh prefers the leftmost of
> "|" alternatives over the longest of alternatives.
>
I can only guess what the parsing engine for this stuff looks like
and what immortal wrote it.

Re: backreferences

[Peter Stephenson]

On Fri, 16 Oct 2015 06:11:16 +0200
Mikael Magnusson <mikachu@gmail.com> wrote:
> As a sidenote, (^foo)* is always useless to write, since (^foo) will
> expand to the empty string, and then the * will consume anything else.
> A useful way to think of (^foo) is a * that will exclude any matches
> that don't match the pattern foo.

The warnings about the dangers of pattern matching elsewhere are
useful, though it's perhaps worth pointing out here that if what you
mean is a string of three characters that are not the characters f, o,
o, zsh allows you to do exactly this:

  (???~foo)

pws

Re: backreferences

[Ray Andrews]

On 10/15/2015 07:30 PM, Bart Schaefer wrote:
> On Oct 15,  6:16pm, Ray Andrews wrote:
> } Subject: Re: backreferences
> }
> }     if [[ "$sstring" = (#b)([(^(edcba))]*)(edcba)(*) ]];
>
> Umm, no.
But ... but ... it worked.  So it works for the wrong reason then you 
say.  Ok
the right answer for the wrong reason could do a great deal of damage so
thanks for rescuing me.

> [(^(edcba))] is still a character class (open paren, caret,
> e,d,c,b,a, close paren).  Just (^(edcba)) without the square brackets.
> And you have more parens then, so your $match[] indexes are wrong.
>
> 	if [[ "$sstring" = (#b)((^(edcba))*)(edcba)(*) ]]
>
> There are 5 sets of parens, and you care about $match[1], $match[4],
> and $match[5].  $match[2] is the prefix of $match[1] that was not
> consumed by the middle *, and $match[3] is an empty substring of
> $match[2] (because it was excluded from matching).  Count off the
> open parens left to right to see this.
At the moment I'm quite confounded.  So '(b#)' is looking at every set 
of '()' even
when nested and even when 'doing something else'? (That is to say even when
they are doing other syntax work?)  I may be doing something wrong but using
the above my output is:

one   abcde
two   edcba
three abcde
four
five


> In fact you don't even need the middle * because (^edcba) will eat
> an arbitrarily long string as long as it is not literally "edcba".
Well, that's the original question.

> So you can reduce this to
>
> 	if [[ "$sstring" = (#b)(^edcba)(edcba)(*) ]]
>
> and then you're back to only needing $match[1,3].
>
God knows.  But your simplified command works fine too, and I'll
take it on faith.  I've never seen any sort of 'any number of characters'
sort  of thing look other than:
[....]*
... so you can see where I'd go astray there.  Ok, so
^(edcba)
is individual character matches and
(^edcba)
is  anything up to "edcba" ... which is exactly what I wanted. Let's leave
my other effort to the devil--I don't even want to understand it.

Re: backreferences

[Bart Schaefer]

On Oct 15, 10:36pm, Ray Andrews wrote:
}
} > 	if [[ "$sstring" = (#b)((^(edcba))*)(edcba)(*) ]]
} >
} > There are 5 sets of parens, and you care about $match[1], $match[4],
} > and $match[5].  $match[2] is the prefix of $match[1] that was not
} > consumed by the middle *, and $match[3] is an empty substring of
} > $match[2] (because it was excluded from matching).  Count off the
} > open parens left to right to see this.
} At the moment I'm quite confounded.  So '(b#)' is looking at every set 
} of '()' even
} when nested and even when 'doing something else'?

Yes.

} > In fact you don't even need the middle * because (^edcba) will eat
} > an arbitrarily long string as long as it is not literally "edcba".
} Well, that's the original question.

As noted in my follow-up mail, what I wrote there is actually not right;
the part after "because" is correct, but the part about not needing the
middle * is wrong, because (^edcba) matches xxxxedcbaxxxx just fine, and
I assume you don't want that.

} > So you can reduce this to
} >
} > 	if [[ "$sstring" = (#b)(^edcba)(edcba)(*) ]]

This needs to be (#b)(^edcba*)(edcba)(*)

} God knows.  But your simplified command works fine too, and I'll
} take it on faith.  I've never seen any sort of 'any number of characters'
} sort  of thing look other than:
} [....]*

No, now you're confusing grep-style regular expressions with zsh patterns.
    EGREP	ZSH
    .		?
    .*		* or ?#
    .+		?##
    .?		(?|)
    [xyz]	[xyz]
    [xyz]*	[xyz]#

There's a lot more but those are the most important bits.

} ... so you can see where I'd go astray there.  Ok, so
} ^(edcba)
} is individual character matches and
} (^edcba)
} is  anything up to "edcba"

No.  [^edcba] is individual character matches and (^edcba) is anything
other than the literal string edcba, including longer strings that have
edcba as a substring.  Negated patterns are really tricky.

Re: backreferences

[Ray Andrews]

On 10/16/2015 05:35 AM, Bart Schaefer wrote:
> } when nested and even when 'doing something else'?
>
> Yes.

Ok, good to know.  I've only been using backreferences since two days 
ago and up till now it's seemed that you had to add parentheses to 
create the reference and existing parentheses only had their existing 
syntax.  I suppose this means that when you do have existing parentheses 
you'll get a 'match' whether you want one or not but so what, just  
ignore it.  Yup, that's best.
> ... but the part about not needing the
> middle * is wrong, because (^edcba) matches xxxxedcbaxxxx just fine, and
> I assume you don't want that.

    test2 ()
    {
    match=
    sstring="abcdeedcbaabcde"
    # Bart doesn't like:
    #if [[ "$sstring" = (#b)([(^(edcba))]*)(edcba)(*) ]];
    # Bart likes:
    if [[ "$sstring" = (#b)(^edcba)(edcba)(*) ]];

    then
       echo "\nIt's a poyfect match\n"
    fi

    echo "one   $match[1]"
    echo "two   $match[2]"
    echo "three $match[3]"
    echo "four  $match[4]"
    echo "five  $match[5]"
    }

    It's a poyfect match

    one   abcde
    two   edcba
    three abcde
    four
    five

... match[1]  seems to agree with your previous interpretation, no?


> This needs to be (#b)(^edcba*)(edcba)(*)

That produces identical output as well, so what's the diff? Probably one 
of those things that blows up in your face one day ...
> } God knows.  But your simplified command works fine too, and I'll
> } take it on faith.  I've never seen any sort of 'any number of characters'
> } sort  of thing look other than:
> } [....]*
>
> No, now you're confusing grep-style regular expressions with zsh patterns.

... which is what I meant to say.
>      EGREP	ZSH
>      .		?
>      .*		* or ?#
>      .+		?##
>      .?		(?|)
>      [xyz]	[xyz]
>      [xyz]*	[xyz]#
>
> There's a lot more but those are the most important bits.

A table like that is worth tattooing onto one's arm.  Seriously once a 
fella has learned a bit of regex it becomes burnt into the brain, and 
it's an act of deliberation to use the other syntax.  It sorta makes it 
worse that they are similar :(  Is a complete table available somewhere?
> } ... so you can see where I'd go astray there.  Ok, so
> } ^(edcba)
> } is individual character matches and
> } (^edcba)
> } is  anything up to "edcba"
>
> No.  [^edcba] is individual character matches and (^edcba) is anything
> other than the literal string edcba,

Ok, got it.  A mortal's guide to this stuff would sure be useful. All 
the docs tend to dive right in to the deep end and immediately start 
explaining all the possible obscure permutations when KSH_GLOB is set 
and it's not leap year but it IS a Friday.  Such control!  But we start 
with the basics.
> including longer strings that have
> edcba as a substring.  Negated patterns are really tricky.

Na.

Re: backreferences

[Bart Schaefer]

On Fri, Oct 16, 2015 at 9:37 AM, Ray Andrews <rayandrews@eastlink.ca> wrote:
> On 10/16/2015 05:35 AM, Bart Schaefer wrote:
>>
>    test2 ()
>    {
>    match=
>    sstring="abcdeedcbaabcde"
>    # Bart doesn't like:
>    #if [[ "$sstring" = (#b)([(^(edcba))]*)(edcba)(*) ]];
>    # Bart likes:
>    if [[ "$sstring" = (#b)(^edcba)(edcba)(*) ]];
>
>    then
>       echo "\nIt's a poyfect match\n"
>    fi
>
>    echo "one   $match[1]"
>    echo "two   $match[2]"
>    echo "three $match[3]"
>    echo "four  $match[4]"
>    echo "five  $match[5]"
>    }
>
>    It's a poyfect match
>
>    one   abcde
>    two   edcba
>    three abcde
>    four
>    five
>
> ... match[1]  seems to agree with your previous interpretation, no?

Only seems to.  Try sstring="edcbaabcdeedcbaabcde" and I don't think
you'll get the result you expected.

>> This needs to be (#b)(^edcba*)(edcba)(*)
>
> That produces identical output as well, so what's the diff? Probably one of
> those things that blows up in your face one day ...

Something like that , yeah.

>>      EGREP      ZSH
>
> Is a complete table available somewhere?

There might be something in the Bash to Zsh book -- I don't have it
handy to confirm.

Re: backreferences

[Ray Andrews]

On 10/16/2015 08:33 PM, Bart Schaefer wrote:

> Only seems to. Try sstring="edcbaabcdeedcbaabcde" and I don't think 
> you'll get the result you expected. 

Correct.  It fails if the very start of the string is the negation, 
which seems strange.  Any change at all to the 'bait' leading 'edcba' 
fixes it.  The empty string is the intuitive expectation for match[1].
>>> This needs to be (#b)(^edcba*)(edcba)(*)
... and that does indeed seem to cover every situation that I've tried.  
Am I correct in reading the '*' as saying: "Anything OR NOTHING in front 
of 'edcba' "?  That's not hard to fathom.  Anyway I guess the difference 
is something that could be put to use in some conceivable situation. BTW 
does sed have that functionality?  I did quite a bit of research and 
came up with nothing.  Even at StackExchange no one had an answer, but 
it seems to me like a very fundamental sort of thing.  Cool if zsh can 
do something that sed can't tho you'd hardly expect that to be the case.
>> There might be something in the Bash to Zsh book -- I don't have it 
>> handy to confirm. 
Nope, nothing that I can find.