greps pipes and eval bad patterns

greps pipes and eval bad patterns

[Ray Andrews]

Gentlemen:

test1 ()
{
     gstring=" | grep \[01;34m "
     tree --du -haC | grep -Ev "^[^\[]{$levels}\[*" "$gstring"
}

That's a cut down version of my 'tree' wrapper. If $gstring is appended 
as it's literal content (as opposed to the variable itself) it works 
fine, but if I use the variable as shown , grep tries to eat it as part 
of it's own argument string. I've solved problems like that elsewhere by 
using 'eval' however it doesn't work in the above situation because eval 
complains about the contents of the first grep search:

(eval):1: bad pattern: ^[^[]{12}[*

As a point of principal can this be done?  The thing would be to stop 
$gstring from being interpreted as an argument to the first grep and I'm 
betting it's easy if you know how.  In practice my wrapper uses two 
separate lines, one with and one without the " | grep \[01;34m " (Which, 
BTW, selects directories only for display based on the color of the 
output of 'tree'--blue for directories.) But it would be elegant to use 
one line, and append $gstring (set to null or to the above depending on 
whether I want directories only as the output, or not.)

Re: greps pipes and eval bad patterns

[Bart Schaefer]

On Oct 25, 12:47pm, Ray Andrews wrote:
}
} test1 ()
} {
}      gstring=" | grep \[01;34m "
}      tree --du -haC | grep -Ev "^[^\[]{$levels}\[*" "$gstring"
} }

One doesn't normally build up a pipeline that way, but if you must do
so, you're on the right track with "eval" -- you just haven't applied
enough quoting.  "eval" is going to re-parse everything, so you need
to quote everyhing to the same depth:

    eval 'tree --du -haC | grep -Ev "^[^\[]{$levels}\[*"' "$gstring"

The single quotes (before tree and after the levels pattern) keep the
first pipeline (and importantly the double-quotes that are around the
grep pattern) from being interpreted until eval does so.  The use of
the parameter for $gstring has the same effect.

You might be able to see this better if you assign everything to
variables before eval-ing, e.g.

  test1 ()
  {
     gstring="| grep \[01;34m "
     glevels='| grep -Ev "^[^\[]{$levels}\[*"'
     tree="tree --du -haC"
     eval "$tree" $glevels" "$gstring"
  }

Re: greps pipes and eval bad patterns

[Kurtis Rader]

[-- Attachment #1: Type: text/plain, Size: 1676 bytes --]

On Sun, Oct 25, 2015 at 6:02 PM, Bart Schaefer <schaefer@brasslantern.com>
wrote:

> On Oct 25, 12:47pm, Ray Andrews wrote:
> }
> } test1 ()
> } {
> }      gstring=" | grep \[01;34m "
> }      tree --du -haC | grep -Ev "^[^\[]{$levels}\[*" "$gstring"
> } }
>
> One doesn't normally build up a pipeline that way, but if you must do
> so, you're on the right track with "eval" -- you just haven't applied
> enough quoting....


In addition to what Bart said I'll point out that it's usually easier and
safer to avoid the eval by using an if/else block. Getting the quoting
right when using eval can be very difficult. Especially if you don't have
direct control over the text being eval'd; e.g., if some of it is supplied
by the user; or even just built up from earlier parts of the function.

I have several functions where 99% of the time I want the output
automatically piped into my pager (e.g., /usr/bin/less). But if the output
of the function is redirected away from my tty I don't want the pager in
the pipeline. So instead of using a variable that would normally contain "|
$PAGER" and sometimes be empty (i.e., using Ray's approach) I simply spell
it out:

    if [[ -t 1 ]] ; then
      grep -h -E $case_insensitive -- "$search_for" $files | $PAGER
    else
      grep -h -E $case_insensitive -- "$search_for" $files
    fi

Yes, that introduces some redundancy. But it's far clearer and safer than
writing something like this (totally untested):

    if [[ -t 1 ]] ; then
      pager='| $PAGER'
    else
      pager=''
    fi
    eval grep -h -E $case_insensitive -- "$search_for" $files $pager

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

Re: greps pipes and eval bad patterns

[Ray Andrews]

On 10/25/2015 06:02 PM, Bart Schaefer wrote:
> On Oct 25, 12:47pm, Ray Andrews wrote:
> }
> } test1 ()
> } {
> }      gstring=" | grep \[01;34m "
> }      tree --du -haC | grep -Ev "^[^\[]{$levels}\[*" "$gstring"
> } }
>
> One doesn't normally build up a pipeline that way, but if you must

What would be the better way?  I'm not wedded to anything, just looking 
for the
appropriate method.
>   do
> so, you're on the right track with "eval" -- you just haven't applied
> enough quoting.  "eval" is going to re-parse everything, so you need
> to quote everyhing to the same depth:
>
>      eval 'tree --du -haC | grep -Ev "^[^\[]{$levels}\[*"' "$gstring"
>
> The single quotes (before tree and after the levels pattern) keep the
> first pipeline (and importantly the double-quotes that are around the
> grep pattern) from being interpreted until eval does so.

Enlightenment.  We freeze all expansions with single quotes until eval 
sorts it all out in
one go.

> The use of
> the parameter for $gstring has the same effect.
>
> You might be able to see this better if you assign everything to
> variables before eval-ing, e.g.
>
>    test1 ()
>    {
>       gstring="| grep \[01;34m "
>       glevels='| grep -Ev "^[^\[]{$levels}\[*"'
>       tree="tree --du -haC"
>       eval "$tree" $glevels" "$gstring"

Yeah, that's the sort of thing I'm used to doing I just didn't know how 
to handle the
tricky characters.  It makes nothing but sense now that I see it. So the 
final product
becomes:

t ()
{
     local gstring=
     [ "$1" = ',f' ] && { gstring=' | grep "\[01;34m" '; shift }
     integer levels=$(( ($1 + 1) * 4 ))
     eval ' tree --du -haC | grep -Ev "^[^\[]{$levels}\[*" ' $gstring
     du -sh .
}

... a better tree than tree.

Re: greps pipes and eval bad patterns

[Ray Andrews]

On 10/25/2015 06:17 PM, Kurtis Rader wrote:
> In addition to what Bart said I'll point out that it's usually easier and
> safer to avoid the eval by using an if/else block. Getting the quoting
> right when using eval can be very difficult. Especially if you don't have
> direct control over the text being eval'd; e.g., if some of it is supplied
> by the user; or even just built up from earlier parts of the function.
>
> I have several functions where 99% of the time I want the output
> automatically piped into my pager (e.g., /usr/bin/less). But if the output
> of the function is redirected away from my tty I don't want the pager in
> the pipeline. So instead of using a variable that would normally contain "|
> $PAGER" and sometimes be empty (i.e., using Ray's approach) I simply spell
> it out:
>
>      if [[ -t 1 ]] ; then
>        grep -h -E $case_insensitive -- "$search_for" $files | $PAGER
>      else
>        grep -h -E $case_insensitive -- "$search_for" $files
>      fi

That's exactly the sort of thing I did before in this sort of 
situation.  I take your caveats  seriously tho.  Mostly I was just 
interested in the problem for my education.  I can quite see that eval 
could do some weird stuff if the input was uncertain.  What matters most 
is that the code is clear both to read and to execute. Having a pipe 
inside a variable seems unsound.

Re: greps pipes and eval bad patterns

[ZyX]

26.10.2015, 06:32, "Ray Andrews" <rayandrews@eastlink.ca>:
> On 10/25/2015 06:02 PM, Bart Schaefer wrote:
>>  On Oct 25, 12:47pm, Ray Andrews wrote:
>>  }
>>  } test1 ()
>>  } {
>>  } gstring=" | grep \[01;34m "
>>  } tree --du -haC | grep -Ev "^[^\[]{$levels}\[*" "$gstring"
>>  } }
>>
>>  One doesn't normally build up a pipeline that way, but if you must
>
> What would be the better way? I'm not wedded to anything, just looking
> for the
> appropriate method.
>>    do
>>  so, you're on the right track with "eval" -- you just haven't applied
>>  enough quoting. "eval" is going to re-parse everything, so you need
>>  to quote everyhing to the same depth:
>>
>>       eval 'tree --du -haC | grep -Ev "^[^\[]{$levels}\[*"' "$gstring"
>>
>>  The single quotes (before tree and after the levels pattern) keep the
>>  first pipeline (and importantly the double-quotes that are around the
>>  grep pattern) from being interpreted until eval does so.
>
> Enlightenment. We freeze all expansions with single quotes until eval
> sorts it all out in
> one go.
>
>>  The use of
>>  the parameter for $gstring has the same effect.
>>
>>  You might be able to see this better if you assign everything to
>>  variables before eval-ing, e.g.
>>
>>     test1 ()
>>     {
>>        gstring="| grep \[01;34m "
>>        glevels='| grep -Ev "^[^\[]{$levels}\[*"'
>>        tree="tree --du -haC"
>>        eval "$tree" $glevels" "$gstring"
>
> Yeah, that's the sort of thing I'm used to doing I just didn't know how
> to handle the
> tricky characters. It makes nothing but sense now that I see it. So the
> final product
> becomes:
>
> t ()
> {
>      local gstring=
>      [ "$1" = ',f' ] && { gstring=' | grep "\[01;34m" '; shift }
>      integer levels=$(( ($1 + 1) * 4 ))
>      eval ' tree --du -haC | grep -Ev "^[^\[]{$levels}\[*" ' $gstring
>      du -sh .
> }

Specifically this I would write as

    local -a gcmd
    gcmd=( cat )
    if [[ $1 == ,f ]] ; then
        gcmd=( grep '\[01;34m' )
        shift
    endif
    integer levels=$(( ($1 + 1) * 4 ))
    tree --du -haC | grep -Ev '^[^\[]{'"$levels"'\[* ' | $gcmd

>
> ... a better tree than tree.

Re: greps pipes and eval bad patterns

[Ray Andrews]

On 10/26/2015 06:04 AM, ZyX wrote:
> Specifically this I would write as
>
>      local -a gcmd
>      gcmd=( cat )
>      if [[ $1 == ,f ]] ; then
>          gcmd=( grep '\[01;34m' )
>          shift
>      endif
>      integer levels=$(( ($1 + 1) * 4 ))
>      tree --du -haC | grep -Ev '^[^\[]{'"$levels"'\[* ' | $gcmd
That is better.  The pipe  shouldn't be in the variable.  But because 
you can't have nothing on the right side of the pipe, I had been using ' 
grep .* ' as the default for '$gcmd' which seemed rather clumsy.

Question: Why can't we have nothing on the right side?  Why not just 
ignore the pipe in that case?  It would obviate this situation. Logic 
being that a pipe to nowhere is a null command thus ignored. Or is there 
some other syntactic use for a pipe to nowhere?

Re: greps pipes and eval bad patterns

[Bart Schaefer]

On Oct 26,  6:36am, Ray Andrews wrote:
}
} Question: Why can't we have nothing on the right side?

Because then you have an incomplete command, syntax-wise.  You can write

    echo the command starts here |
	sed s/start/end/

Re: greps pipes and eval bad patterns

[Ray Andrews]

On 10/26/2015 07:30 AM, Bart Schaefer wrote:
> On Oct 26,  6:36am, Ray Andrews wrote:
> }
> } Question: Why can't we have nothing on the right side?
>
> Because then you have an incomplete command, syntax-wise.  You can write
>
>      echo the command starts here |
> 	sed s/start/end/
>
>
I see, the interpreter would continue on the next line. Yeah, that's a 
necessary thing without the mandatory ';' as in C.  The one exception 
that I can think of, as we've discussed, is that you can't:

echo true
&& next_command

... which is too bad.