subsitutions and beginning of lines.

subsitutions and beginning of lines.

[Ray Andrews]

${var// /}

When doing that sort of substitution, is is possible to test for newlines?
I'm playing with a function that grabs history and I'm trying to strip
off the leading numbers.  At one point I have the output captured in a
variable and various forms of substitution come close, but the substitution
treats the entire variable as one entity whereas I want the substitution
to be performed fresh on each new line of output.  I can't find any
specific newline character.  OTOH if the variable was an array there'd
probably be some way of processing each line as a separate entity, which
would serve fine. I know I'm close, but I can't quite bag it.

Re: subsitutions and beginning of lines.

[Peter Stephenson]

On Sun, 11 Oct 2015 12:12:26 -0700
Ray Andrews <rayandrews@eastlink.ca> wrote:
> ${var// /}
> 
> When doing that sort of substitution, is is possible to test for newlines?
> I'm playing with a function that grabs history and I'm trying to strip
> off the leading numbers.  At one point I have the output captured in a
> variable and various forms of substitution come close, but the substitution
> treats the entire variable as one entity whereas I want the substitution
> to be performed fresh on each new line of output.  I can't find any
> specific newline character.  OTOH if the variable was an array there'd
> probably be some way of processing each line as a separate entity, which
> would serve fine. I know I'm close, but I can't quite bag it.

If I'm following, you are indeed nearly there: you need to split the
variable to an array on newlines, while not splitting on anything else.
To do that, you quote the array, then force the splitting which ensures it
only gets split on what you tell it and not other other random spaces:

var="1:one two
2:three four"
print -l -- ${(f)"${var}"}

The -l prints one element per line for clarity.

You've now got the lines you want as if they were array elements.  You
can then surround that substitution with the sort you first thought of.
For example, to strip the "number:",

print -l -- ${${(f)"${var}"}##<->:}

where, as you now know, <-> matches any set of digits.

pws

Re: subsitutions and beginning of lines.

[Ray Andrews]

On 10/11/2015 01:09 PM, Peter Stephenson wrote:
> On Sun, 11 Oct 2015 12:12:26 -0700
> Ray Andrews <rayandrews@eastlink.ca> wrote:
>> ${var// /}
>>
>> When doing that sort of substitution, is is possible to test for newlines?
>> I'm playing with a function that grabs history and I'm trying to strip
>> off the leading numbers.  At one point I have the output captured in a
>> variable and various forms of substitution come close, but the substitution
>> treats the entire variable as one entity whereas I want the substitution
>> to be performed fresh on each new line of output.  I can't find any
>> specific newline character.  OTOH if the variable was an array there'd
>> probably be some way of processing each line as a separate entity, which
>> would serve fine. I know I'm close, but I can't quite bag it.
> If I'm following, you are indeed nearly there: you need to split the
> variable to an array on newlines, while not splitting on anything else.
> To do that, you quote the array, then force the splitting which ensures it
> only gets split on what you tell it and not other other random spaces:
>
> var="1:one two
> 2:three four"
> print -l -- ${(f)"${var}"}
>
> The -l prints one element per line for clarity.
>
> You've now got the lines you want as if they were array elements.  You
> can then surround that substitution with the sort you first thought of.
> For example, to strip the "number:",
>
> print -l -- ${${(f)"${var}"}##<->:}
>
> where, as you now know, <-> matches any set of digits.
>
> pws
Thanks Peter,  I just came up with this:

     echo "${(F)var[@]//#???????/}"

... it seems the numeric leader is always the same width, so mine is 
more stupidly
linear, but it was my enlightenment as to '<->' that got me going with this.
I pretty much understand it; the trick was to get the substitution on 
each element.
And get the newline as the separator.  I'll study your version too.

BTW, I just rediscovered your:

A User's Guide to the Z-Shell
Peter Stephenson
2003/03/23

In the fog of war I'd considered it to be 'the manual' back when I first 
started,
then when I started reading the real manual I had this powerful 
sensation that
the manual had gone downhill.  Now I'm realizing that they're not the same
tome after all and I must say that yours is a pleasure to read which 
can't be
said of the manual.  Is there an update?

Re: subsitutions and beginning of lines.

[Daniel Shahaf]

Ray Andrews wrote on Sun, Oct 11, 2015 at 12:12:26 -0700:
> ${var// /}
> 
> When doing that sort of substitution, is is possible to test for newlines?
> I'm playing with a function that grabs history and I'm trying to strip
> off the leading numbers.

If you're trying to parse something like this:

    % history | tail -2
     1004  echo foo
     1005  echo bar

You could avoid parsing the line numbers (and unescaping the commands)
altogether by accessing the history differently:

    % print -r - $history[1004]
    echo foo

Re: subsitutions and beginning of lines.

[Peter Stephenson]

On Sun, 11 Oct 2015 16:05:19 -0700
Ray Andrews <rayandrews@eastlink.ca> wrote:
> BTW, I just rediscovered your:
> 
> A User's Guide to the Z-Shell
> Peter Stephenson
> 2003/03/23
> 
> In the fog of war I'd considered it to be 'the manual' back when I first 
> started,
> then when I started reading the real manual I had this powerful 
> sensation that
> the manual had gone downhill.  Now I'm realizing that they're not the same
> tome after all and I must say that yours is a pleasure to read which 
> can't be
> said of the manual.  Is there an update?

No, but nothing there has stopped being valid and there's little
fundamentally new on the subject (what you're asking about here
has been in the shell for most of its history, but subject to
tweaks).

I kind of stopped work on the guide when the book came along, but
actually at that point it was complete up to the sort of things I
thought it needed.  Anything more would depend a bit what the questions
were --- I suppose it would need some sort of specific project to
work through.

pws

Re: subsitutions and beginning of lines.

[Ray Andrews]

On 10/12/2015 02:34 AM, Peter Stephenson wrote:
>
> said of the manual.  Is there an update?
> No, but nothing there has stopped being valid and there's little
> fundamentally new on the subject (what you're asking about here
> has been in the shell for most of its history, but subject to
> tweaks).
>
> I kind of stopped work on the guide when the book came along, but
> actually at that point it was complete up to the sort of things I
> thought it needed.  Anything more would depend a bit what the questions
> were --- I suppose it would need some sort of specific project to
> work through.
Yeah, I'm not too worried about it being obsolete, I just think it's the 
best doc
I've come across and I'd hope it would be a going concern.  I'm 
preferring it
to the book, even, mostly due to the fact that the former spends less time
digressing into other shells.  The Googleverse is a sad place when one is
looking for the best docs because it shows you everything.  Nice to find the
really best doc and trust it.  BTW, I figured out how/why it came to be that
I thought the Guide and the Manual were the same thing:  It's because they
show the same icon in a browser tab so I closed one at random, thinking
they were the same doc anyway and I guessed wrong.

Re: subsitutions and beginning of lines.

[Ray Andrews]

On 10/11/2015 07:26 PM, Daniel Shahaf wrote:
> You could avoid parsing the line numbers (and unescaping the commands) 
> altogether by accessing the history differently: 

> % print -r - $history[1004] echo foo 

I could never get that to work with anything besides a single 
subscript.  I'd expect it
to drive like any other array but nothing seems to work:

     print -lr - $history[-5,-1]

... and the point of what I'm doing is to show a range.

Re: subsitutions and beginning of lines.

[Peter Stephenson]

On Mon, 12 Oct 2015 08:53:21 -0700
Ray Andrews <rayandrews@eastlink.ca> wrote:

> On 10/11/2015 07:26 PM, Daniel Shahaf wrote:
> > You could avoid parsing the line numbers (and unescaping the commands) 
> > altogether by accessing the history differently: 
> 
> > % print -r - $history[1004] echo foo 
> 
> I could never get that to work with anything besides a single 
> subscript.  I'd expect it
> to drive like any other array but nothing seems to work:
> 
>      print -lr - $history[-5,-1]
> 
> ... and the point of what I'm doing is to show a range.

It's an associative array, so you need the exact index as a decimal with
no leading zeros; ranges and reverse indices don't work.

% print ${(t)history}
association-readonly-hide-hideval-special

But you can fake something like that above up:


local -a myhist=() indices=({$((HISTCMD-5))..$((HISTCMD-1))})
integer ind
for ind in $indices; do
  myhist+=(${history[$ind]})
done


pws

Re: subsitutions and beginning of lines.

[Ray Andrews]

Peter,

A few niggles with the substitution thing:

$ history -10

    85* s h okIFS
12586* freshensource
12587* cp Navtools/navtools Source
12588zsh
12589  echo "Shall I compare thee to a Summer's day?"
12590* echo "Thou art more lovely and more temperate"
12591  echo "Rough winds do shake the darling buds of May"
12592* echo And Summer's lease hath all to short a date"
12593* history -10

... it all lines up like North Korean soldiers.  ('85' is of course 
ancient, but it would line uptoo if it was in the output.)

So, my 'h' wrapper is an incremental colorized 'grep' of history output:

    $ echo "Shall I compare thee to a Summer's day?"
    $ echo "Thou art more lovely and more temperate"
    $ echo "Rough winds do shake the darling buds of May"
    $ echo "And Summer's lease hath all to short a date"

    $ h 100 to

    print -r - $history[-5,-1]
    print -lr - $history[-5,-1]
    print -l - $history[-5,-1]
    print -l - $history[-1]
    print -r $history[-1]
    cp Navtools/navtools Source
    echo "Shall I compare thee to a Summer's day?"
    echo And Summer's lease hath all to short a date

     $ h 100 to 'a date'

     echo And Summer's lease hath all to short a date
**       ******

... Each search parameter is colorized incrementally.  (I add the minus 
sign automatically) It now strips off the leading stuff as we've 
discussed.  Code as I have it now is:

     OLDIFS=$IFS
     IFS=$'\n'
var=($(eval history $nnumber $sstring))

      echo "${(F)var[@]//#???????/}"
      IFS=$OLDIFS

... where $nnumber is obvious and $sstring is put together like this:

     while [ -n "$1" ]; do
       sstring="$sstring | grep '$1'"
       shift
     done

So, the 'eval'ed expression ends up like:

'history -100 | grep --color=always to | grep --color=always 'a date' | 
... '

(Pardon the long lead-up)

So ... the issue is that if I use your method it's much simpler *but* 
when the
splittings and joinings and all the rest of that invisible stuff happens 
the outputno longer lines up because double spaces become single spaces 
whereas
double spaces are needed for the brutal ' //#???????/' substitution to work
properly.

Is there an elegant solution? My code is obviously cumbersome.A more 
refined substitution could fix it, but on principal I want to leave 
doublespaces alone without needing to play with $IFS.All this splitting 
stuff is wonderful when it works, but there are times when I wish I 
could turn it all off.

Re: subsitutions and beginning of lines.

[Bart Schaefer]

On Oct 12, 10:07am, Ray Andrews wrote:
}
} ... Each search parameter is colorized incrementally.  (I add the minus 
} sign automatically) It now strips off the leading stuff as we've 
} discussed.  Code as I have it now is:
} 
}      OLDIFS=$IFS
}      IFS=$'\n' var=($(eval history $nnumber $sstring))
} 
}       echo "${(F)var[@]//#???????/}"
}       IFS=$OLDIFS
} 
} So, the 'eval'ed expression ends up like:
} 
} 'history -100 | grep --color=always to | grep --color=always 'a date' | ... '

If you're using "grep" to do the colorizing, there's no point in using
the $history variable to get the history entries, because you need them
to end up on stdout for passing through grep anyway.

(Generic advice to anyone reading this:  Asking about one small piece of
a larger problem is quite likely to get you an inappropriate solution.
Context is important.)

} the outputno longer lines up because double spaces become single
} spaces whereas double spaces are needed for the brutal ' //#???????/'
} substitution to work properly.

Why not simply use "history -n" so the numbers are omitted in the first
place?  Then you don't need to assign to var=(...) and apply //#.../.
That whole chunk of code just reduces to

    eval history -n $nnumber $sstring

and you're done.

Re: subsitutions and beginning of lines.

[Ray Andrews]

On 10/12/2015 07:29 PM, Bart Schaefer wrote:
>      eval history -n $nnumber $sstring
>
> and you're done.
>
Good idea :(

No matter how much I read, I seem to always miss
what I'm looking for.  There's a huge section "HISTORY"
in the man page, but it doesn't mention that and I'll probably
have to eat my hat when you show me that it does.

Re: subsitutions and beginning of lines.

[Bart Schaefer]

On Oct 12,  7:50pm, Ray Andrews wrote:
}
} There's a huge section "HISTORY"
} in the man page

There is?  Where?  There's "HISTORY EXPANSION" but it's all about using
"!"-references on the command line.  There's a subsection on "History"
in the section about options.  Where are you looking?

} but it doesn't mention that and I'll probably
} have to eat my hat when you show me that it does.

This is in "man zshbuiltins" under the "fc" command (to which the
"history" builtin's doc directs you).

Re: subsitutions and beginning of lines.

[Ray Andrews]

On 10/12/2015 10:03 PM, Bart Schaefer wrote:
> On Oct 12,  7:50pm, Ray Andrews wrote:
> }
> } There's a huge section "HISTORY"
> } in the man page
>
> There is?  Where?  There's "HISTORY EXPANSION" but it's all about using
> "!"-references on the command line.  There's a subsection on "History"
> in the section about options.  Where are you looking?
Yup, that's it.
>
> } but it doesn't mention that and I'll probably
> } have to eat my hat when you show me that it does.
>
> This is in "man zshbuiltins" under the "fc" command (to which the
> "history" builtin's doc directs you).
Ooops, so it does.  Still a brief summary of history specific switches 
would
be friendly.  'fc' in it's totality is a bit intimidating.

BTW Bart, the way I was trying to do it was educational so I don't 
regret it,
and I think my question is still a useful exercise: how would we do that
if there was no easy way?  Combining the words results in a 'command not
found' error,  since it seems to want to believe that 'history -10 | 
....' is one
command so I hafta leave the words unmagically joined yet the normal thing
seems to be to strip spaces down to a single one when making an array,
and that's a problem with the methods I was trying.  I don't doubt it's 
easy if you
get the incantation exactly right.  It doesn't help that all this splitting
is invisible.  As you pointed out:
> eval history -n $nnumber $sstring

... is a easy as pie so long as there's no need to massage the output,
but even if you do, I'm betting it's still only a couple of  lines. Of 
course
there's sed, but I'm banking on an internal solution.


>

Re: subsitutions and beginning of lines.

[Bart Schaefer]

On Oct 12, 10:24pm, Ray Andrews wrote:
}
} and I think my question is still a useful exercise: how would we do that
} if there was no easy way?  Combining the words results in a 'command not
} found' error,  since it seems to want to believe that 'history -10 | 
} ....' is one command

Aren't those two unrelated issues?  The "eval" you used is already the
correct way to turn a string formulated as a command line into an actual
command.  The array manipulation you asked about would only apply to
the output of the command after eval.

} so I hafta leave the words unmagically joined yet the normal thing
} seems to be to strip spaces down to a single one when making an array

You may need to go back to a specific example, I fear.  Making an array
how?  By capturing the output of a command?

The normal thing when capturing $(...) in an array context is to treat
all adjacent (consecutive) whitespace (characters listed in $IFS) as a
single word break, yes.  But the spaces aren't being "stripped down to
a single one".  They're being stripped out ENTIRELY, leaving nothing.
It's when putting the array back together into a string again that a
single space is inserted between words.

} It doesn't help that all this splitting is invisible.

This is part and parcel of the whole quoting behavior.  Using double
quotes when you don't want splitting is something you need to habituate.
Once the quotes are in place, you can make splitting explicit/visible.

Peter already gave you the ${(f)"${var}"} and ${${(f)"${var}"}##<->:}
incantations; you can use this directly with $(...) as well:

    ${${(f)"$(eval history $nnumber $sstring)"}##<->:}

Re: subsitutions and beginning of lines.

[Ray Andrews]

On 10/13/2015 12:58 PM, Bart Schaefer wrote:
> The normal thing when capturing $(...) in an array context is to treat
> all adjacent (consecutive) whitespace (characters listed in $IFS) as a
> single word break, yes.  But the spaces aren't being "stripped down to
> a single one".  They're being stripped out ENTIRELY, leaving nothing.
> It's when putting the array back together into a string again that a
> single space is inserted between words.

Ah ... a subtle point but critical.  Yes ... I need to 'think array' not 
'think string'.
I apologize for my C headedness which refuses to go away.
>      ${${(f)"$(eval history $nnumber $sstring)"}##<->:}

I couldn't get the substitution there to work, but this works fine:

	print -l ${${(f)"$(eval history $nnumber $sstring)"}/ #<->??/}

... and I can see that the inner '$(eval ..)' layer will not
molest any spaces at all and each output line is an element unto
itself in an array split on newlines which is then sent for surgery
and then printed one line per element.  It's so easy to get it
wrong, but when you get it right, it's down right elegant and not
really obscure at all.

BTW, is there some way of asking history not to search
thru a given number of entries, but to continue searching up to
a given number of hits? Making syntax up:

history -#10 intricate_command

... I find when searching, that what I'm really wanting is a
reasonable number of hits to select from, and my choice of
number for the switch is just a guess which I scale up or
down to achieve some number of hits.  I might know that
some time in the last year I entered some command that I'd
like back and I don't really care how far back in history
it was, I just want history to scan backwards until
a match (or several) is found.  Can it be done? If not
it would be a feature.

Re: subsitutions and beginning of lines.

[Bart Schaefer]

On Oct 13,  2:42pm, Ray Andrews wrote:
}
} BTW, is there some way of asking history not to search
} thru a given number of entries, but to continue searching up to
} a given number of hits?

What defines a "hit" here?  Your "history" command isn't "searching"
AT ALL.  Rather it's your series of "grep" commands that are doing
the searching.  There's no way an argument to the "history" command
can tell it how many times "grep" of its output is going to succeed.

So it sounds like what you really want is

    eval history -r -n -$HISTSIZE $sstring | head $nnumber

That is, dump the entire history, grep it, and then show the most
recent few events that match the greps.

Re: subsitutions and beginning of lines.

[Ray Andrews]

On 10/13/2015 05:23 PM, Bart Schaefer wrote:
> What defines a "hit" here?  Your "history" command isn't "searching"
> AT ALL.  Rather it's your series of "grep" commands that are doing
> the searching.  There's no way an argument to the "history" command
> can tell it how many times "grep" of its output is going to succeed.
Yes, how silly of me. history by itself knows nothing of my grepping.  
Duhhhh.
> So it sounds like what you really want is
>
>      eval history -r -n -$HISTSIZE $sstring | head $nnumber
>
> That is, dump the entire history, grep it, and then show the most
> recent few events that match the greps.
Sure, but I was hoping for something a little more delicate.
The -m switch does do some internal finding, so maybe that's the
starting point. Couldn't history count it's matches when the '-m' is used?

     history -M$nnumber "*$sstring"*" ... or something.

Never mind tho.  Working from your example above I come up with:

     $ history -n -m "*$sstring*" 1 | tail -n $nnumber | grep 
--color=auto "$sstring"

And it's very satisfactory.  She burns through the entire history so fast
that there's nothing to complain about.   Thanks Bart.