Replacing sed for zsh portability

Replacing sed for zsh portability

[DervishD]

    Hi all :)

    I'm going to release some scripts, and they have the
documentation within the own script. In my system I extract it using
GNU sed, but now I cannot do it since I'm using a GNU extension. In
order to avoid the non-portable '\|' sed regex operator, I must do
the following:

    sed -ne '/<=$/,/=>$/p' < $0 | while read line
    do
        print -- ${${line#\#}#(<=|=>| )}
    done

    The documentation is enclosed between "<=" and "=>" delimiters,
something like this:

#<=
# This is an example of documentation
# Here are more lines
#
# Yet even more lines here
...
#=>

    Since I cannot use sed for the entire work unless I complicate
the script (doing multiple substitutions and the like), I want to get
rid of the current script and doing all the job in ZSH. It doesn't
matter if it is very slow, speed here is not an issue.

    Can this be done in zsh easily or the only way is to use a while
loop to ignore lines before the delimiter and another to process the
documentation itself? Otherwise I'll try to write a portable sed
script, although I would prefer to remove any dependency except for
zsh itself of course.

    Another question, related to this. It seems that the X|Y glob
operator doesn't work left-to-right, but shortest-first. If I do
this:

    print ${line#\#(<=|=>| |)}

    then the 'nothing' at the end is used as the match, and the
delimiters are not matched. Is doing two substitutions the only way
(as I'm doing currently)?

    Thanks a lot in advance :)

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Replacing sed for zsh portability

[DervishD]

    Hi all :)

    Sorry for self replying... again.

 * DervishD <zsh@dervishd.net> dixit:
>     Another question, related to this. It seems that the X|Y glob
> operator doesn't work left-to-right, but shortest-first. If I do
> this:
> 
>     print ${line#\#(<=|=>| |)}

    I've tried to be a bit less stupid and use the correct form:

    print ${line##\#(<=|=>| |)}

    Obviously that zsh will use the shortest-first match if
instructed to do so. Sorry for the noise, I didn't noticed I was
using "#" instead of "##".

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Replacing sed for zsh portability

[Peter Stephenson]

DervishD wrote:
> #<=
> # This is an example of documentation
> # Here are more lines
> #
> # Yet even more lines here
> ...
> #=>
> 
>     Since I cannot use sed for the entire work unless I complicate
> the script (doing multiple substitutions and the like), I want to get
> rid of the current script and doing all the job in ZSH. It doesn't
> matter if it is very slow, speed here is not an issue.
> 
>     Can this be done in zsh easily or the only way is to use a while
> loop to ignore lines before the delimiter and another to process the
> documentation itself?

A while loop is the obvious way.  You can read in the entire file
(file="$(<file)") and substitute on that.  My example succeeded with this:

print -r ${(S)file//$'\n#<='*$'\n#=>'}

although the following is better since it takes account of the fact
that a comment may be at the start: 

print -r ${(S)file//($'\n'|(#s))\#\<\=*$'\n#=>'}

This requires extended_glob.

>     Another question, related to this. It seems that the X|Y glob
> operator doesn't work left-to-right, but shortest-first. If I do
> this:
> 
>     print ${line#\#(<=|=>| |)}
> 
>     then the 'nothing' at the end is used as the match, and the
> delimiters are not matched.

You're missing the fact that your substitution is of the form ${line#...}.
This is explicitly documented to remove the shortest matching chunk at
the head of the line.  Try adding another "#":

    print ${line##\#(<=|=>| |)}

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

**********************************************************************

Re: Replacing sed for zsh portability

[DervishD]

    Hi Peter :)

 * Peter Stephenson <pws@csr.com> dixit:
> DervishD wrote:
> >     Can this be done in zsh easily or the only way is to use a while
> > loop to ignore lines before the delimiter and another to process the
> > documentation itself?
> A while loop is the obvious way.  You can read in the entire file
> (file="$(<file)") and substitute on that.  My example succeeded with this:
> 
> print -r ${(S)file//$'\n#<='*$'\n#=>'}

    This prints all the script *except* the lines I want ;)

    Anyway, can I get rid of the 'S' flag if I know that the
delimiter won't appear in the script code?

    Using the 'read-entire-file' solution, I'm using this expression
right now:

    print -r ${file//(#b)*'#<='(*)'#=>'*/${match//\#/}}

    Which gives a segmentation fault! It happens even when doing "zsh
-f", using 4.2.5 (i686-pc-linux-gnu)

    This expression below:
    print -r ${file//(#b)*'#<='(*)'#=>'*/$match}

    gives me the result but with the hashes before the lines, and
uses the '\n' already in the text as separators. A bit prettier,
don't know. And this expression works OK:

    print -r ${${file//(#b)*'#<='(*)'#=>'*/$match}//\#}

    Don't know why the second substitution works and the first gave a
sigsegv. I may do a little research if you want me to. I'm not
familiar with the sources, but the fail may be on a memcpy call, at
least is there where it fails using gdb without debugging information
:((

> although the following is better since it takes account of the fact
> that a comment may be at the start: 
> 
> print -r ${(S)file//($'\n'|(#s))\#\<\=*$'\n#=>'}

    Fortunately the comment won't never be at the start (the GPL
disclaimer is the first thing in the script just after the bangpath.

> >     Another question, related to this. It seems that the X|Y glob
> > operator doesn't work left-to-right, but shortest-first. If I do
> > this:
> > 
> >     print ${line#\#(<=|=>| |)}
> > 
> >     then the 'nothing' at the end is used as the match, and the
> > delimiters are not matched.
> You're missing the fact that your substitution is of the form ${line#...}.
> This is explicitly documented to remove the shortest matching chunk at
> the head of the line.  Try adding another "#":
> 
>     print ${line##\#(<=|=>| |)}

    Yes, read my autoreply about my stupidity when writing the
expression. I just missed the second '#'. I noticed when rereading my
own message. Only then the single '#' appeared as a beacon. Sorry O:)

    Thanks for your answer, Peter :) Very good solution. I refrained
from reading the entire file into memory although I am pretty sure
the scripts are small enough.

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Replacing sed for zsh portability

[Vincent Lefevre]

On 2005-08-15 12:35:57 +0200, DervishD wrote:
>     Since I cannot use sed for the entire work unless I complicate
> the script (doing multiple substitutions and the like), I want to get
> rid of the current script and doing all the job in ZSH. It doesn't
> matter if it is very slow, speed here is not an issue.

For things related to regexp, it would probably be much easier with Perl.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: Replacing sed for zsh portability

[DervishD]

    Hi Vincent :)

 * Vincent Lefevre <vincent@vinc17.org> dixit:
> On 2005-08-15 12:35:57 +0200, DervishD wrote:
> >     Since I cannot use sed for the entire work unless I complicate
> > the script (doing multiple substitutions and the like), I want to get
> > rid of the current script and doing all the job in ZSH. It doesn't
> > matter if it is very slow, speed here is not an issue.
> For things related to regexp, it would probably be much easier with Perl.

    The thing is perfectly suitable to be done in sed, in fact it is
a one-liner (three lines if non-GNU sed), and it can even be done in
one or two lines in Zsh. Introducing a dependency with Perl just for
that... no. Moreover, I don't think that *anything* is easier with
Perl. I have a theory: if you just hit your keyboard randomly at your
favourite editor, chances are that the noise produced are a perfectly
legal (although non functional) Perl script. And Perl-6... well.

    For regexes I prefer sed, or awk sometimes, although I must
confess that since I use zsh my needs for using sed or awk have
dropped dramatically.

    Thanks anyway, Vincent :), but the problem is not complex enough
to need Perl for being solved.

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Replacing sed for zsh portability

[Vincent Lefevre]

On 2005-08-16 17:51:14 +0200, DervishD wrote:
>     The thing is perfectly suitable to be done in sed, in fact it is
> a one-liner (three lines if non-GNU sed), and it can even be done in
> one or two lines in Zsh. Introducing a dependency with Perl just for
[...]

The problem with sed is that it is non-portable, and it is difficult
to be sure that a sed call will work as expected on any machine.

With zsh, things related to regexp are difficult to read, and the
behaviour often depends on options (one needs to remember to use
"emulate -LR zsh", and this should be OK, however).

I've had to switch to Perl for a simple replacement just because
in-place editing with sed isn't supported everywhere, and when it
is supported, it doesn't behave in the same way on every machine.
Though this is in a zsh function, I haven't tried with zsh.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

Re: Replacing sed for zsh portability

[Oliver Kiddle]

Vincent Lefevre wrote:
> I've had to switch to Perl for a simple replacement just because
> in-place editing with sed isn't supported everywhere, and when it
> is supported, it doesn't behave in the same way on every machine.

If it is in-place editing you want, you can always use ed.

You need to use a here-document for the replacement command but it works
well.

Oliver


This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.

Re: Replacing sed for zsh portability

[DervishD]

    Hi Vincent :)

 * Vincent Lefevre <vincent@vinc17.org> dixit:
> On 2005-08-16 17:51:14 +0200, DervishD wrote:
> >     The thing is perfectly suitable to be done in sed, in fact it is
> > a one-liner (three lines if non-GNU sed), and it can even be done in
> > one or two lines in Zsh. Introducing a dependency with Perl just for
> [...]
> The problem with sed is that it is non-portable, and it is difficult
> to be sure that a sed call will work as expected on any machine.

    I know, but both POSIX and SUS have standarized a (big) subset of
sed programs, and I would tell that 100% of the sed's out there
support the 's' command. In fact, a system without a working sed (I
mean, a sed that doesn't understand the 's' command) won't run
'configure' scripts properly and so is possible that zsh won't
compile on them, so...
 
> With zsh, things related to regexp are difficult to read, and the
> behaviour often depends on options (one needs to remember to use
> "emulate -LR zsh", and this should be OK, however).

    That's for sure, I will use 'emulate' because it ensures (more or
less) a sane default, known environment. And I don't care about
readability because Perl is not readable and the end user won't ever
see the regex, it's me who is going to maintain it ;)
 
> I've had to switch to Perl for a simple replacement just because
> in-place editing with sed isn't supported everywhere, and when it
> is supported, it doesn't behave in the same way on every machine.

    Well, in place editing is just a shortcut for doing a couple of
'mv's. The only problem can be side effects in the middle of the
moves, but if Perl does the in-place editing using a temporary file
and a rename... I don't know how it is done neither in Perl nor in
sed, anyway.

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to...

Re: Replacing sed for zsh portability

[Vincent Lefevre]

On 2005-08-16 18:34:20 +0200, Oliver Kiddle wrote:
> If it is in-place editing you want, you can always use ed.

Yet another language to learn... I think I'll stick with Perl
(OK for both simple and complex needs) and zsh.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA