Re: (feature request) Shell script within shell script

Re: (feature request) Shell script within shell script

[William Park]

Dear Zsh expert,

When writing a shell script, quite often, I have to write bunch of
separate Awk/Python/Perl/etc scripts that are more than few lines.  I
tend to lose track of which script does what, and get bitten over two
levels of quotes.

Essentially, I wish I could do something like
    
    herefile test1 << "EOF"
    #! /usr/bin/gawk -f
    ...
    ...
    EOF

and

    herefile test2 << "EOF"
    #! /usr/bin/python
    ...
    ...
    EOF

so that invocation and usage of 'test1' and 'test2' will be the same as
if they were separate real files.

Can Zsh do this now?  If not, does Zsh have standard mechanism to
provide this kind of extensions?

-- 
William Park, Open Geometry Consulting, <opengeometry@yahoo.ca>
Linux solution for data management and processing. 

RE: (feature request) Shell script within shell script

[Clifford Caoile]

William Park:

I am not an expert, but I was exploring here documents in bash and zsh just
this last week. Let me see if I get this right:

In zsh-4.0.x, you can use "here documents" or "here-strings". A here
document has this syntax (note there is no double quotations necessary):

>> /etc/hosts <<EOF
192.168.0.26 mynewcomputer.zsh.org
EOF

This adds the line to the /etc/hosts file. The cat command is implicit.
(It's still a useless cat trick, right?)

A "here string" uses the <<< construct, but I have not used this. Please
check the docs.

You can also strip indentation by typing "<<-" instead of just "<<".

There is also interpretation done within the here document. This has bitten
me a couple of times. Remember in a bash or zsh here document that the '$'
(dollar sign), '\' (backslash) and the '`' (backquote) are interpreted even
inside a here document, so you must prepend a backslash, i.e. '\$' and '\\'
and '\`'. This make the here document look gastly horrid. You mention you
will be using Awk/Python/Perl/etc. I know that Perl uses $ a lot. Also Awk?
What you will be typing will not be easy to keep track because zsh silently
substitutes text. Debugging will be hard. Is it possible to turn this off?

Of course, you could always try to construct your foreign language script
line by line. That is what I did before, like this:

rm $tmpcmdfile
echo '#! /usr/bin/perl' >> $tmpcmdfile
...
chmod 700 $tmpcmdfile
$tmpcmdfile
rm $tmpcmdfile

Do you like typing? ;-) This example actually led me to exploring here
documents.

Specifically with your example, I am unable to ascertain what you want to
do. Do you want to execute that here document on the line where you declare
it? What does that "herefile" mean? Is that syntactic sugar meaning "foreign
language eval?" In any event, I do not know how to do that, and I don't
think that's a zsh-4.0.x capability.

Can a real expert answer please? ;-)

HTH
  ,-~-.
 < ^ ; ~,  Clifford Escobar CAOILE  (aka "Piyo-kun")
  (_  _,
   J~~>  _.___...:.__..__.: __.:_. .:_.__::_..:._::._...  _____ p(^_^)q


| -----Original Message-----
| From: William Park [mailto:opengeometry@yahoo.ca]
| Sent: Saturday, February 01, 2003 16:37
| To: zsh-users@sunsite.dk
| Subject: Re: (feature request) Shell script within shell script
|
|
| Dear Zsh expert,
|
| When writing a shell script, quite often, I have to write bunch of
| separate Awk/Python/Perl/etc scripts that are more than few lines.  I
| tend to lose track of which script does what, and get bitten over two
| levels of quotes.
|
| Essentially, I wish I could do something like
|
|     herefile test1 << "EOF"
|     #! /usr/bin/gawk -f
|     ...
|     ...
|     EOF
|
| and
|
|     herefile test2 << "EOF"
|     #! /usr/bin/python
|     ...
|     ...
|     EOF
|
| so that invocation and usage of 'test1' and 'test2' will be the same as
| if they were separate real files.
|
| Can Zsh do this now?  If not, does Zsh have standard mechanism to
| provide this kind of extensions?
|
| --
| William Park, Open Geometry Consulting, <opengeometry@yahoo.ca>
| Linux solution for data management and processing.

Re: (feature request) Shell script within shell script

[Will Yardley]

On Sun, Feb 02, 2003 at 12:48:01AM +0900, Clifford Caoile wrote:
 
> I am not an expert, but I was exploring here documents in bash and zsh just
> this last week. Let me see if I get this right:
> 
> In zsh-4.0.x, you can use "here documents" or "here-strings". A here
> document has this syntax (note there is no double quotations necessary):
> 
> >> /etc/hosts <<EOF
> 192.168.0.26 mynewcomputer.zsh.org
> EOF
> 
> This adds the line to the /etc/hosts file. The cat command is implicit.
> (It's still a useless cat trick, right?)

Zsh can already do heredocs; whether you use quotes or not affects how
quotes and stuff are treated in the actual heredoc.

-- 
"Since when is skepticism un-American?
Dissent's not treason but they talk like it's the same..."
(Sleater-Kinney - "Combat Rock")

Re: (feature request) Shell script within shell script

[William Park]

On Sun, Feb 02, 2003 at 12:48:01AM +0900, Clifford Caoile wrote:
> William Park:
> 
> I am not an expert, but I was exploring here documents in bash and zsh
> just this last week. Let me see if I get this right:
> 
> In zsh-4.0.x, you can use "here documents" or "here-strings". A here
> document has this syntax (note there is no double quotations
> necessary):
> 
> >> /etc/hosts <<EOF
> 192.168.0.26 mynewcomputer.zsh.org
> EOF
> 
> This adds the line to the /etc/hosts file. The cat command is
> implicit.  (It's still a useless cat trick, right?)
> 
> A "here string" uses the <<< construct, but I have not used this.
> Please check the docs.
> 
> You can also strip indentation by typing "<<-" instead of just "<<".
> 
> There is also interpretation done within the here document. This has
> bitten me a couple of times. Remember in a bash or zsh here document
> that the '$' (dollar sign), '\' (backslash) and the '`' (backquote)
> are interpreted even inside a here document, so you must prepend a
> backslash, i.e. '\$' and '\\' and '\`'. This make the here document
> look gastly horrid. You mention you will be using Awk/Python/Perl/etc.
> I know that Perl uses $ a lot. Also Awk?  What you will be typing will
> not be easy to keep track because zsh silently substitutes text.
> Debugging will be hard. Is it possible to turn this off?

If you quote EOF (ie. "EOF"), then the content of Here Documents will
not be expanded.  So, no problem with '$', '\', and '`'.

> 
> Of course, you could always try to construct your foreign language
> script line by line. That is what I did before, like this:
> 
> rm $tmpcmdfile
> echo '#! /usr/bin/perl' >> $tmpcmdfile
> ...
> chmod 700 $tmpcmdfile
> $tmpcmdfile
> rm $tmpcmdfile
> 
> Do you like typing? ;-) This example actually led me to exploring here
> documents.
> 
> Specifically with your example, I am unable to ascertain what you want
> to do. Do you want to execute that here document on the line where you
> declare it? What does that "herefile" mean? Is that syntactic sugar
> meaning "foreign language eval?" In any event, I do not know how to do
> that, and I don't think that's a zsh-4.0.x capability.
> 
> Can a real expert answer please? ;-)

However, I'm not talking about redirecting into a real file.  You can do
that with Here Document easily enough.  What I want is similiar to
function.  You define it in the main shell script, and shell will read
and store it in memory.  When you call it, shell will execute the
content just as though it is a real external file script.

If the here content is short, then I can do something like
	awk '...'
or
	python '
	...	(due to indentation)
	'
instead of Here Document.  Of course, the 2 levels of quotes are pain.
The convenience is that you only have to edit 1 file.


> | When writing a shell script, quite often, I have to write bunch of
> | separate Awk/Python/Perl/etc scripts that are more than few lines.  I
> | tend to lose track of which script does what, and get bitten over two
> | levels of quotes.
> |
> | Essentially, I wish I could do something like
> |
> |     herefile test1 << "EOF"
> |     #! /usr/bin/gawk -f
> |     ...
> |     ...
> |     EOF
> |
> | and
> |
> |     herefile test2 << "EOF"
> |     #! /usr/bin/python
> |     ...
> |     ...
> |     EOF
> |
> | so that invocation and usage of 'test1' and 'test2' will be the same as
> | if they were separate real files.
> |
> | Can Zsh do this now?  If not, does Zsh have standard mechanism to
> | provide this kind of extensions?

-- 
William Park, Open Geometry Consulting, <opengeometry@yahoo.ca>
Linux solution for data management and processing. 

RE: (feature request) Shell script within shell script

[Clifford Caoile]

Quotes... mmm zsh-goodness... Thanks for the heads up I didn't know that.

  ,-~-.
 < ^ ; ~,  Clifford Escobar CAOILE  (aka "Piyo-kun")
  (_  _,
   J~~>  _.___...:.__..__.: __.:_. .:_.__::_..:._::._...  _____ p(^_^)q


| -----Original Message-----
| From: Will Yardley [mailto:william@hq.dreamhost.com]
| Sent: Sunday, February 02, 2003 01:57
| To: zsh-users@sunsite.dk
| Subject: Re: (feature request) Shell script within shell script
|
|
| On Sun, Feb 02, 2003 at 12:48:01AM +0900, Clifford Caoile wrote:
|
| > I am not an expert, but I was exploring here documents in bash
| and zsh just
| > this last week. Let me see if I get this right:
| >
| > In zsh-4.0.x, you can use "here documents" or "here-strings". A here
| > document has this syntax (note there is no double quotations necessary):
| >
| > >> /etc/hosts <<EOF
| > 192.168.0.26 mynewcomputer.zsh.org
| > EOF
| >
| > This adds the line to the /etc/hosts file. The cat command is implicit.
| > (It's still a useless cat trick, right?)
|
| Zsh can already do heredocs; whether you use quotes or not affects how
| quotes and stuff are treated in the actual heredoc.
|
| --
| "Since when is skepticism un-American?
| Dissent's not treason but they talk like it's the same..."
| (Sleater-Kinney - "Combat Rock")
|
|

Re: (feature request) Shell script within shell script

[Bart Schaefer]

On Feb 1,  2:36am, William Park wrote:
} Subject: Re: (feature request) Shell script within shell script
}
} Essentially, I wish I could do something like
}     
}     herefile test1 << "EOF"
}     #! /usr/bin/gawk -f
}     ...
}     ...
}     EOF

Effectively, what you want to do is create a temp file and then execute
it.

That's as simple as:

    herefile () {
      setopt localtraps noshwordsplit
      local tmp=${TMPPREFIX}HERE$$
      trap "command rm -f $tmp" EXIT
      : >| $tmp				|| return
      chmod u=rwx,go-rwx $tmp		|| return
      cat >| $tmp			|| return
      $tmp $@
    }

Try it with this silly example:

    herefile <<\EOF
    #! /bin/ls -l
    EOF

The tricky bit of course is that the redirection becomes the standard
input of the "herefile" function, so your script will see end-of-file
as soon as it starts running.

You can avoid this by doing some file descriptor manipulations:

    _herefile () {
      setopt localtraps noshwordsplit
      local tmp=${TMPPREFIX}HERE$$
      trap "command rm -f $tmp" EXIT
      : >| $tmp				|| return
      chmod u=rwx,go-rwx $tmp		|| return
      cat >| $tmp			|| return
      $tmp $@ <&3 3<&-
    }
    alias herefile='_herefile 3<&0'

    herefile /etc/HOSTNAME /etc/issue <<\EOF
    #! /usr/bin/perl -p
    BEGIN { print join("\n\t", 'Printing these files: ', @ARGV)."\n\n"; }
    EOF

This of course has the drawback of always consuming file descriptor 3,
so it may not work well with other scripts or functions that do lots
of file-descriptor swapping.  You can pick another descriptor number 
if you think it'll make collisions less likely.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: (feature request) Shell script within shell script

[Peter Stephenson]

"Bart Schaefer" wrote:
> The tricky bit of course is that the redirection becomes the standard
> input of the "herefile" function, so your script will see end-of-file
> as soon as it starts running.

Of course you don't necessarily need to use a here-document at all.

  therefile() {
      setopt localtraps localoptions noshwordsplit
      local tmp=${TMPPREFIX}HERE$$
      trap "command rm -f $tmp" EXIT
      touch $tmp		|| return
      chmod u=rwx,go-rwx $tmp	|| return
      print -r "$1" >>$tmp	|| return
      shift
      $tmp "$@"
  }

  therefile \
  '#!/usr/local/bin/perl
  print "You typed: @ARGV\n";
  ' arg1 arg2

You typed: arg1 arg2

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************

Re: (feature request) Shell script within shell script

[William Park]

On Sat, Feb 01, 2003 at 02:36:55AM -0500, William Park wrote:
> Dear Zsh expert,
> 
> When writing a shell script, quite often, I have to write bunch of
> separate Awk/Python/Perl/etc scripts that are more than few lines.  I
> tend to lose track of which script does what, and get bitten over two
> levels of quotes.
> 
> Essentially, I wish I could do something like
>     
>     herefile test1 << "EOF"
>     #! /usr/bin/gawk -f
>     ...
>     ...
>     EOF
> 
> and
> 
>     herefile test2 << "EOF"
>     #! /usr/bin/python
>     ...
>     ...
>     EOF
> 
> so that invocation and usage of 'test1' and 'test2' will be the same as
> if they were separate real files.
> 
> Can Zsh do this now?  If not, does Zsh have standard mechanism to
> provide this kind of extensions?


I've finally managed to find a solution.  It's called "process
substitution":
    function test1 () {
	awk -f <( cat <<"EOF"
	    ...
	    ...
	EOF
	)
    }
and
    function test2 () {
	python <( cat <<"EOF"
	    ...
	    ...
	EOF
	)
    }

-- 
William Park, Open Geometry Consulting, <opengeometry@yahoo.ca>
Linux solution for data management and processing. 

Re: (feature request) Shell script within shell script

[Bart Schaefer]

On Feb 3,  6:15pm, William Park wrote:
}
} I've finally managed to find a solution.  It's called "process
} substitution":
}     function test1 () {
} 	awk -f <( cat <<"EOF"
} 	    ...
} 	    ...
} 	EOF
} 	)
}     }

Just watch out for what happens if you ever have unbalanced parentheses in
the content of the here-document.

Re: (feature request) Shell script within shell script

[William Park]

On Tue, Feb 04, 2003 at 09:18:32AM +0000, Bart Schaefer wrote:
> On Feb 3,  6:15pm, William Park wrote:
> }
> } I've finally managed to find a solution.  It's called "process
> } substitution":
> }     function test1 () {
> } 	awk -f <( cat <<"EOF"
> } 	    ...
> } 	    ...
> } 	EOF
> } 	)
> }     }
> 
> Just watch out for what happens if you ever have unbalanced parentheses in
> the content of the here-document.

Wouldn't here-document have priority over <(...)?  Here-document will
read the content up to 'EOF' verbatim, as I understand.

-- 
William Park, Open Geometry Consulting, <opengeometry@yahoo.ca>
Linux solution for data management and processing. 

Re: (feature request) Shell script within shell script

[Bart Schaefer]

On Feb 4,  4:28am, William Park wrote:
}
} > Just watch out for what happens if you ever have unbalanced
} > parentheses in the content of the here-document.
} 
} Wouldn't here-document have priority over <(...)?

You'd think so, but that's not how <(...) et al. are parsed.

schaefer[501] ( cat <<EOF
subsh heredoc> heredoc
subsh heredoc> EOF)
subsh heredoc> )
subsh heredoc> EOF
subsh> )
heredoc
EOF)
)
schaefer[502] cat =( cat <<EOF
cmdsubst> heredoc
cmdsubst> EOF)
heredoc
schaefer[503] schaefer[504] cat =( cat <<EOF
cmdsubst> )

schaefer[504] 

Re: (feature request) Shell script within shell script

[William Park]

On Tue, Feb 04, 2003 at 05:24:27PM +0000, Bart Schaefer wrote:
> On Feb 4,  4:28am, William Park wrote:
> }
> } > Just watch out for what happens if you ever have unbalanced
> } > parentheses in the content of the here-document.
> } 
> } Wouldn't here-document have priority over <(...)?
> 
> You'd think so, but that's not how <(...) et al. are parsed.

Hmm, I just noticed.  It completes <(...) as soon as unmatched ')'
occurs.

-- 
William Park, Open Geometry Consulting, <opengeometry@yahoo.ca>
Linux solution for data management and processing. 

Re: (feature request) Shell script within shell script

[William Park]

On Tue, Feb 04, 2003 at 09:18:32AM +0000, Bart Schaefer wrote:
> On Feb 3,  6:15pm, William Park wrote:
> }
> } I've finally managed to find a solution.  It's called "process
> } substitution":
> }     function test1 () {
> } 	awk -f <( cat <<"EOF"
> } 	    ...
> } 	    ...
> } 	EOF
> } 	)
> }     }
> 
> Just watch out for what happens if you ever have unbalanced
> parentheses in the content of the here-document.

A classic solution:

    function test1 () {
    exec 10<<"EOF"
    ...
    ...
    EOF
	awk -f /dev/fd/10
    exec 10<&-
    }

I didn't have to patch anything.  I just have to keep track of fd's
instead of external files.  If the here-documents don't change, then I
can put 'exec' outside the function.
-- 
William Park, Open Geometry Consulting, <opengeometry@yahoo.ca>
Linux solution for data management and processing. 

Re: (feature request) Shell script within shell script

[Bart Schaefer]

On Feb 7,  3:23pm, William Park wrote:
}
} A classic solution:
} 
}     function test1 () {
}     exec 10<<"EOF"
}     ...
}     ...
}     EOF
} 	awk -f /dev/fd/10
}     exec 10<&-
}     }
} 
} I didn't have to patch anything.  I just have to keep track of fd's
} instead of external files.

Did you actually try that?  It certainly doesn't work for me.  You
can't use two-or-more-digit numbers to represent FDs for redirection:

zagzig% cat 10<<EOF  
heredoc> foo bar
heredoc> EOF 
cat: 10: No such file or directory

It works if you change 10 to 9, but it's overkill.  All you need is:

    awk -f /dev/fd/9 9<<\EOF
    ...
    ...
    EOF

And if you happen to encounter a command that requires a real seekable
file for its input, you can do this:

    awk -f =(<&9) 9<<\EOF
    ...
    ...
    EOF

However, I'd like to point out that this is not what you originally
asked for.  You asked how to create a script, complete with #! line,
and execute it, not how to feed a here-document to a command that
expects a file name.

Re: (feature request) Shell script within shell script

[William Park]

On Sat, Feb 08, 2003 at 02:31:26AM +0000, Bart Schaefer wrote:
> Did you actually try that?  It certainly doesn't work for me.  You
> can't use two-or-more-digit numbers to represent FDs for redirection:

I use Bash.  I asked this group, because I wanted to know if a cleaner
solution is possible (or easily patchable) than the classic method of
    function test1 () {
	cat >/tmp/test1 <<"EOF"
	#! ...
	...
	EOF
	chmod +x /tmp/test1
	/tmp/test1 "$@"
    }

> However, I'd like to point out that this is not what you originally
> asked for.  You asked how to create a script, complete with #! line,
> and execute it, not how to feed a here-document to a command that
> expects a file name.

It's the best I could come up with.  It's okey for now, since most
external programs (ie. awk, python) can take script file.  But, my
search continues...

-- 
William Park, Open Geometry Consulting, <opengeometry@yahoo.ca>
Linux solution for data management and processing.