convolutions

convolutions

[Ray Andrews]

Gentlemen:

     echo "$(eval echo "\${$(cat in_file)}")" >! out_file

That's the best I've been able to do expanding color variables, eg. " 
${red} " into their native " \e[31;1m " in a file.  It's not a prize 
winner as far as zsh obfuscation goes, still, can it be done more 
simply? Also, I'd not be surprised to learn that there's one of those 
ancient little utilities that already does that:

     cat in_file | ancient_utility > outfile

is there?

Re: convolutions

[ZyX]

07.11.2015, 05:17, "Ray Andrews" <rayandrews@eastlink.ca>:
> Gentlemen:
>
>      echo "$(eval echo "\${$(cat in_file)}")" >! out_file
>
> That's the best I've been able to do expanding color variables, eg. "
> ${red} " into their native " \e[31;1m " in a file. It's not a prize
> winner as far as zsh obfuscation goes, still, can it be done more
> simply? Also, I'd not be surprised to learn that there's one of those
> ancient little utilities that already does that:
>
>      cat in_file | ancient_utility > outfile
>
> is there?

“Ancient utility” is called `sed`. Specifically you are able to substitute text with the result of the evaluation of the shell code:

    echo '${red}red' | sed -r 's/\$\{(\w+)\}/zsh -c ''echo ${(%%):-%F{\1}}''/e'

Note: sed uses /bin/sh for /e and not $SHELL, so you must call zsh explicitly.
Note 2: see “SIMPLE PROMPT ESCAPES” (specifically `%F`) in `man zshmisc`. See “PARAMETER EXPANSION”/“Parameter Expansion Flags” in `man zshexpn` (specifically `%` flag). See `man zshoptions`, option `RCQUOTES` if you are wondering why I wrote `''`.

You may also do the same thing with almost every other script language. But while in perl this is mostly as verbose as in sed, most other are less oriented on text processing and will be more verbose. Also sed and awk are rather standard, perl and python also are to a less extent, everything else better be avoided in shell scripts.

Re: convolutions

[Ray Andrews]

On 11/06/2015 06:49 PM, ZyX wrote:
>
>       echo "$(eval echo "\${$(cat in_file)}")" >! out_file
>
> That's the best I've been able to do expanding color variables, eg. "
> ${red} " into their native " \e[31;1m " in a file. It's not a prize
> winner as far as zsh obfuscation goes, still, can it be done more
> simply? Also, I'd not be surprised to learn that there's one of those
> ancient little utilities that already does that:
>
>
> “Ancient utility” is called `sed`. Specifically you are able to substitute text with the result of the evaluation of the shell code:
>
>      echo '${red}red' | sed -r 's/\$\{(\w+)\}/zsh -c ''echo ${(%%):-%F{\1}}''/e'
>
>
Well sure, but that's even more opaque than what I'm doing.  I'm a grade 
three sedder but the whole point was to see if I could get zsh to handle 
this internally.

Re: convolutions

[Mikael Magnusson]

On Sat, Nov 7, 2015 at 3:15 AM, Ray Andrews <rayandrews@eastlink.ca> wrote:
> Gentlemen:
>
>     echo "$(eval echo "\${$(cat in_file)}")" >! out_file
>
> That's the best I've been able to do expanding color variables, eg. " ${red}
> " into their native " \e[31;1m " in a file.  It's not a prize winner as far
> as zsh obfuscation goes, still, can it be done more simply? Also, I'd not be
> surprised to learn that there's one of those ancient little utilities that
> already does that:
>
>     cat in_file | ancient_utility > outfile
>
> is there?

A smart thing to include when asking for a simpler way to do an
obfuscated thing is what it is you're actually trying to do. Saves
people the effort of trying to deobfuscate your code first. It's also
super unclear what you mean by "expanding $red into native".

-- 
Mikael Magnusson

Re: convolutions

[Bart Schaefer]

On Fri, Nov 6, 2015 at 6:15 PM, Ray Andrews <rayandrews@eastlink.ca> wrote:
> Gentlemen:
>
>     echo "$(eval echo "\${$(cat in_file)}")" >! out_file
>
> That's the best I've been able to do expanding color variables, eg. " ${red}
> " into their native " \e[31;1m " in a file.

Just as an aside, this is the second code example you've posted
recently that can't possibly work in practice.  I can only assume
you're mis-transcribing in an attempt to simplify whatever your real
code is.

If I understand this correctly, you have a file that contains text
which looks like

   The following ${fg[red]}text is red${reset_color} and this is not

and you want to end up with the variables expanded in out_file as the
actual escape sequences?

    print -R "${(e)$(<in_file)}" >| out_file

Of course that will also replace other possible expansions that may
appear in the text of in_file.  If you really want to replace exactly
and only the specific references to color variables, you need
something like Zyx's sed expression.

Re: convolutions

[Ray Andrews]

On 11/06/2015 09:37 PM, Mikael Magnusson wrote:
> A smart thing to include when asking for a simpler way to do an 
> obfuscated thing is what it is you're actually trying to do. Saves 
> people the effort of trying to deobfuscate your code first. It's also 
> super unclear what you mean by "expanding $red into native". 

... expanding color variables, eg. " ${red} " into their native " \e[31;1m " in a file.

That isn't clear? Variables holding color codes seem like a very routine thing  to me, there are many of them as a standard part of the shell. If you have some long string with color variables ( eg. "$_red" ) in it, and you echo the string, the colors execute, but if you save that same string to a file and 'cat' it, the color variables do not expand--it is necessary to convert them to the actual escape codes. My code snippet does that, I'm just wondering if it's as simple as it can be.

Re: convolutions

[Ray Andrews]

On 11/06/2015 09:38 PM, Bart Schaefer wrote:
> On Fri, Nov 6, 2015 at 6:15 PM, Ray Andrews <rayandrews@eastlink.ca> wrote:
>> Gentlemen:
>>
>>      echo "$(eval echo "\${$(cat in_file)}")" >! out_file
>>
>> That's the best I've been able to do expanding color variables, eg. " ${red}
>> " into their native " \e[31;1m " in a file.
> Just as an aside, this is the second code example you've posted
> recently that can't possibly work in practice.  I can only assume
> you're mis-transcribing in an attempt to simplify whatever your real
> code is.
>
I did indeed edit it a bit.  Here's the pasted actual test file:



test2 ()
{
     echo "$(eval echo "\${$(cat junk1)}")" >! junk2
     cat junk2
}

... which works as advertised.
> If I understand this correctly, you have a file that contains text
> which looks like
>
>     The following ${fg[red]}text is red${reset_color} and this is not
>
> and you want to end up with the variables expanded in out_file as the
> actual escape sequences?
>
>      print -R "${(e)$(<in_file)}" >| out_file

Does the trick, thanks, it's no shorter, but much more respectable. 
print has many tricks up it's sleeve.
>
> Of course that will also replace other possible expansions that may
> appear in the text of in_file.  If you really want to replace exactly
> and only the specific references to color variables, you needs
> something like Zyx's sed expression.
Yeah, my snippet would have the same issue so that's fine.  It seems so 
laborious tho.  Too bad there wasn't some way of forcing expansions in a 
prescribed order to avoid the echoing and evaling.
>

Re: convolutions

[Mikael Magnusson]

On Sat, Nov 7, 2015 at 7:16 AM, Ray Andrews <rayandrews@eastlink.ca> wrote:
> On 11/06/2015 09:37 PM, Mikael Magnusson wrote:
>>
>> A smart thing to include when asking for a simpler way to do an obfuscated
>> thing is what it is you're actually trying to do. Saves people the effort of
>> trying to deobfuscate your code first. It's also super unclear what you mean
>> by "expanding $red into native".
>
>
> ... expanding color variables, eg. " ${red} " into their native " \e[31;1m "
> in a file.
>
> That isn't clear?

No, because 'expanding into "\e"' means nothing useful, did you mean a
literal backslash followed by an e, or a literal escape?

> Variables holding color codes seem like a very routine
> thing  to me, there are many of them as a standard part of the shell. If you
> have some long string with color variables ( eg. "$_red" ) in it, and you
> echo the string, the colors execute, but if you save that same string to a
> file and 'cat' it, the color variables do not expand

If you store the string in a parameter and echo that parameter, the
color variables also don't expand.

>--it is necessary to
> convert them to the actual escape codes. My code snippet does that, I'm just
> wondering if it's as simple as it can be.

If you have a parameter with \e in it, and want to expand it to a
literal escape byte, the easiest way is to use ${(g::)red}.

So you have a file with "${red}" in it, and you want to read in the
file, and substitute the value of any parameter references. Moreover,
these parameters are not defined in the file, but are set in the shell
environment and you happen to know that they are string
representations of color escape sequences, and you want to expand
these as well.

Perhaps you should take a step back and redesign whatever it is you're
doing, because it really doesn't seem like a good way to handle things
:).

If your file has a semicolon in it somewhere, then your command will
run arbitrary code. Bart's example doesn't have that exact problem,
but will still execute code if there are things like $(command) in the
file.

-- 
Mikael Magnusson

Re: convolutions

[Bart Schaefer]

On Nov 6, 10:33pm, Ray Andrews wrote:
} Subject: Re: convolutions
}
} I did indeed edit it a bit.  Here's the pasted actual test file:
} 
} test2 ()
} {
}      echo "$(eval echo "\${$(cat junk1)}")" >! junk2
}      cat junk2
} }
} 
} ... which works as advertised.

That will fail with "bad substitution" for anything more than a single
word in the file "junk1".

torch% echo "this is some text" > junk1
torch% test2 ()
{
     echo "$(eval echo "\${$(cat junk1)}")" >! junk2
     cat junk2
}
torch% test2
(eval):1: bad substitution

torch% 

You can't just throw ${ } around any arbitrary thing and have it act
like a parameter expansion -- which is what is happening during the
eval when you have used \$ in the original expression:

torch% setopt xtrace verbose
torch% test2
test2
+Src/zsh:25> test2
+test2:2> cat junk1
+test2:2> eval echo '${this is some text}'
(eval):1: bad substitution
+test2:2> echo ''
+test2:3> cat junk2

torch% 

-- and you don't even need it there in this case:

torch% test2 ()
{
     echo "$(eval echo "$(cat junk1)")" >! junk2 
     cat junk2
}
torch% some=OTHER
torch% echo 'this is $some text' >| junk1
torch% test2
this is OTHER text
torch% 

} >      print -R "${(e)$(<in_file)}" >| out_file
} 
} Does the trick, thanks, it's no shorter, but much more respectable. 
} print has many tricks up it's sleeve.

Sigh.  No, "-R" is explicitly telling "print" not to do any tricks at
all.  It's the (e) parameter expansion flag that's doing the work.

Re: convolutions

[Bart Schaefer]

On Nov 7,  1:32am, Bart Schaefer wrote:
}
} -- and you don't even need it there in this case:
} 
} torch% test2 ()
} {
}      echo "$(eval echo "$(cat junk1)")" >! junk2 
}      cat junk2
} }

Although, to be clear, the above isn't really correct either, because
the "eval" is going to remove the quotes around the contents of junk1,
so you can end up actually executing parts of junk1 as code, as Mikael
pointed out.

Re: convolutions

[Ray Andrews]

On 11/06/2015 10:38 PM, Mikael Magnusson wrote:
> No, because 'expanding into "\e"' means nothing useful, did you mean a
> literal backslash followed by an e, or a literal escape?

It's not arguable sir.  If you found it vague, then you found it vague, 
but to me:

      " \e[31;1m "

... is very obviously a color code which I hold in my variable " ${red} 
" and I want to expand it inside a file.

> If you store the string in a parameter and echo that parameter, the 
> color variables also don't expand.
>> So you have a file with "${red}" in it, and you want to read in the
>> file, and substitute the value of any parameter references.

Exactly.
>> Moreover,
>> these parameters are not defined in the file, but are set in the shell
>> environment and you happen to know that they are string
>> representations of color escape sequences, and you want to expand
>> these as well.

Of course. Who but me knows what's in my environment variables?
>> If your file has a semicolon in it somewhere, then your command will
>> run arbitrary code. Bart's example doesn't have that exact problem,
>> but will still execute code if there are things like $(command) in the
>> file.
>>
Yes, I know how dangerous eval is, that's why I'm asking for a better 
idea--some way of expanding parameters  inside a file.  Not just color 
variables really, any variables.

Re: convolutions

[Ray Andrews]

On 11/07/2015 01:32 AM, Bart Schaefer wrote:

> That will fail with "bad substitution" for anything more than a single
> word in the file "junk1".

test2 ()
{
     rm junk2
     cat junk1
     echo "$(eval echo "\${$(cat junk1)}")" >! junk2
     echo
     cat junk2
}

$ . ./test: test2
12345678-10-345678-20-345678-30-345678-40-345678-50-345678-60-345678-70-345678-80-345678-90-345678-100
         let    me    not    to    the marriage${red}of true 
minds${yellow}    admit impediments.${reset}
         Love    is    not    love which    alters${green} when ${reset}

[ in glorious color: ]

10-345678-20-345678-30-345678-40-345678-50-345678-60-345678-70-345678-80-345678-90-345678-100
         let    me    not    to    the marriageof true minds    admit 
impediments.
         Love    is    not    love which    alters when


The file is for a deliberate torture test of color handling and tab 
expansion, so pardon the ugliness, it's the point of it.  But it's 
working here, tho the first part of my 'measuring string' is cut off for 
some reason.  I don't like this, I hate it, it's dangerous. Don't tell 
me it sux, I know it sux.  But I would like some way of performing 
expansions inside a file without any gottchas.  Surely that's simple at 
least to verbalize:

     $ cat in_file | expand_all_parameters > out_file

Re: convolutions

[Martin Vaeth]

Bart Schaefer <schaefer@brasslantern.com> wrote:
> If you really want to replace exactly
> and only the specific references to color variables, you need
> something like Zyx's sed expression.

Or envsubst from gettext.
In fact, envsubst was exactly written for the purpose of
*safely* expanding variables (and *only* that).

Re: convolutions

[ZyX]

07.11.2015, 18:37, "Ray Andrews" <rayandrews@eastlink.ca>:
> On 11/06/2015 10:38 PM, Mikael Magnusson wrote:
>>  No, because 'expanding into "\e"' means nothing useful, did you mean a
>>  literal backslash followed by an e, or a literal escape?
>
> It's not arguable sir. If you found it vague, then you found it vague,
> but to me:
>
>       " \e[31;1m "
>
> ... is very obviously a color code which I hold in my variable " ${red}
> " and I want to expand it inside a file.
>
>>  If you store the string in a parameter and echo that parameter, the
>>  color variables also don't expand.
>>>  So you have a file with "${red}" in it, and you want to read in the
>>>  file, and substitute the value of any parameter references.
>
> Exactly.
>>>  Moreover,
>>>  these parameters are not defined in the file, but are set in the shell
>>>  environment and you happen to know that they are string
>>>  representations of color escape sequences, and you want to expand
>>>  these as well.
>
> Of course. Who but me knows what's in my environment variables?
>>>  If your file has a semicolon in it somewhere, then your command will
>>>  run arbitrary code. Bart's example doesn't have that exact problem,
>>>  but will still execute code if there are things like $(command) in the
>>>  file.
>
> Yes, I know how dangerous eval is, that's why I'm asking for a better
> idea--some way of expanding parameters inside a file. Not just color
> variables really, any variables.

I have no idea why you need such a thing, and need this *in zsh*. This is a job for a template engine, numerous of which exist for every script language out there used for writing something for web (which does not include zsh). Otherwise there already were three solutions, with different downsides. Also see http://stackoverflow.com/questions/2914220/bash-templating-how-to-build-configuration-files-from-templates-with-bash/2916159?noredirect=1#comment46735969_2916159, this can be ported to zsh very easily (note the comments).

Still asking this here looks either like an XY problem or wrong choice of instruments.

Re: convolutions

[Bart Schaefer]

On Nov 7,  8:09am, Ray Andrews wrote:
} Subject: Re: convolutions
}
} $ . ./test: test2
} 12345678-10-345678-20-345678-30-345678-40-345678-50-345678-60-345678-70-345678-80-345678-90-345678-100
} 
} [ in glorious color: ]
} 
} 10-345678-20-345678-30-345678-40-345678-50-345678-60-345678-70-345678-80-345678-90-345678-100

[...]

} working here, tho the first part of my 'measuring string' is cut off for 
} some reason.

    "\${$(cat junk1)}"

becomes

    ${12345678-...}

which is the syntax for "the 12345678th positional parameter, or if that
is not set, then ..." and your expansion "works" entirely by accident
but lops off the 12345678 from the front.

So your stress test isn't anywhere near as stressful as you thought.

} But I would like some way of performing 
} expansions inside a file without any gottchas.  Surely that's simple at 
} least to verbalize:
} 
}      $ cat in_file | expand_all_parameters > out_file

There's no shell operation for "expand parameters but not any of the
other kinds of expansions you know about."  If what you want is some
kind of text replacement, you should be writing in_file in a text
processing language (HTML maybe?) rather than in shell syntax.

Re: convolutions

[Ray Andrews]

On 11/07/2015 09:33 AM, Bart Schaefer wrote:
> So your stress test isn't anywhere near as stressful as you thought.
I have no faith in what I was trying to do, I was just looking for a 
useful method and that was an experiment.  I quite understand that eval 
is far to dangerous to really be useful for what  I was trying to do there.

> There's no shell operation for "expand parameters but not any of the
> other kinds of expansions you know about."  If what you want is some
> kind of text replacement, you should be writing in_file in a text
> processing language (HTML maybe?) rather than in shell syntax.
>
But I should stop whining, cuz I've had a most satisfactory answer:

$ print -R "${(e)$(<junk1)}" >| junk2
$ cat junk2

... just what I want unless there's some gottcha with that, but I doubt 
it. So debating the demerits of a failed idea that I know has failed is 
pointless.  Thanks all.

Re: convolutions

[Ray Andrews]

On 11/07/2015 09:17 AM, Martin Vaeth wrote:
> Bart Schaefer <schaefer@brasslantern.com> wrote:
>> If you really want to replace exactly
>> and only the specific references to color variables, you need
>> something like Zyx's sed expression.
> Or envsubst from gettext.
> In fact, envsubst was exactly written for the purpose of
> *safely* expanding variables (and *only* that).
>
>
Sounds like precisely the thing. Alas, I tried it, and it strips the 
parameters but does not convert them unless each parameter is explicitly 
exported first, which makes the thing a bit cumbersome for general use. 
Is there a work-around?

Re: convolutions

[Bart Schaefer]

On Nov 8,  7:27am, Ray Andrews wrote:
}
} > Or envsubst from gettext.
} >
} Sounds like precisely the thing. Alas, I tried it, and it strips the 
} parameters but does not convert them unless each parameter is explicitly 
} exported first, which makes the thing a bit cumbersome for general use. 
} Is there a work-around?

setopt ALL_EXPORT before you assign any of the parameters involved.  And
then probably unsetopt afterward ...

Re: convolutions

[Ray Andrews]

On 11/08/2015 11:14 AM, Bart Schaefer wrote:
> On Nov 8,  7:27am, Ray Andrews wrote:
> }
> } > Or envsubst from gettext.
> } >
> } Sounds like precisely the thing. Alas, I tried it, and it strips the
> } parameters but does not convert them unless each parameter is explicitly
> } exported first, which makes the thing a bit cumbersome for general use.
> } Is there a work-around?
>
> setopt ALL_EXPORT before you assign any of the parameters involved.  And
> then probably unsetopt afterward ...
>
Ok thanks. Mind, it looks like we have perfectly satisfactory solutions 
entirely within zsh itself.

Re: convolutions

[Martin Vaeth]

Ray Andrews <rayandrews@eastlink.ca> wrote:
> On 11/07/2015 09:17 AM, Martin Vaeth wrote:
>> In fact, envsubst was exactly written for the purpose of
>> *safely* expanding variables (and *only* that).
>>
> Sounds like precisely the thing. Alas, I tried it, and it strips the
> parameters but does not convert them unless each parameter is explicitly
> exported first

eval_gettext() from gettext.sh does it like this
(where $1 contains the text):

(export PATH `envsubst --variables "$1"`; envsubst "$1")