Commenting and large pipelines

Commenting and large pipelines

[Zach Riggle]

[-- Attachment #1: Type: text/plain, Size: 1694 bytes --]

Let's assume I have a series of large commands which need to be piped to
eachother.

Additionally, the pipeline itself must behave differently based on some
global flags or arguments.

There's a lot of way to make logic for this (e.g. functions), but one that
I've become a fan of is something like this below.

    cmd1=( some long command $FLAG bar baz )
    cmd2=( grep -E "$@" )
    cmd3=(
        grep
            -v # Don't match these
            -e 'fizz|buzz'
    )

    "${cmd1[@]}" | "${cmd2[@]}" | "${cmd3[@]}"

However, conditional logic makes this annoying and a bit opaque --
functions definitely excel here.

Another issue is that of adding comments to the arguments and the pipeline
itself -- lines that end with '\' cannot contain comments.  Functions
obviously can contain comments within themselves.

What I've discovered is that something like this works out pretty well...

    { printf "%s\n" a b foo c bar d fizz XfooX XbuzzX } |
    { grep -E 'foo|bar' } |
    {
        # If the user specified '--no-fizz-buzz', remove those entries
        if (( NO_FIZZ_BUZZ )); then
            grep -vE 'fizz|buzz'
        else
        # Pass through everything
            cat
        fi
    }

I have a few questions about this construct.

    1. Am I insane for doing this?
    2. In what ways is this a terrible idea?
    3. Is the use of {} better than ()?
    4. How much of a performance hit does this make, versus hand-writing a
different pipeline?
    5. Are there any ways to improve this?  For example, replacing 'cat' in
the default case.

Thanks for the attention, just curious what everybody thinks about this
abuse of pipelines and conditional logic.

*Zach Riggle*

[-- Attachment #2: Type: text/html, Size: 2224 bytes --]

Re: Commenting and large pipelines

[Lawrence Velázquez]

> On May 11, 2022, at 2:37 AM, Zach Riggle <zachriggle@gmail.com> wrote:
> What I've discovered is that something like this works out pretty well...
> 
>     { printf "%s\n" a b foo c bar d fizz XfooX XbuzzX } |
>     { grep -E 'foo|bar' } |
>     {
>         # If the user specified '--no-fizz-buzz', remove those entries
>         if (( NO_FIZZ_BUZZ )); then
>             grep -vE 'fizz|buzz'
>         else
>         # Pass through everything
>             cat
>         fi
>     }
> 
> I have a few questions about this construct.
> 
>     1. Am I insane for doing this?
>     2. In what ways is this a terrible idea?

There isn’t anything wrong with (or novel about) using grouped commands as pipeline stages.

(However, in your example the braces in the first two stages are unnecessary and distracting. They make me wonder whether there’s some obscure reason you added them.)

>     3. Is the use of {} better than ()?

I can’t think of a compelling reason to use () other than to force the final stage to run in a subshell. (I believe the other stages run in subshells regardless.)

> Thanks for the attention, just curious what everybody thinks about this abuse of pipelines and conditional logic.

It’s not abuse. It’s entirely valid to use complex commands as pipeline stages.

Now, whether or not that is more legible or maintainable than the alternatives is a different, context-dependent question.

-- 
vq
Sent from my iPhone

Re: Commenting and large pipelines

[Bart Schaefer]

In addition to what Lawrence said ...

On Tue, May 10, 2022 at 11:35 PM Zach Riggle <zachriggle@gmail.com> wrote:
>
>     3. Is the use of {} better than ()?

In most cases it won't matter.  Putting something on the left side of
a pipe in zsh already starts it in a subshell, and most such subshells
are subject to an optimization such that zsh will exec the last
command to replace the shell process.

On the right side of a pipe, however commands in { } stay in the
current shell whereas those in ( ) run in a forked subshell.

>     4. How much of a performance hit does this make, versus hand-writing a different pipeline?

It's definitely going to perform better than a pipeline built by
assigning and expanding array parameters.  Otherwise the performance
difference should be minimal.

>     5. Are there any ways to improve this?  For example, replacing 'cat' in the default case.

You're not going to do better than "cat" in that context.