What does REDIRF_FROM_HEREDOC flag represent?

What does REDIRF_FROM_HEREDOC flag represent?

[Jett Husher]

Good day!

I was peeking at the source code and found REDIRF_FROM_HEREDOC flag. The comments say that it's for a here-string that came from a here-document but I'm still not clear how this looks exactly. Is it that every line in heredoc is a here-string? I've tried to read the part in parser.c where this flag is being set, but couldn't make sense of it since I'm not at all acquainted with C or the codebase as a whole.

Would very much appreciate the explanation.

- Jett

Re: What does REDIRF_FROM_HEREDOC flag represent?

[Daniel Shahaf]

Jett Husher wrote on Thu, Oct 21, 2021 at 09:53:42 +0000:
> Good day!
> 
> I was peeking at the source code

Welcome :)

> and found REDIRF_FROM_HEREDOC flag.  The comments say that it's for a
> here-string that came from a here-document but I'm still not clear how
> this looks exactly.

A here-string is this thing:

    item(tt(<<<) var(word))(
    Perform shell expansion on var(word) and pass the result
    to standard input.  This is known as a em(here-string).
    Compare the use of var(word) in here-documents above, where var(word)
    does not undergo shell expansion.
    )

Example:

    % nl -ba <<< $'foo\nbar'  
         1	foo
         2	bar
    % 

> Is it that every line in heredoc is a here-string?

No.

> I've tried to read the part in parser.c where this flag is being set,
> but couldn't make sense of it since I'm not at all acquainted with C
> or the codebase as a whole.

I think what happens is this:

- After parsing, a here-doc is represented as a here-string with
  REDIRF_FROM_HEREDOC set

- text.c looks for that bit and does the reverse transformation, so
  stuff like
  .
      f() { foo <<< lorem; bar <<'EOF'
      ipsum
      EOF
      }
  .
  gets emitted by `which f` using the same type as input redirections as
  it used when the shell first lexed/parsed it.

- exec.c appends a newline to here-strings' data before passing it to
  commands; see:
  .
      % xxd <<<foo 
      00000000: 666f 6f0a                                foo.

- There are no other mentions of REDIRF_FROM_HEREDOC.

It did help that I already knew what each file does in general, but
yeah, I don't have access to zsh-with-comments.git either ☹

Cheers,

Daniel

> Would very much appreciate the explanation.
> 
> - Jett
> 

Re: What does REDIRF_FROM_HEREDOC flag represent?

[Jett Husher]

On Thursday, October 21st, 2021 at 16:39, Daniel Shahaf
<d.s@daniel.shahaf.name> wrote:

> Welcome :)

Thanks! c:

> > Is it that every line in heredoc is a here-string?
>
> No.

I got that idea from the fact that heredoc and here-string, despite having
separate entries in redirection types enum in zsh.h, share the same case in
exec.c (line 3687) and are both acquired with getherestr function. I saw that
just above this very function is gethere fn with a comment that it converts a
heredoc to a here-string, but I thought that the flag would be set there as
well.

I guess I just got confused and a comment from getherestr that reads "For
here-strings from here documents, we use the original" got me an idea that a
here-string can somehow exist in heredoc (?????) in a very strange fashion. So
I asked to make sure.

Your explanation definitely helped, thank you a ton for that, Daniel!

> It did help that I already knew what each file does in general, but
> yeah, I don't have access to zsh-with-comments.git either ☹

It's surprisingly easy to navigate in the code base, even without entirely
understanding C syntax. So good job keeping it organized!

- Jett

Re: What does REDIRF_FROM_HEREDOC flag represent?

[Daniel Shahaf]

Jett Husher wrote on Fri, 22 Oct 2021 04:42 +00:00:
> On Thursday, October 21st, 2021 at 16:39, Daniel Shahaf
> <d.s@daniel.shahaf.name> wrote:
>> > Is it that every line in heredoc is a here-string?
>>
>> No.
>
> I got that idea from the fact that heredoc and here-string, despite having
> separate entries in redirection types enum in zsh.h, share the same case in
> exec.c (line 3687) and are both acquired with getherestr function. I saw that
> just above this very function is gethere fn with a comment that it converts a
> heredoc to a here-string, but I thought that the flag would be set there as
> well.
>
> I guess I just got confused and a comment from getherestr that reads "For
> here-strings from here documents, we use the original" got me an idea that a
> here-string can somehow exist in heredoc (?????) in a very strange fashion. So
> I asked to make sure.
>

Thanks for explaining what derailed you.  Sounds like we should add some
comments to clarify what's going on.  I don't immediately have a concrete
proposal; anyone wants to do the honours?

Thanks again,

Daniel