YAML metadata files' override order in a single command line

YAML metadata files' override order in a single command line

[K4zuki]

[-- Attachment #1.1: Type: text/plain, Size: 1034 bytes --]

Hello,

I have a question regarding multiple yaml input in one command line.

I have two yaml metadata files: system/config.yaml and user/config.yaml. 
They have some overlaps in entries.
I tested the following command with expectation that user/config.yaml 
overrides system/config.yaml:

pandoc -s -t markdown system/config.yaml user/config.yaml


but in fact system/config.yaml wins. What system/config.yaml does not have 
is merged as expected (and vice versa).

So the question is when multiple yaml with entry overlap given in a single 
command line, what kind of override order is applied?

Thanks && Regards,
Kazuki

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/2921e399-c659-4cba-9357-577c2b05aece%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2788 bytes --]

FEEDBACK WANTED: Re: YAML metadata files' override order in a single command line

[John MacFarlane]

It is as you discovered: the first one wins, currently.
This is somewhat arbitrary and could be changed.

Using foldl1 instead of foldr1 in line 230 of Text.Pandoc.App
would give the other behavior.

The change allowing multiple metadata files is not yet in
any released version, so it would be easy to change this.
I'd welcome feedback from people.

consistency is good, so we should also ask:

- which takes precedence if you do
  --metadata foo=1 --metadata foo=2
  on the command line?

- which takes precedence if you have two YAML metadata blocks
  in a markdown file, and they set the same field?

K4zuki <k.yamamoto.08136891-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> Hello,
>
> I have a question regarding multiple yaml input in one command line.
>
> I have two yaml metadata files: system/config.yaml and user/config.yaml. 
> They have some overlaps in entries.
> I tested the following command with expectation that user/config.yaml 
> overrides system/config.yaml:
>
> pandoc -s -t markdown system/config.yaml user/config.yaml
>
>
> but in fact system/config.yaml wins. What system/config.yaml does not have 
> is merged as expected (and vice versa).
>
> So the question is when multiple yaml with entry overlap given in a single 
> command line, what kind of override order is applied?
>
> Thanks && Regards,
> Kazuki

Re: FEEDBACK WANTED: Re: YAML metadata files' override order in a single command line

[BPJ]

In the manual <https://pandoc.org/MANUAL.html#metadata-blocks>:

 > A document may contain multiple metadata blocks. The metadata fields 
will be combined through a left-biased union: if two metadata blocks 
attempt to set the same field, the value from the first block will be taken.

That is, if I understand correctly: if there are two or more YAML 
blocks, regardless of whether they are passed as separate input files or 
embedded in other input files, and some key occurs at the top level of 
two or more of those YAML blocks, the value from the first YAML block 
which contains the key "wins".

It *is* IMO a little counter-intuitive, i.e. I would have expected 
right-biassed/last occurrence wins, but the question is whether it is 
too late to change it or not.  People may well have work flows which 
rely on the current behavior. For example I have several Makefiles with 
rules which pass one output-format-specific and one general metadata 
file, and the order in which they are passed assumes the current 
behavior, with the format-specific one before the general, so that the 
format-specific one wins in case of a clash.  I could of course reverse 
the order of those input files as needed, but I would have to
always leave a comment saying that the metadata files are already in
right-biased order, and I would always have to check an existing 
Makefile to see if it needs updating.  I *know* that I'll forget to do 
so after a while!

On 2019-10-02 20:48, John MacFarlane wrote:
> 
> It is as you discovered: the first one wins, currently.
> This is somewhat arbitrary and could be changed.
> 
> Using foldl1 instead of foldr1 in line 230 of Text.Pandoc.App
> would give the other behavior.
> 
> The change allowing multiple metadata files is not yet in
> any released version, so it would be easy to change this.
> I'd welcome feedback from people.
> 
> consistency is good, so we should also ask:
> 
> - which takes precedence if you do
>    --metadata foo=1 --metadata foo=2
>    on the command line?
> 
> - which takes precedence if you have two YAML metadata blocks
>    in a markdown file, and they set the same field?
> 
> K4zuki <k.yamamoto.08136891-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
> 
>> Hello,
>>
>> I have a question regarding multiple yaml input in one command line.
>>
>> I have two yaml metadata files: system/config.yaml and user/config.yaml.
>> They have some overlaps in entries.
>> I tested the following command with expectation that user/config.yaml
>> overrides system/config.yaml:
>>
>> pandoc -s -t markdown system/config.yaml user/config.yaml
>>
>>
>> but in fact system/config.yaml wins. What system/config.yaml does not have
>> is merged as expected (and vice versa).
>>
>> So the question is when multiple yaml with entry overlap given in a single
>> command line, what kind of override order is applied?
>>
>> Thanks && Regards,
>> Kazuki
> 

Re: FEEDBACK WANTED: Re: YAML metadata files' override order in a single command line

[Gareth Stockwell]

[-- Attachment #1: Type: text/plain, Size: 4659 bytes --]

>
> It *is* IMO a little counter-intuitive, i.e. I would have expected
> right-biassed/last occurrence wins, but the question is whether it is
> too late to change it or not.  People may well have work flows which
> rely on the current behavior. For example I have several Makefiles with
> rules which pass one output-format-specific and one general metadata
> file, and the order in which they are passed assumes the current
> behavior, with the format-specific one before the general, so that the
> format-specific one wins in case of a clash.


I have exactly the same workflow, but would be happy to fix it to
accommodate a change to the more intuitive (to me) right-biased model.

Gareth



On Wed, Oct 2, 2019 at 8:45 PM BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> In the manual <https://pandoc.org/MANUAL.html#metadata-blocks>:
>
>  > A document may contain multiple metadata blocks. The metadata fields
> will be combined through a left-biased union: if two metadata blocks
> attempt to set the same field, the value from the first block will be
> taken.
>
> That is, if I understand correctly: if there are two or more YAML
> blocks, regardless of whether they are passed as separate input files or
> embedded in other input files, and some key occurs at the top level of
> two or more of those YAML blocks, the value from the first YAML block
> which contains the key "wins".
>
> It *is* IMO a little counter-intuitive, i.e. I would have expected
> right-biassed/last occurrence wins, but the question is whether it is
> too late to change it or not.  People may well have work flows which
> rely on the current behavior. For example I have several Makefiles with
> rules which pass one output-format-specific and one general metadata
> file, and the order in which they are passed assumes the current
> behavior, with the format-specific one before the general, so that the
> format-specific one wins in case of a clash.  I could of course reverse
> the order of those input files as needed, but I would have to
> always leave a comment saying that the metadata files are already in
> right-biased order, and I would always have to check an existing
> Makefile to see if it needs updating.  I *know* that I'll forget to do
> so after a while!
>
> On 2019-10-02 20:48, John MacFarlane wrote:
> >
> > It is as you discovered: the first one wins, currently.
> > This is somewhat arbitrary and could be changed.
> >
> > Using foldl1 instead of foldr1 in line 230 of Text.Pandoc.App
> > would give the other behavior.
> >
> > The change allowing multiple metadata files is not yet in
> > any released version, so it would be easy to change this.
> > I'd welcome feedback from people.
> >
> > consistency is good, so we should also ask:
> >
> > - which takes precedence if you do
> >    --metadata foo=1 --metadata foo=2
> >    on the command line?
> >
> > - which takes precedence if you have two YAML metadata blocks
> >    in a markdown file, and they set the same field?
> >
> > K4zuki <k.yamamoto.08136891-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
> >
> >> Hello,
> >>
> >> I have a question regarding multiple yaml input in one command line.
> >>
> >> I have two yaml metadata files: system/config.yaml and user/config.yaml.
> >> They have some overlaps in entries.
> >> I tested the following command with expectation that user/config.yaml
> >> overrides system/config.yaml:
> >>
> >> pandoc -s -t markdown system/config.yaml user/config.yaml
> >>
> >>
> >> but in fact system/config.yaml wins. What system/config.yaml does not
> have
> >> is merged as expected (and vice versa).
> >>
> >> So the question is when multiple yaml with entry overlap given in a
> single
> >> command line, what kind of override order is applied?
> >>
> >> Thanks && Regards,
> >> Kazuki
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/4998a075-0f70-4b30-7406-135d2e160b97%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAGewFGCu%3DYNfiYxj1HYb-H3ewnLPN0LNe2PSccKNaJi6TWsk%3DQ%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 6401 bytes --]

Re: FEEDBACK WANTED: Re: YAML metadata files' override order in a single command line

[John MacFarlane]

Yeah, it's true. Even though the ability to specify
--metadata-file multiple times is new, you were always
able to specify multiple YAML files when reading pandoc
markdown (since these are valid pandoc markdown files).
And changing the precedence at this point would break
too much.

Also, consistency with the way multiple YAML blocks in
a document works recommends the "first wins" strategy.

I think these are pretty strong reasons for keeping
the order as it is, even if it is a bit counterintuitive.

BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> In the manual <https://pandoc.org/MANUAL.html#metadata-blocks>:
>
>  > A document may contain multiple metadata blocks. The metadata fields 
> will be combined through a left-biased union: if two metadata blocks 
> attempt to set the same field, the value from the first block will be taken.
>
> That is, if I understand correctly: if there are two or more YAML 
> blocks, regardless of whether they are passed as separate input files or 
> embedded in other input files, and some key occurs at the top level of 
> two or more of those YAML blocks, the value from the first YAML block 
> which contains the key "wins".
>
> It *is* IMO a little counter-intuitive, i.e. I would have expected 
> right-biassed/last occurrence wins, but the question is whether it is 
> too late to change it or not.  People may well have work flows which 
> rely on the current behavior. For example I have several Makefiles with 
> rules which pass one output-format-specific and one general metadata 
> file, and the order in which they are passed assumes the current 
> behavior, with the format-specific one before the general, so that the 
> format-specific one wins in case of a clash.  I could of course reverse 
> the order of those input files as needed, but I would have to
> always leave a comment saying that the metadata files are already in
> right-biased order, and I would always have to check an existing 
> Makefile to see if it needs updating.  I *know* that I'll forget to do 
> so after a while!
>
> On 2019-10-02 20:48, John MacFarlane wrote:
>> 
>> It is as you discovered: the first one wins, currently.
>> This is somewhat arbitrary and could be changed.
>> 
>> Using foldl1 instead of foldr1 in line 230 of Text.Pandoc.App
>> would give the other behavior.
>> 
>> The change allowing multiple metadata files is not yet in
>> any released version, so it would be easy to change this.
>> I'd welcome feedback from people.
>> 
>> consistency is good, so we should also ask:
>> 
>> - which takes precedence if you do
>>    --metadata foo=1 --metadata foo=2
>>    on the command line?
>> 
>> - which takes precedence if you have two YAML metadata blocks
>>    in a markdown file, and they set the same field?
>> 
>> K4zuki <k.yamamoto.08136891-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>> 
>>> Hello,
>>>
>>> I have a question regarding multiple yaml input in one command line.
>>>
>>> I have two yaml metadata files: system/config.yaml and user/config.yaml.
>>> They have some overlaps in entries.
>>> I tested the following command with expectation that user/config.yaml
>>> overrides system/config.yaml:
>>>
>>> pandoc -s -t markdown system/config.yaml user/config.yaml
>>>
>>>
>>> but in fact system/config.yaml wins. What system/config.yaml does not have
>>> is merged as expected (and vice versa).
>>>
>>> So the question is when multiple yaml with entry overlap given in a single
>>> command line, what kind of override order is applied?
>>>
>>> Thanks && Regards,
>>> Kazuki
>> 
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/4998a075-0f70-4b30-7406-135d2e160b97%40gmail.com.