Reading a multi line yaml key - results in text surrounded in html <p> tags?

Reading a multi line yaml key - results in text surrounded in html <p> tags?

[Russ]

[-- Attachment #1.1: Type: text/plain, Size: 852 bytes --]

pandoc 2.17.0.1
Reading a multi line YAML front matter and outputing the text now gets a 
text string with html <p> tags.

For example
```yaml
volume: | Volume 16: A Subtitle ```
Should produce:
"Volume 16: A Subtitle"

But now produces:
"<p>Volume 16: A Subtitle</p>"

Is there a way to avoid these html tags or is it a bug? I cannot say for 
sure that this has changed in this specific version, but it is certainly 
quite a recent change.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/e17cd72d-2900-476c-a945-77d9e2e722bcn%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1948 bytes --]

Re: Reading a multi line yaml key - results in text surrounded in html <p> tags?

[John MacFarlane]

It has worked this way for a long time, hasn't it?

% pandoc -t native -s
---
foo: |
  multiline
bar: one line
...
Pandoc
  Meta
    { unMeta =
        fromList
          [ ( "bar" , MetaInlines [ Str "one" , Space , Str "line" ] )
          , ( "foo" , MetaBlocks [ Para [ Str "multiline" ] ] )
          ]
    }
  []

Mulitline content getns parsed as blocks, single-line as inlines.

Russ <info-ckBjHRpK5r9MPTj2movljOTW4wlIGRCZ@public.gmane.org> writes:

> pandoc 2.17.0.1
> Reading a multi line YAML front matter and outputing the text now gets a 
> text string with html <p> tags.
>
> For example
> ```yaml
> volume: | Volume 16: A Subtitle ```
> Should produce:
> "Volume 16: A Subtitle"
>
> But now produces:
> "<p>Volume 16: A Subtitle</p>"
>
> Is there a way to avoid these html tags or is it a bug? I cannot say for 
> sure that this has changed in this specific version, but it is certainly 
> quite a recent change.
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/e17cd72d-2900-476c-a945-77d9e2e722bcn%40googlegroups.com.

Re: Reading a multi line yaml key - results in text surrounded in html <p> tags?

[John MacFarlane]

One workaround is to use a Lua filter that
applies pandoc.utils.stringify to the contents of the
'volume' field in metadata.

Russ <info-ckBjHRpK5r9MPTj2movljOTW4wlIGRCZ@public.gmane.org> writes:

> pandoc 2.17.0.1
> Reading a multi line YAML front matter and outputing the text now gets a 
> text string with html <p> tags.
>
> For example
> ```yaml
> volume: | Volume 16: A Subtitle ```
> Should produce:
> "Volume 16: A Subtitle"
>
> But now produces:
> "<p>Volume 16: A Subtitle</p>"
>
> Is there a way to avoid these html tags or is it a bug? I cannot say for 
> sure that this has changed in this specific version, but it is certainly 
> quite a recent change.
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/e17cd72d-2900-476c-a945-77d9e2e722bcn%40googlegroups.com.

Re: Reading a multi line yaml key - results in text surrounded in html <p> tags?

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 2272 bytes --]

Or if needed use single quotes around the value to protect punctuation
marks, noting that to include a single quote in a YAML single-quoted string
you use two single quotes, as in `'don''t'`.

Den tors 20 jan. 2022 22:42John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> skrev:

>
> One workaround is to use a Lua filter that
> applies pandoc.utils.stringify to the contents of the
> 'volume' field in metadata.
>
> Russ <info-ckBjHRpK5r9MPTj2movljOTW4wlIGRCZ@public.gmane.org> writes:
>
> > pandoc 2.17.0.1
> > Reading a multi line YAML front matter and outputing the text now gets a
> > text string with html <p> tags.
> >
> > For example
> > ```yaml
> > volume: | Volume 16: A Subtitle ```
> > Should produce:
> > "Volume 16: A Subtitle"
> >
> > But now produces:
> > "<p>Volume 16: A Subtitle</p>"
> >
> > Is there a way to avoid these html tags or is it a bug? I cannot say for
> > sure that this has changed in this specific version, but it is certainly
> > quite a recent change.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "pandoc-discuss" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/e17cd72d-2900-476c-a945-77d9e2e722bcn%40googlegroups.com
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/yh480ksfti2igy.fsf%40johnmacfarlane.net
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhBEViWZsLRRtAr78yn3Q3fSZswg85vut21vkrpfsvszXQ%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 3668 bytes --]

Re: Reading a multi line yaml key - results in text surrounded in html <p> tags?

[Russ]

[-- Attachment #1.1: Type: text/plain, Size: 3383 bytes --]

Firstly, thanks to everyone for their help and suggestions. Sorry John, I 
could not go back to an older version (homebrew). So I guess something else 
has changed? Perhaps a fuller explanation might help:

What we are doing is extracting some YAML frontmatter metadata into 
variables for a bash script. We use the following (simplified) command 
pandoc --lua-filter *remove-meta-html.lua* -f markdown+yaml_metadata_block 
 book-metadata-file.md --template *meta-data.template*  --to html -o 
file.tmp

*remove-meta-html.lua*
function Pandoc(doc)
 for k,v in pairs(doc.meta) do
  if v.t == 'MetaBlocks' then
   for i,x in ipairs(v) do
     if x.t == 'Para' then v[i] = pandoc.Plain(v[i].c) end
   end 
  end
 end
 return doc
end


The *meta-data.template* has lines like:
$if(volume)$BOOK_VOLUME="$volume$"$endif$

We then read the variables from the created file.tmp. 

This has worked for some time, can anyone one see anything that would need 
to be corrected or could have changed in recent versions?
On Friday, 21 January 2022 at 08:24:36 UTC BP wrote:

> Or if needed use single quotes around the value to protect punctuation 
> marks, noting that to include a single quote in a YAML single-quoted string 
> you use two single quotes, as in `'don''t'`.
>
> Den tors 20 jan. 2022 22:42John MacFarlane <j...-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> skrev:
>
>>
>> One workaround is to use a Lua filter that
>> applies pandoc.utils.stringify to the contents of the
>> 'volume' field in metadata.
>>
>> Russ <in...-ckBjHRpK5r9MPTj2movljOTW4wlIGRCZ@public.gmane.org> writes:
>>
>> > pandoc 2.17.0.1
>> > Reading a multi line YAML front matter and outputing the text now gets 
>> a 
>> > text string with html <p> tags.
>> >
>> > For example
>> > ```yaml
>> > volume: | Volume 16: A Subtitle ```
>> > Should produce:
>> > "Volume 16: A Subtitle"
>> >
>> > But now produces:
>> > "<p>Volume 16: A Subtitle</p>"
>> >
>> > Is there a way to avoid these html tags or is it a bug? I cannot say 
>> for 
>> > sure that this has changed in this specific version, but it is 
>> certainly 
>> > quite a recent change.
>> >
>> > -- 
>> > You received this message because you are subscribed to the Google 
>> Groups "pandoc-discuss" group.
>> > To unsubscribe from this group and stop receiving emails from it, send 
>> an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> > To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pandoc-discuss/e17cd72d-2900-476c-a945-77d9e2e722bcn%40googlegroups.com
>> .
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pandoc-discuss/yh480ksfti2igy.fsf%40johnmacfarlane.net
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/cef14d70-eda9-4555-85ec-a39e71fabf29n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 5723 bytes --]

Re: Reading a multi line yaml key - results in text surrounded in html <p> tags?

[Albert Krewinkel]

Russ <info-ckBjHRpK5r9MPTj2movljOTW4wlIGRCZ@public.gmane.org> writes:

>    remove-meta-html.lua
>
>    function Pandoc(doc)
>     for k,v in pairs(doc.meta) do
>      if v.t == 'MetaBlocks' then
>       for i,x in ipairs(v) do
>         if x.t == 'Para' then v[i] = pandoc.Plain(v[i].c) end
>       end
>      end
>     end
>     return doc
>    end

Looks like you are affected by this change in pandoc 2.17:

> Elements of type `MetaValue` are no longer pushed as values which have
> `.t` and `.tag` properties. This was already true for `MetaString` and
> `MetaBool` values, which are still marshaled as Lua strings and
> booleans, respectively. Affected values:
>
>  - `MetaBlocks` values are marshaled as a Blocks list;
>  - `MetaInlines` values are marshaled as a Inlines list;
>  - `MetaList` values are marshaled as a generic pandoc Lists.
>  - `MetaMap` values are marshaled as plain tables and no longer given
>    any metatable.

You can use the new function `pandoc.utils.type` to inspect the value:

``` lua
function Pandoc(doc)
  for k, v in pairs(doc.meta) do
    if pandoc.utils.type(v) == 'Blocks' then
      doc.meta[k] = v:map(function(x)
        return x.t == 'Para' and pandoc.Plain(x.content) or x
      end)
    end
  end
  return doc
end
```

-- 
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Reading a multi line yaml key - results in text surrounded in html <p> tags?

[Albert Krewinkel]

There's also the option to use `>-` instead of `|`. It works almost the
same as `|`, but removes all line breaks (including the final one).

BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

>    Or if needed use single quotes around the value to protect punctuation
>    marks, noting that to include a single quote in a YAML single-quoted
>    string you use two single quotes, as in `'don''t'`.
>
>    Den tors 20 jan. 2022 22:42John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> skrev:
>
>      One workaround is to use a Lua filter that
>      applies pandoc.utils.stringify to the contents of the
>      'volume' field in metadata.
>      Russ <info-ckBjHRpK5r9MPTj2movljOTW4wlIGRCZ@public.gmane.org> writes:
>      > pandoc 2.17.0.1
>      > Reading a multi line YAML front matter and outputing the text now
>      gets a
>      > text string with html <p> tags.
>      >
>      > For example
>      > ```yaml
>      > volume: | Volume 16: A Subtitle ```
>      > Should produce:
>      > "Volume 16: A Subtitle"
>      >
>      > But now produces:
>      > "<p>Volume 16: A Subtitle</p>"
>      >
>      > Is there a way to avoid these html tags or is it a bug? I cannot
>      say for
>      > sure that this has changed in this specific version, but it is
>      certainly
>      > quite a recent change.
>      >
>      > --
>      > You received this message because you are subscribed to the Google
>      Groups "pandoc-discuss" group.
>      > To unsubscribe from this group and stop receiving emails from it,
>      send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>      > To view this discussion on the web visit
>      https://groups.google.com/d/msgid/pandoc-discuss/e17cd72d-2900-476c-
>      a945-77d9e2e722bcn%40googlegroups.com.
>      --
>      You received this message because you are subscribed to the Google
>      Groups "pandoc-discuss" group.
>      To unsubscribe from this group and stop receiving emails from it,
>      send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>      To view this discussion on the web visit
>      https://groups.google.com/d/msgid/pandoc-discuss/yh480ksfti2igy.fsf%
>      40johnmacfarlane.net.
>
>    --
>    You received this message because you are subscribed to the Google
>    Groups "pandoc-discuss" group.
>    To unsubscribe from this group and stop receiving emails from it, send
>    an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>    To view this discussion on the web visit
>    https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhBEViWZsLRRtAr78
>    yn3Q3fSZswg85vut21vkrpfsvszXQ%40mail.gmail.com.


-- 
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Reading a multi line yaml key - results in text surrounded in html <p> tags?

[Russ]

[-- Attachment #1.1: Type: text/plain, Size: 3854 bytes --]

Thanks everyone for your help, the post by Albert about the changes to Lua 
and the splendid code of course solved the problem, well nearly...

In the YAML Frontmatter we have a description:

description: | Some long, perhaps multi-line description.

Sadly this *does not* appear in the epub (2) metadata when we build it, it 
used to work. This does seem like some sort of bug? Because if we use 
commas to surround the description text or use  '>-' as suggested by 
Albert, the description *does* appear in the epub metadata. However, we 
have a lot of markdown which use the '|' formatting for multi-line so I'd 
rather not have to change this is I don't have to.

Does anybody have any ideas?


On Friday, 21 January 2022 at 10:28:45 UTC Albert Krewinkel wrote:

> There's also the option to use `>-` instead of `|`. It works almost the
> same as `|`, but removes all line breaks (including the final one).
>
> BPJ <mel...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
> > Or if needed use single quotes around the value to protect punctuation
> > marks, noting that to include a single quote in a YAML single-quoted
> > string you use two single quotes, as in `'don''t'`.
> >
> > Den tors 20 jan. 2022 22:42John MacFarlane <j...-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> skrev:
> >
> > One workaround is to use a Lua filter that
> > applies pandoc.utils.stringify to the contents of the
> > 'volume' field in metadata.
> > Russ <in...-ckBjHRpK5r9MPTj2movljOTW4wlIGRCZ@public.gmane.org> writes:
> > > pandoc 2.17.0.1
> > > Reading a multi line YAML front matter and outputing the text now
> > gets a
> > > text string with html <p> tags.
> > >
> > > For example
> > > ```yaml
> > > volume: | Volume 16: A Subtitle ```
> > > Should produce:
> > > "Volume 16: A Subtitle"
> > >
> > > But now produces:
> > > "<p>Volume 16: A Subtitle</p>"
> > >
> > > Is there a way to avoid these html tags or is it a bug? I cannot
> > say for
> > > sure that this has changed in this specific version, but it is
> > certainly
> > > quite a recent change.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > Groups "pandoc-discuss" group.
> > > To unsubscribe from this group and stop receiving emails from it,
> > send an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> > > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/pandoc-discuss/e17cd72d-2900-476c-
> > a945-77d9e2e722bcn%40googlegroups.com.
> > --
> > You received this message because you are subscribed to the Google
> > Groups "pandoc-discuss" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/pandoc-discuss/yh480ksfti2igy.fsf%
> > 40johnmacfarlane.net.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "pandoc-discuss" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhBEViWZsLRRtAr78
> > yn3Q3fSZswg85vut21vkrpfsvszXQ%40mail.gmail.com.
>
>
> -- 
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe e836 388d c0b2 1f63 1124
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/899d9fd4-dfae-48d7-ab33-3d55b81f540cn%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 7354 bytes --]

Re: Reading a multi line yaml key - results in text surrounded in html <p> tags?

[John MacFarlane]

> description: |
>   Some long, perhaps multi-line description.
>
> Sadly this *does not* appear in the epub (2) metadata when we build it, it 
> used to work. This does seem like some sort of bug? Because if we use 
> commas to surround the description text or use  '>-' as suggested by 
> Albert, the description *does* appear in the epub metadata. However, we 
> have a lot of markdown which use the '|' formatting for multi-line so I'd 
> rather not have to change this is I don't have to.

It works fine when I try it...

If you want us to look into it, upload a minimal markdown
source and exact command (and pandoc version) you used,
together with the resulting epub.

Re: Reading a multi line yaml key - results in text surrounded in html <p> tags?

[Russ]

[-- Attachment #1.1: Type: text/plain, Size: 1401 bytes --]

Thanks for your suggestion John, I did try this and it did work, so 
apologies. I realised it had to be something to do with our build script. 
We have it working now so many thanks for everyone with their help.

On Friday, 21 January 2022 at 19:37:53 UTC John MacFarlane wrote:

>
> > description: |
> > Some long, perhaps multi-line description.
> >
> > Sadly this *does not* appear in the epub (2) metadata when we build it, 
> it 
> > used to work. This does seem like some sort of bug? Because if we use 
> > commas to surround the description text or use '>-' as suggested by 
> > Albert, the description *does* appear in the epub metadata. However, we 
> > have a lot of markdown which use the '|' formatting for multi-line so 
> I'd 
> > rather not have to change this is I don't have to.
>
> It works fine when I try it...
>
> If you want us to look into it, upload a minimal markdown
> source and exact command (and pandoc version) you used,
> together with the resulting epub.
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/761a5c9b-aa9e-4164-90b2-082085dbc85an%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2002 bytes --]