proposal for new metadata

proposal for new metadata

[John MacFarlane]

See https://github.com/jgm/pandoc/issues/851
Comments welcome, there or here.

Re: proposal for new metadata

[Tillmann Rendel]

Hi,

John MacFarlane wrote:
> See https://github.com/jgm/pandoc/issues/851
> Comments welcome, there or here.

Looks very good. Couple of questions/comments/ideas:

(1) Can we use metadata to set arbitrary pandoc settings that are 
currently set as command-line parameters. For example, can we activate 
"--smart" with metadata?

(2) Will there be support for global metadata defaults, maybe in a 
configuration file? Then we could have the following overriding chain: 
command-line data overrides in-file data, and in-file data overrides 
global data.

(3) How are fields with string-content parsed, as Block or [Block] or 
[Inline]? What happens if meta-data contains references to footnotes in 
the main document? (That actually makes sense for author lists of 
academic papers). What happens if meta-data contains bibliographic 
references, numbered examples etc?

(4) How will the need for Yaml escaping interact with parsing string 
fields as markdown? For example, will it be convenient to put the whole 
abstract of a paper in the metadata block? That could be a couple of 
paragraphs, maybe with a bulleted list and some references.

(5) Can metadata also be stored in a file different from the content, 
and then referenced by filename at the command line?

(6) Can we use metadata to pull in more metadata from an external file, 
in order to share settings between similar documents.

(7) Did you consider using the existing definition list syntax for the 
key-value-lists you need for metadata, instead of YAML?

(8) Will the pandoc executable offer some way to extract the meta-data 
block from a file, so that third-party tools don't have to look for it 
themselves? For shell scripts, it would even be nice to extract the 
value of single meta-data field, something like `pandoc --get author 
file.md`.

   Tillmann

Re: proposal for new metadata

[Tillmann Rendel]

Hi,

Tillmann Rendel wrote:
> Looks very good. Couple of questions/comments/ideas:

One more:

(9) Could structured metadata be made available in a structure way, for 
example in a template like the following?

     $for(author)$
     \author{$author.name$}{$author.affiliation$}
     $endfor$

The value of author would be a list of structured items.

   Tillmann

Re: proposal for new metadata

[BP Jonsson]

2013-05-10 07:51, John MacFarlane skrev:
> See https://github.com/jgm/pandoc/issues/851
> Comments welcome, there or here.
>
 > Why an association list instead of a map? We want to allow
 > several entries with the same key (e.g. multiple "author"
 > entries). The lists typically won't be long enough to affect
 > performance.

Is the equivalent of the following an option?

````json
{"author":["Alice","Bob","Charles"]}
````
It would be especially nice to be able to get/use such a
representation in JSON in/output. (Not that the conversion would
be terribly hard to do as a part of processing.)

 > ### Retrofit MMD title blocks to use the new Meta type

 >
 > This should be straightforward. (Do we want to impose some
 > discipline on the key names, to ensure that they can all be
 > used in templates? If so, we might need some conversion
 > scheme. Note also that MMD metadata is unstructured
 > strings.)

Wouldn't something akin to the way headings are turned
into id attributes suffice?: turn all illegal substrings
into something suitable which is legal, preferably hyphens
or underscores.
 >
 > ### Change writers to set template variables for all metadata 
fields
 >
 > Currently only predefined variables are set.
 >
 > Note: variables specified on the command line should
 > override those set in the document. (Or should they?)

Yes, I think they should.

What about allowing to load metadata from YAML files
through an option (preferably with `-y` as short
form! :-), and perhaps also from a file at
`~/.pandoc/metadata.yml`? Then one doesn't need to
repeatedly type information like ones own name and
contact info which never/seldom changes, or such data
which usually is the same but sometimes needs to be
overridden.

Metadata given later on the commandline either through
`--variable` or through loading a file should override
metadata given earlier on the commandline, the commandline
should override metadata given in the document file and
the document file should override `~/.pandoc/metadata.yml`.
I use a similar system in a perl program which takes
metadata from YAML files, although there the override
order is option -> loaded files -> default file due
to the fact that Getopt::Long preserves the relative
order of arguments to the same option but not of
arguments to different options.

It would also be nice if one could include a metadata
variable not only in a template but also in the markdown
text, e.g. (with a syntax combining current link syntax
and template variable inclusion syntax):

````markdown
Please respond within a forthnight from [$date$]!
````
In the unlikely event the text of a reference link
consists of a math string only (is that even possible?)
one would be required to use a second empty pair of brackets.

 > Metadata sections may occur anywhere in the document.

Since `---` and `...` are legal text elements (emdash
and ellipsis) I think metadata should come only at the
beginning or end of the document, or should use some
other delimiter. I propose to use `%---` and `%...` as
delimiters, following the current use of `%` as title
block marker.

 > A document may contain multiple metadata sections.

I'm divided on this: it might be good to be able to place
metadata near the part of the text it pertains to, but it
would also tear up the text-as-text.  Part of me wants to
place all metadata after the text, and of course anyone
would be free to do so -- and I guess the markdown writer
would do so too!

 > Which fields create metadata records?
 >
 > -   If a field ends with `_`, it does not create a metadata
 >     record. (This is primarily intended for content that
 >     should not be parsed as markdown, but might be used by
 >     an external program.)

I think this should be "beginning with `_`" -- that's far
easier to spot when scanning the text!

 > -   If a field has a YAML object as its content, it will not
 >     create a metadata record at all (e.g. `structured` in
 >     the example above).

Other than list content, I take it.

 >
 > This system allows arbitrary structured metadata to be put
 > into the document. (One could, for example, include a whole
 > bibtex bibliography.) However, only fields that might be
 > used in the rendition of the document in an output format
 > are actually stored in the pandoc document. The rest are
 > ignored by pandoc but may be used by other programs (gitit,
 > hakyll, etc.).

Couldn't pandoc just store it so that it can spit it out in
JSON output?

/bpj

Re: proposal for new metadata

[John MacFarlane]

+++ Tillmann Rendel [May 11 13 13:46 ]:
> Hi,
> 
> John MacFarlane wrote:
> >See https://github.com/jgm/pandoc/issues/851
> >Comments welcome, there or here.
> 
> Looks very good. Couple of questions/comments/ideas:
> 
> (1) Can we use metadata to set arbitrary pandoc settings that are
> currently set as command-line parameters. For example, can we
> activate "--smart" with metadata?

In general, no, because the command-line parameters are processed
in pandoc.hs, before the markdown reader gets called.  (Since
the metadata is markdown specific, it will only get parsed in the
markdown reader.)

However, we might be able to special-case a few options that
affect the readers, like --smart.

> (2) Will there be support for global metadata defaults, maybe in a
> configuration file? Then we could have the following overriding
> chain: command-line data overrides in-file data, and in-file data
> overrides global data.

I'm not inclined to do this, at least at first.

> (3) How are fields with string-content parsed, as Block or [Block]
> or [Inline]? What happens if meta-data contains references to
> footnotes in the main document? (That actually makes sense for
> author lists of academic papers). What happens if meta-data contains
> bibliographic references, numbered examples etc?

As [Block].  Footnotes and citations should work.  There might be
an issue with footnotes in some formats, concerning the order of processing.
For example, in HTML, the writer needs to put the footnotes in order.
So it needs to know in what order to process the metadata bits.
I suppose it could learn this in principle by scanning the template
to see in what order the variables occur, but this seems a bit
clumsy.

> (4) How will the need for Yaml escaping interact with parsing string
> fields as markdown? For example, will it be convenient to put the
> whole abstract of a paper in the metadata block? That could be a
> couple of paragraphs, maybe with a bulleted list and some
> references.

Yes, it will be convenient, because you can use this YAML syntax
to avoid the need to escape anything:

abstract: |
  An indented block, which will be parsed
  literally.

  It can contain whatever.

  1. one
  2. two

> (5) Can metadata also be stored in a file different from the
> content, and then referenced by filename at the command line?

Sure.  Pandoc concatenates files on the command line, so you can
just do:

pandoc metadata.txt content.txt

> (6) Can we use metadata to pull in more metadata from an external
> file, in order to share settings between similar documents.

I think that's getting a bit fancier than I'd like to (partly
because it requires the reader to do IO, and currently the
reader is pure).  Note, however, that you can have several
metadata sections in the same document.  So you could do:

pandoc shared_metadata.txt content_with_additional_metadata.txt

> (7) Did you consider using the existing definition list syntax for
> the key-value-lists you need for metadata, instead of YAML?

One drawback is that this syntax is cumbersome for the most usual
use-cases -- one line titles, authors, and the like -- since you
need multiple lines for each entry.

Another drawback is that it doesn't allow the metadata to be
structured.

I also wanted to allow metadata fields containing arbitrary data,
that would not be parsed as markdown.  (These can be marked with
a trailing _ in the name.)  For example, you could put your
entire bibtex bibliography in a 'bibliography_' field.  This
would essentially just be ignored by pandoc.  (Though, in this
case, we might eventually want pandoc to look for a bibliography_
field and use it if --biblio is not specified.)

> (8) Will the pandoc executable offer some way to extract the
> meta-data block from a file, so that third-party tools don't have to
> look for it themselves? For shell scripts, it would even be nice to
> extract the value of single meta-data field, something like `pandoc
> --get author file.md`.

That's a nice idea.  My current implementation idea doesn't give a
good way to do this, but I'll think about how I might.

John

Re: proposal for new metadata

[John MacFarlane]

+++ Tillmann Rendel [May 11 13 13:59 ]:
> (9) Could structured metadata be made available in a structure way,
> for example in a template like the following?
> 
>     $for(author)$
>     \author{$author.name$}{$author.affiliation$}
>     $endfor$
> 
> The value of author would be a list of structured items.

I see how this would be useful.  My original idea was to use
a simple association list, which wouldn't allow this kind of
structuring, because it would fit well with the current templating
system.  But the templating system could be changed.

This would require a fancier Meta type.  I can't just use
the type used by the YAML parser (which is actually a Json object),
because the contents need to contain [Block]s.  Maybe something
like this:

data MMap      = M.Map String MetaValue

data MetaValue = MString String
               | MBlocks [Block]
               | MList [MetaValue]
               | MMap

type Meta      = MMap

We'd then just walk the tree produced by the YAML parser and
convert it to this type.  List nodes would go to MList,
object nodes would go to MMap, and leaf nodes would either be
MString (if they end in _) or MBlocks (otherwise).

The template engine would then have to be changed to take
an MMap as context rather than an association list.  And
the syntax 'a.b' could be used, as you suggest, to do map
lookups.  I think I like the idea.

John

Re: proposal for new metadata

[David Sanson]

[-- Attachment #1: Type: text/plain, Size: 3242 bytes --]

I like the proposal. I like the use of a standard format. I like the power 
and flexibility. I think the tradeoffs---avoiding lines that accidentally 
start with unescaped `---` and `...`, for example, are worth it. Some 
comments and comments on comments below.

On Saturday, May 11, 2013 9:59:34 AM UTC-5, BP Jonsson wrote:
>
> 2013-05-10 07:51, John MacFarlane skrev: 
>
> What about allowing to load metadata from YAML files 
> through an option (preferably with `-y` as short 
> form! :-)
>
 
I'm not sure I see the need for a `-y` option: assuming the YAML file 
begins and ends with the proper delimiters, it is just another input file: 
`pandoc example.md metadata.yml`. 
 

> Metadata given later on the commandline either through 
> `--variable` or through loading a file should override 
> metadata given earlier on the commandline, the commandline 
> should override metadata given in the document file and 
> the document file should override `~/.pandoc/metadata.yml`. 
>

I'm not sure how to think about overrides here, and maybe this is where a 
`-y` option starts to make more sense to me. 

Start with the simplest question: what happens with metadata within a 
document? Does later metadata override earlier metadata? The proposal 
suggests not, and that instead the later metadata gets added to the earlier 
metadata, so that something like

~~~
---
title: Our Book
author: Laura Smith
author: James Munoz
title: Still Our Book
...
~~~

has two authors and two titles. Is it up to the template to figure out what 
to do with that? Or does pandoc keep track of some of the known metadata 
keys and treat them special?

Suppose I co-author a paper with Bob Dobbs. Should

    pandoc -V="author:Bob Dobbs"

add Bob as an additional author, or should it replace the authors specified 
in the document with Bob? I can see an argument for both use cases.

 > Metadata sections may occur anywhere in the document. 
>

I'm a big fan of this.
 

> Since `---` and `...` are legal text elements (emdash 
> and ellipsis) I think metadata should come only at the 
> beginning or end of the document, or should use some 
> other delimiter. I propose to use `%---` and `%...` as 
> delimiters, following the current use of `%` as title 
> block marker. 
>

I see the cost, but I like the proposal, because `---` and `...` are the 
standard YAML delimiters. Using them means that you can use any existing 
tools that process YAML to process the metadata in the document.

 > A document may contain multiple metadata sections. 
>

Yes! For one thing, this allows for concatenation of multiple files at 
once. Though, once again, that raises questions about overrides within a 
single document.

-David

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msg/pandoc-discuss/-/Es2gxEGuNccJ.
For more options, visit https://groups.google.com/groups/opt_out.



[-- Attachment #2: Type: text/html, Size: 4349 bytes --]

Re: proposal for new metadata

[Joseph Reagle]

On 05/11/2013 11:16 AM, John MacFarlane wrote:
>> (1) Can we use metadata to set arbitrary pandoc settings that are
>> currently set as command-line parameters. For example, can we
>> activate "--smart" with metadata?
>
> In general, no, because the command-line parameters are processed
> in pandoc.hs, before the markdown reader gets called.  (Since
> the metadata is markdown specific, it will only get parsed in the
> markdown reader.)

I've often thought of requesting this but imagined this would be tricky. 
Since you implemented multimarkdown metadata I *do* do something like 
this in my wrappers though: my static html generator `wiki-update` reads 
markdown files for `md` options (which is the pandoc wrapper.) I also 
use custom options for special directories (like courses, talks, etc.)

~~~~

         match_md_opts = re.search('^md_opts: (.*)', content, re.MULTILINE)
         if match_md_opts:
             md_opts = match_md_opts.group(1).split(' ')
             info("md_opts = %s" % md_opts)
             md_args.extend(md_opts)

~~~~

https://github.com/reagle/pandoc-wrappers/blob/master/wiki-update.py#L158

Re: proposal for new metadata

[BP Jonsson]

2013-05-11 17:35, David Sanson skrev:
> I see the cost, but I like the proposal, because `---` and `...` are the
> standard YAML delimiters. Using them means that you can use any existing
> tools that process YAML to process the metadata in the document.

You must still extract them somehow *before* presenting
them to the YAML processor, or it will complain that
the rest of the file isn't valid YAML, and in case your
YAML tool can scan for the delimiters itself you'd
still need to pre-scan for codeblocks so that you don't
include some YAML from a codeblock, so probably you are
using regular expressions at some point, and there
isn't really much difference between
`/^(---\s*$.*?^\.\.\.\s*\n)/gms` and
`/^%(---\s*$.*?)^%\.\.\.\s*\n/gms`
the only difference being that in the second case
you'll need to add two characters and move the closing
paren leftwards in the regex and smack on a "..." at
the end of the captured text if you feel you really
need it. The hardest bit is to find and pass over
codeblocks if necessary, and yes, it happens that
people put YAML in their codeblocks; at least it's not
safe to assume they never would!

It matters as little if you go through the file line by
line and collect lines between a line matching the
opening delimiter and the next line matching the
closing delimiter: a check that you are on one of the
delimiter lines and removing the first character in
that case. And you would anyway have to check for code
blocks. (That's why fenced codeblocks are so great! :-)

It would be much better if *pandoc* could spit out the
metadata it has extracted from a document as YAML or
JSON, since pandoc would anyway need to know how to
tell the difference between YAML in a codeblock and a
real metadata block!

/bpj

Re: proposal for new metadata

[John MacFarlane]

+++ BP Jonsson [May 11 13 22:21 ]:
> 2013-05-11 17:35, David Sanson skrev:
> >I see the cost, but I like the proposal, because `---` and `...` are the
> >standard YAML delimiters. Using them means that you can use any existing
> >tools that process YAML to process the metadata in the document.
> 
> You must still extract them somehow *before* presenting
> them to the YAML processor, or it will complain that
> the rest of the file isn't valid YAML, and in case your
> YAML tool can scan for the delimiters itself you'd
> still need to pre-scan for codeblocks so that you don't
> include some YAML from a codeblock, so probably you are
> using regular expressions at some point, and there
> isn't really much difference between
> `/^(---\s*$.*?^\.\.\.\s*\n)/gms` and
> `/^%(---\s*$.*?)^%\.\.\.\s*\n/gms`
> the only difference being that in the second case
> you'll need to add two characters and move the closing
> paren leftwards in the regex and smack on a "..." at
> the end of the captured text if you feel you really
> need it. The hardest bit is to find and pass over
> codeblocks if necessary, and yes, it happens that
> people put YAML in their codeblocks; at least it's not
> safe to assume they never would!
> 
> It matters as little if you go through the file line by
> line and collect lines between a line matching the
> opening delimiter and the next line matching the
> closing delimiter: a check that you are on one of the
> delimiter lines and removing the first character in
> that case. And you would anyway have to check for code
> blocks. (That's why fenced codeblocks are so great! :-)

I think that, if the YAML sections are delimited in the way
I suggested, standard tools should be able to extract all
the YAML.  See http://yaml.org/spec/1.0/#id2561559 sec. 4.3,
YAML Stream.

However, as you point out, there is a gotcha here if you
have a YAML section inside a code block.

> It would be much better if *pandoc* could spit out the
> metadata it has extracted from a document as YAML or
> JSON, since pandoc would anyway need to know how to
> tell the difference between YAML in a codeblock and a
> real metadata block!

Right.  This would be a nice feature.

Re: proposal for new metadata

[John MacFarlane]

+++ David Sanson [May 11 13 08:35 ]:
>    I'm not sure how to think about overrides here, and maybe this is where
>    a `-y` option starts to make more sense to me.
>    Start with the simplest question: what happens with metadata within a
>    document? Does later metadata override earlier metadata? The proposal
>    suggests not, and that instead the later metadata gets added to the
>    earlier metadata, so that something like
>    ~~~
>    ---
>    title: Our Book
>    author: Laura Smith
>    author: James Munoz
>    title: Still Our Book
>    ...
>    ~~~
>    has two authors and two titles. Is it up to the template to figure out
>    what to do with that? Or does pandoc keep track of some of the known
>    metadata keys and treat them special?

Well, here's how it would work with the original proposal, with
association lists.  You'd get  an association list with two
entries for "title".  If your template just has $title$, then
it will take the first value only.  If, on the other hand, it
has $for(title)$...$title$...$endfor$, it will iterate through
all the values.  It's up to the template which you do, and hence
up to the template whether values other than the first will be
significant.

It still needs to be defined more precisely what counts as "first".
Order within a document should be preserved.  Probably variables
specified on the command line should come before those in a document.

Re: proposal for new metadata

[John MacFarlane]

+++ John MacFarlane [May 11 13 15:42 ]:
> +++ David Sanson [May 11 13 08:35 ]:
> >    I'm not sure how to think about overrides here, and maybe this is where
> >    a `-y` option starts to make more sense to me.
> >    Start with the simplest question: what happens with metadata within a
> >    document? Does later metadata override earlier metadata? The proposal
> >    suggests not, and that instead the later metadata gets added to the
> >    earlier metadata, so that something like
> >    ~~~
> >    ---
> >    title: Our Book
> >    author: Laura Smith
> >    author: James Munoz
> >    title: Still Our Book
> >    ...
> >    ~~~
> >    has two authors and two titles. Is it up to the template to figure out
> >    what to do with that? Or does pandoc keep track of some of the known
> >    metadata keys and treat them special?
> 
> Well, here's how it would work with the original proposal, with
> association lists.  You'd get  an association list with two
> entries for "title".

Sorry, I was wrong.  You'd simply get one entry (the last I believe) for
each duplicate.  To get multiples, you'd need to use an explicit list.

author:
  - Author One
  - Author Two

Re: proposal for new metadata

[John MacFarlane]

I'm still back-and-forth about whether JSON metadata blocks should be
allowed anywhere in the document, or just at the beginning.

Arguments for anywhere
----------------------

1.  It would often be nice to have a regular pandoc title block at
the top, with more extensive metadata given at the end of the document.
This makes documents more readable.

2.  It would be nice, eventually, to be able to include bibliographies
(in, say, bibtex format) in the document itself, rather than relying
on an external file.  Putting them in metadata at the end of the
document would be a good way to do this.

Argument for just at the beginning
----------------------------------

1.  If metadata is allowed anywhere, it makes it difficult for
external tools to extract the metadata reliably.  One can't just scan
for a line beginning '---', since such a line might be a setext header
underline, or part of a code block.  One would really have to parse
the whole document as pandoc markdown, which is difficult and
takes time.  If metadata is only allowed at the beginning, then tools
can just check there and move on.  (One could add a pandoc option to
dump metadata as JSON.  This would allow external tools to get the
metadata easily, but it might take a long time to extract metadata
from a long document, since the whole document would be parsed.)


A possible compromise would be to allow metadata blocks just at
the very beginning and the very end of the document.

John

Re: proposal for new metadata

[Fedor Sheremetyev]

On 20 May 2013, at 17:32, John MacFarlane <fiddlosopher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> I'm still back-and-forth about whether JSON metadata blocks should be
> allowed anywhere in the document, or just at the beginning.
> 
> Arguments for anywhere
> ----------------------
> 
> 1.  It would often be nice to have a regular pandoc title block at
> the top, with more extensive metadata given at the end of the document.
> This makes documents more readable.
> 
> 2.  It would be nice, eventually, to be able to include bibliographies
> (in, say, bibtex format) in the document itself, rather than relying
> on an external file.  Putting them in metadata at the end of the
> document would be a good way to do this.

Are there other real use cases for additional metadata at the end of the document besides bibliography?

I think bibliography is use case on it own and has nothing to do with title, authors etc. It would probably make sense to have Markdown-like markup for bibliography (not BibTeX inside YAML) — so that document is readable “as is”. Also one might want to have bibliography items embedded in text near the place they are used (like footnotes).

> Argument for just at the beginning
> ----------------------------------
> 
> 1.  If metadata is allowed anywhere, it makes it difficult for
> external tools to extract the metadata reliably.  One can't just scan
> for a line beginning '---', since such a line might be a setext header
> underline, or part of a code block.  One would really have to parse
> the whole document as pandoc markdown, which is difficult and
> takes time.  If metadata is only allowed at the beginning, then tools
> can just check there and move on.  (One could add a pandoc option to
> dump metadata as JSON.  This would allow external tools to get the
> metadata easily, but it might take a long time to extract metadata
> from a long document, since the whole document would be parsed.)

Relying on Pandoc to extract metadata might be too restrictive. One might want to parse document with metadata via JavaScript code in web browser, for example.

> A possible compromise would be to allow metadata blocks just at
> the very beginning and the very end of the document.


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit https://groups.google.com/groups/opt_out.

Re: proposal for new metadata

[Eric Watson]

On May 20, 2013, at 11:32 AM, John MacFarlane <fiddlosopher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> I'm still back-and-forth about whether JSON metadata blocks should be
> allowed anywhere in the document, or just at the beginning.

I secretly want pandoc to be able to produce HTML slides using [impress.js][1], which requires setting `data-*` attributes on each slide, like this: 

    <div id="my-impressive-slide" class="step" data-x="850" data-y="3000" data-rotate="90" data-scale="5">
	<p>See it rotate! See it scale! See it <em>move!</em></p>
    </div>

    <div id="my-next-big-slide" class="step" data-x="3500" data-y="2100" data-rotate="180" data-scale="6">
        <p>Look, new metadata values!</p>
    </div>


If you could sprinkle metadata throughout a pandoc document, this might work nicely :)

    ---
    x: 850
    y: 3000
    rotate: 90
    scale: 5
    ...


    ## My Impressive Slide

    See it rotate! See it scale! See it *move!*


    ---
    x: 3500
    y: 2100
    rotate: 180
    scale: 6
    ...

    ## My Next Big Slide

    Look, new metadata values!



This is a pretty specialized use case, but the general idea of being able to provide (and change?) metadata at any point in the document seems like it could enable lots of cool new use cases.

Thanks for the great thinking on this tricky problem!

Eric


[1]https://github.com/bartaz/impress.js

Re: proposal for new metadata

[John Gabriele]

[-- Attachment #1: Type: text/plain, Size: 1774 bytes --]

On Mon, May 20, 2013 at 12:32 PM, John MacFarlane <fiddlosopher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>wrote:

> I'm still back-and-forth about whether JSON metadata blocks should be
> allowed anywhere in the document, or just at the beginning.
>
> > A possible compromise would be to allow metadata blocks just at

> the very beginning and the very end of the document.
>

I think "metadata anywhere" would lead to docs with metadata strewn about
the place. Also, I'd rather not need to use a special tool to show me all
the metadata in a given doc.

I like the idea of metadata only at the beginning of a doc because it seems
the neatest and cleanest way to go.

Sidenote: I don't have any experience using BibTeX. Could you please give
an example of what it would look like to include a bibliography (in, bibtex
format) in the (proposed) document metadata?

BTW: Instead of using metadata for bibliographic info, might it be better
to instead extend Pandoc to specifically have support for raw BibTeX (or
maybe create a special block syntax for a bibliography)? I mean, it seems
to be a pretty common need, and Pandoc already supports LaTeX math...

Thanks,
-- John

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAO9PwMXKSM9_XJFTmUnUGC07V2hVx8hjgG_Un322aYs%2BLLYsvA%40mail.gmail.com?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.



[-- Attachment #2: Type: text/html, Size: 2548 bytes --]

Re: proposal for new metadata

[John MacFarlane]

+++ Eric Watson [May 20 13 13:17 ]:
> 
> On May 20, 2013, at 11:32 AM, John MacFarlane <fiddlosopher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> > I'm still back-and-forth about whether JSON metadata blocks should be
> > allowed anywhere in the document, or just at the beginning.
> 
> I secretly want pandoc to be able to produce HTML slides using [impress.js][1], which requires setting `data-*` attributes on each slide, like this: 
> 
>     <div id="my-impressive-slide" class="step" data-x="850" data-y="3000" data-rotate="90" data-scale="5">
> 	<p>See it rotate! See it scale! See it <em>move!</em></p>
>     </div>
> 
>     <div id="my-next-big-slide" class="step" data-x="3500" data-y="2100" data-rotate="180" data-scale="6">
>         <p>Look, new metadata values!</p>
>     </div>
> 
> 
> If you could sprinkle metadata throughout a pandoc document, this might work nicely :)

Well, you don't need metadata for this, since pandoc already allows you
to put attributes on headers, and headers can turn into divs if you use
--section-divs.

# See it rotate!  {#my-impressive-slide .step data-x="850" data-y="3000" data-rotate="90"}

Re: proposal for new metadata

[Chris Lott]

On Mon, May 20, 2013 at 10:31 AM, John Gabriele <jmg3000-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> I think "metadata anywhere" would lead to docs with metadata strewn about
> the place. Also, I'd rather not need to use a special tool to show me all
> the metadata in a given doc.
>
> I like the idea of metadata only at the beginning of a doc because it seems
> the neatest and cleanest way to go.

I hear you, but I don't want a tool enforcing standards I (or others)
don't necessarily share. Which isn't to say there aren't good
arguments  for only allowing metadata in a few specific places, just
that I don't think disallowing me from putting metadata blocks at the
beginning of, say, each major section in a doc, should be Pandoc's
privilege...

c
--
Chris Lott <chris-qMDBG1KJBpwBXFe83j6qeQ@public.gmane.org>

Re: proposal for new metadata

[Eric Watson]

On May 20, 2013, at 1:32 PM, John MacFarlane <fiddlosopher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Well, you don't need metadata for this, since pandoc already allows you
> to put attributes on headers, and headers can turn into divs if you use
> --section-divs.

Cool! I'll give it a whirl!

Re: proposal for new metadata

[John MacFarlane]

+++ Eric Watson [May 20 13 15:34 ]:
> 
> 
> On May 20, 2013, at 1:32 PM, John MacFarlane <fiddlosopher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> > Well, you don't need metadata for this, since pandoc already allows you
> > to put attributes on headers, and headers can turn into divs if you use
> > --section-divs.
> 
> Cool! I'll give it a whirl!

If you come up with a nice way of using pandoc to produce impress.js
slide shows, please let us know about it here!

Re: proposal for new metadata

[Eric Watson]

On May 20, 2013, at 4:25 PM, John MacFarlane <fiddlosopher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> If you come up with a nice way of using pandoc to produce impress.js
> slide shows, please let us know about it here!

Absolutely!

Re: proposal for new metadata

[David Sanson]

The point of embedding bibliography data in the markdown document is to allows for a self-contained markdown source: 

    pandoc example.md --bibliography example.md

Actual presentation of a bibliography would be done by processing the file.

Are there other practical use cases for embedded structured data? I don't know. Question pools for embedding into SCORM compliant HTML and LaTeX quizzes and exams? I can dream!

How do extant tools handle YAML blocks in files? My understanding was that the YAML standard allows data blocks anywhere, but I don't know if existing tools respect that. If the data blocks anywhere option is too expensive, processing-wise, and doesn't work with existing tools, then I don't see a compelling reason to allow it at this point.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/f9818c1f-db3e-4a28-9ebd-3e5d9281e065%40googlegroups.com?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.

Re: proposal for new metadata

[Tillmann Rendel]

Hi,

John MacFarlane wrote:
> I'm still back-and-forth about whether JSON metadata blocks should be
> allowed anywhere in the document, or just at the beginning.
>
> Arguments for anywhere
> ----------------------

Another argument: Merging files would automatically merge the metadata 
from each of the files.

Also, wasn't it YAML metadata blocks, or is this about something else?

   Tillmann

Re: proposal for new metadata

[John MacFarlane]

+++ Tillmann Rendel [May 21 13 21:43 ]:
> Hi,
> 
> John MacFarlane wrote:
> >I'm still back-and-forth about whether JSON metadata blocks should be
> >allowed anywhere in the document, or just at the beginning.
> >
> >Arguments for anywhere
> >----------------------
> 
> Another argument: Merging files would automatically merge the
> metadata from each of the files.

Yes, though it needs to be defined what counts as merging.
(Does a subsequent field replace the earlier one's value,
or somehow get merged with it?)

> Also, wasn't it YAML metadata blocks, or is this about something else?

Sorry, yes, I meant YAML.  (The Haskell library for YAML uses the
JSON types, which explains my slip.)

Re: proposal for new metadata

[John Gabriele]

[-- Attachment #1: Type: text/plain, Size: 1400 bytes --]

On Tue, May 21, 2013 at 3:57 PM, John MacFarlane <fiddlosopher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>wrote:

> +++ Tillmann Rendel [May 21 13 21:43 ]:
> >
> > John MacFarlane wrote:
> > >I'm still back-and-forth about whether JSON metadata blocks should be
> > >allowed anywhere in the document, or just at the beginning.
> > >
> > >Arguments for anywhere
> > >----------------------
> >
> > Another argument: Merging files would automatically merge the
> > metadata from each of the files.
>
> Yes, though it needs to be defined what counts as merging.
> (Does a subsequent field replace the earlier one's value,
> or somehow get merged with it?)
>
>
Ooof. Right. It would be good if

    pandoc -s -o out.html a.txt b.txt c.txt

worked where a.txt, b.txt, and c.txt all contain metadata.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAO9PwMV3G%2B7zr61XVJ_LAziw%3DFBmDYe2cT6_jWOCq%2Bu88ZkHVw%40mail.gmail.com?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.



[-- Attachment #2: Type: text/html, Size: 2111 bytes --]