Multimarkdown Meta Data

Multimarkdown Meta Data

[Brett Calcott]

Hi all.

As I understand it, pandoc supports a limited bit of metadata at the
top of the file (Title, author, date). How hard would it be to support
multimarkdown style metadata, like this:

Title: A Document
Author: Bob
Language: English
Date: Wednesday
Keywords: stunning, amazing

Brett

Re: Multimarkdown Meta Data

[John MacFarlane]

+++ Brett Calcott [Nov 23 10 11:47 ]:
> Hi all.
> 
> As I understand it, pandoc supports a limited bit of metadata at the
> top of the file (Title, author, date). How hard would it be to support
> multimarkdown style metadata, like this:
> 
> Title: A Document
> Author: Bob
> Language: English
> Date: Wednesday
> Keywords: stunning, amazing
> 
> Brett

I think something like this could be useful, and there have
certainly been requests to this effect before.

One issue is that I've tried, as much as possible, not to make pandoc
English-centric. I wouldn't want authors of texts in Swedish to have to use
English metadata labels. But perhaps this could be made locale-sensitive.

There are many other small syntax decisions to be made.  For example, can
fields be repeated?  If not, how do you handle things like multiple
authors (currently allowed in pandoc)?  Should custom fields be allowed,
or just fields from a standard list?  What if someone wants to start the
document with a regular paragraph beginning "Date: ..."?
(reStructuredText solves this by using ":date: ...." for the metadata.

Etc.

John

Re: Multimarkdown Meta Data

[Paul R]

Hello,

John> One issue is that I've tried, as much as possible, not to make
John> pandoc English-centric. I wouldn't want authors of texts in
John> Swedish to have to use English metadata labels. 

This is very nice from you !

John> But perhaps this could be made locale-sensitive.

Or, simply, left as a configuration to the end user. We will probably
find many people wanting to write keywords in english, whatever the
document language is, and many people wanting consistent language use.
I read in an other thread that you were considering a user configuration
file. This could include a section for keyword aliases.

[keyword-aliases]
author "Auteur :"
date "Date :"
...

That would make documents non portable without appropriate
configuration, but would remove a lot of localisation work on pandoc.

John> There are many other small syntax decisions to be made. For
John> example, can fields be repeated? If not, how do you handle things
John> like multiple authors (currently allowed in pandoc)? 

Using a standard syntax for this type of use case is very convenient.
YAML, like in gitit, probably is a good option. 

John> Should custom fields be allowed, or just fields from a standard
John> list? 

As a first step, maybe you can allow any field but ignore what is not
known. I'm not sure what the use-case would be for those custom fields.

John> What if someone wants to start the document with a regular
John> paragraph beginning "Date: ..."? (reStructuredText solves this by
John> using ":date: ...." for the metadata.

Did you plan to use delimitation markers ? In this case, that may
provide a solution.

-- 
  Paul

Re: Multimarkdown Meta Data

[Nathan Gass]

On 23.11.10 06:45, John MacFarlane wrote:
> +++ Brett Calcott [Nov 23 10 11:47 ]:
>> Hi all.
>>
>> As I understand it, pandoc supports a limited bit of metadata at the
>> top of the file (Title, author, date). How hard would it be to support
>> multimarkdown style metadata, like this:
>>
>> Title: A Document
>> Author: Bob
>> Language: English
>> Date: Wednesday
>> Keywords: stunning, amazing
>>
>> Brett
>
> I think something like this could be useful, and there have
> certainly been requests to this effect before.
>
> One issue is that I've tried, as much as possible, not to make pandoc
> English-centric. I wouldn't want authors of texts in Swedish to have to use
> English metadata labels. But perhaps this could be made locale-sensitive.

I think this is the way to go. Especially as there are some less 
important English-centric features in pandoc already, which could then 
be made locale-sensitive too.

>
> There are many other small syntax decisions to be made.  For example, can
> fields be repeated?  If not, how do you handle things like multiple
> authors (currently allowed in pandoc)?  Should custom fields be allowed,
> or just fields from a standard list?  What if someone wants to start the
> document with a regular paragraph beginning "Date: ..."?
> (reStructuredText solves this by using ":date: ...." for the metadata.

I would not event any new syntax, but use your definition list syntax, 
which solves this problem too. That way, the document looks more 
consistent and users don't have to learn an extra syntax just for 
meta-information.

Nathan

>
> Etc.
>
> John
>

Re: Multimarkdown Meta Data

[Nathan Gass]

On 23.11.10 11:10, Nathan Gass wrote:

> I would not event any new syntax, but use your definition list syntax,

Ups, that should be invent, and not event.

Nathan

Re: Multimarkdown Meta Data

[Brett Calcott]

>
> One issue is that I've tried, as much as possible, not to make pandoc
> English-centric. I wouldn't want authors of texts in Swedish to have to use
> English metadata labels. But perhaps this could be made locale-sensitive.

Locale sensitivity seems like the right way to go.

>
> There are many other small syntax decisions to be made.  For example, can
> fields be repeated?  If not, how do you handle things like multiple
> authors (currently allowed in pandoc)?  Should custom fields be allowed,
> or just fields from a standard list?  What if someone wants to start the
> document with a regular paragraph beginning "Date: ..."?
> (reStructuredText solves this by using ":date: ...." for the metadata.
>

My thought was that pandoc would simply provide a way of providing
metadata (I was suggesting that the multimarkdown format as it fits
with the markdown philosophy and is already in use), and then perhaps
choose to use some fields for itself: Date, Author etc. It would be up
to the pandoc author to define how they would be interpreted. Other
fields would be ignored but could be passed on for use in the
templates. On that note, one nice thing would be able to assign the
template based on meta-data.

Not sure how to handle someone starting a document "Date:". Couldn't
someone start it ":date:" too? Maybe there could be some optional
:EndMeta tag or something for these odd cases.

Hm. I should really own up that I want the multimarkdown format as it
is what Scrivener writes out...

Brett

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to pandoc-discuss+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pandoc-discuss?hl=en.

Re: Multimarkdown Meta Data

[Joost Kremers]

On Tue, Nov 23, 2010 at 10:53:04PM +1100, Brett Calcott wrote:
> My thought was that pandoc would simply provide a way of providing
> metadata (I was suggesting that the multimarkdown format as it fits
> with the markdown philosophy and is already in use), and then perhaps
> choose to use some fields for itself: Date, Author etc. It would be up
> to the pandoc author to define how they would be interpreted. Other
> fields would be ignored but could be passed on for use in the
> templates.

In essence, then, the metadata would actually be a list of variables with
values, wouldn't it? Personally, I'd welcome that, it would void the need for
passing --variable parameters...

> Not sure how to handle someone starting a document "Date:". Couldn't
> someone start it ":date:" too? Maybe there could be some optional
> :EndMeta tag or something for these odd cases.

Perhaps in the off case that someone wants to start a document with "Date:",
they'd simply have to start the document with an empty line? That would tell the
parser to not interpret "Date:" as metadata, but it wouldn't affect the output.


-- 
Joost Kremers
Life has its moments

Re: Multimarkdown Meta Data

[Brett Calcott]

On 24 November 2010 06:08, Joost Kremers <joostkremers-97jfqw80gc6171pxa8y+qA@public.gmane.org> wrote:
> On Tue, Nov 23, 2010 at 10:53:04PM +1100, Brett Calcott wrote:
>> My thought was that pandoc would simply provide a way of providing
>> metadata (I was suggesting that the multimarkdown format as it fits
>> with the markdown philosophy and is already in use), and then perhaps
>> choose to use some fields for itself: Date, Author etc. It would be up
>> to the pandoc author to define how they would be interpreted. Other
>> fields would be ignored but could be passed on for use in the
>> templates.
>
> In essence, then, the metadata would actually be a list of variables with
> values, wouldn't it? Personally, I'd welcome that, it would void the need for
> passing --variable parameters...

That's how I see it.

>
>> Not sure how to handle someone starting a document "Date:". Couldn't
>> someone start it ":date:" too? Maybe there could be some optional
>> :EndMeta tag or something for these odd cases.
>
> Perhaps in the off case that someone wants to start a document with "Date:",
> they'd simply have to start the document with an empty line? That would tell the
> parser to not interpret "Date:" as metadata, but it wouldn't affect the output.
>

That sounds good and easy. Metadata starts on the first line and
continues till there is blank line. Anything after that is text.

Re: Multimarkdown Meta Data

[dsanson]

I'd vote for embedded YAML, delimited by '---' and '...', as gitit
does (is that a YAML standard? a quick glance at the YAML spec
suggests it is). Consider: sometimes people start sentences with a
word followed by a quote. Starting with "Date:" would be unusual, but
the general case not so much. Ordinary writers shouldn't be required
to know that they need to add some extra formatting to avoid
ambiguity: that onus should be on the person who wants to write
metadata.

I'd want the data format and parser to be flexible enough to allow
markdown in the data, e.g., footnotes and emphasis and citations and
links. I'd also want it to be flexible enough to allow multiple block
elements in the data. And I'd also want it to be flexible enough to
allow Boolean switches and other stuff:

---
title:   A *really great* title^[with acknowledgments]
authors:
    -     David Sanson
    -     Foo Bar
status:  in review
comments-enabled: false
abstract: |
    This is a paper that makes three points:

    +    First point
    +    Second point
    +    Third point

    The first point shows that @item1 [p. 22] is mistaken.^[But see my
discussion in @item2.]
...

I currently use multiline markdown-rich metadata in jekyll to specify
excerpts of long posts, to be used in post index templates. I've grown
fond of the power it enables.

I don't know how this interacts with the question raised in previous
threads about metadata type. I suspect it is a mess.

Another reason to go with YAML is the possibility of embedding other
kinds of structured data. For example, people have suggested that it
would be nice to be able to embed the name of a specific bibliography
file in metadata. But go a step further: imagine the ability to embed
a YAML representation of the MODS bibliography data needed to parse
all the citations in a paper. (Is YAML expressively powerful enough to
do this? Can this be implemented via citeproc just by writing an
appropriate CSL file that generates a YAML "works cited" list?) And
suppose then that pandoc could read this data directly from the the
markdown file itself and pass it as XML to citeproc-hs, so that you
have completely self-contained file: good both for portability and for
archival purposes.

And once you are doing that, you could also embed a YAML
representation of the MODS bibliography data for the paper itself. In
some future-land of fairy tales, this data would be easily found and
processed by software like Zotero, Mendeley, Bibdesk, etc.

An earlier thread discussed the question of location: should the
metadata be at the beginning of the document or the end? One nice
thing about delimited metadata is that it can be put anywhere: things
like title, date, and author might be put at the beginning; things
like bibliographic data might be put at the end.

Maybe not all of these things can be done right now. Maybe for now
only a small subset of this can be done, and would be better than
nothing. But starting now with a relatively powerful standard---
delimited YAML, for example---means that the system will be
expressively rich enough to do things like this in the future.

David

Re: Multimarkdown Meta Data

[Bruce]

On Nov 26, 12:23 pm, dsanson <dsan...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> I'd vote for embedded YAML, delimited by '---' and '...', as gitit
> does (is that a YAML standard? a quick glance at the YAML spec
> suggests it is). Consider: sometimes people start sentences with a
> word followed by a quote. Starting with "Date:" would be unusual, but
> the general case not so much. Ordinary writers shouldn't be required
> to know that they need to add some extra formatting to avoid
> ambiguity: that onus should be on the person who wants to write
> metadata.
>
> I'd want the data format and parser to be flexible enough to allow
> markdown in the data, e.g., footnotes and emphasis and citations and
> links. I'd also want it to be flexible enough to allow multiple block
> elements in the data. And I'd also want it to be flexible enough to
> allow Boolean switches and other stuff:
>
> ---
> title:   A *really great* title^[with acknowledgments]
> authors:
>     -     David Sanson
>     -     Foo Bar
> status:  in review
> comments-enabled: false
> abstract: |
>     This is a paper that makes three points:
>
>     +    First point
>     +    Second point
>     +    Third point
>
>     The first point shows that @item1 [p. 22] is mistaken.^[But see my
> discussion in @item2.]
> ...
>
> I currently use multiline markdown-rich metadata in jekyll to specify
> excerpts of long posts, to be used in post index templates. I've grown
> fond of the power it enables.
>
> I don't know how this interacts with the question raised in previous
> threads about metadata type. I suspect it is a mess.
>
> Another reason to go with YAML is the possibility of embedding other
> kinds of structured data. For example, people have suggested that it
> would be nice to be able to embed the name of a specific bibliography
> file in metadata. But go a step further: imagine the ability to embed
> a YAML representation of the MODS bibliography data needed to parse
> all the citations in a paper. (Is YAML expressively powerful enough to
> do this? Can this be implemented via citeproc just by writing an
> appropriate CSL file that generates a YAML "works cited" list?) And
> suppose then that pandoc could read this data directly from the the
> markdown file itself and pass it as XML to citeproc-hs, so that you
> have completely self-contained file: good both for portability and for
> archival purposes.

FWIW, the CSL community has been gravitating towards a common input
format based on JSON (e.g. one that closely parallels the CSL model),
which is a more widely used standard that largely overlaps with YAML.
I believe citeproc-hs supports this already, or will soon.

And I've mentioned before that if you really want to do fully
extensible metadata, I tend to prefer RDF. Admittedly, though, this is
probably overkill here.

Bruce

> And once you are doing that, you could also embed a YAML
> representation of the MODS bibliography data for the paper itself. In
> some future-land of fairy tales, this data would be easily found and
> processed by software like Zotero, Mendeley, Bibdesk, etc.
>
> An earlier thread discussed the question of location: should the
> metadata be at the beginning of the document or the end? One nice
> thing about delimited metadata is that it can be put anywhere: things
> like title, date, and author might be put at the beginning; things
> like bibliographic data might be put at the end.
>
> Maybe not all of these things can be done right now. Maybe for now
> only a small subset of this can be done, and would be better than
> nothing. But starting now with a relatively powerful standard---
> delimited YAML, for example---means that the system will be
> expressively rich enough to do things like this in the future.
>
> David

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to pandoc-discuss+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pandoc-discuss?hl=en.

Re: Multimarkdown Meta Data

[John MacFarlane]

+++ Bruce [Nov 26 10 11:02 ]:
> 
> 
> On Nov 26, 12:23 pm, dsanson <dsan...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > I'd vote for embedded YAML, delimited by '---' and '...', as gitit
> > does (is that a YAML standard? a quick glance at the YAML spec
> > suggests it is). Consider: sometimes people start sentences with a
> > word followed by a quote. Starting with "Date:" would be unusual, but
> > the general case not so much. Ordinary writers shouldn't be required
> > to know that they need to add some extra formatting to avoid
> > ambiguity: that onus should be on the person who wants to write
> > metadata.
> >
> > I'd want the data format and parser to be flexible enough to allow
> > markdown in the data, e.g., footnotes and emphasis and citations and
> > links. I'd also want it to be flexible enough to allow multiple block
> > elements in the data. And I'd also want it to be flexible enough to
> > allow Boolean switches and other stuff:
> >
> > ---
> > title:   A *really great* title^[with acknowledgments]
> > authors:
> >     -     David Sanson
> >     -     Foo Bar
> > status:  in review
> > comments-enabled: false
> > abstract: |
> >     This is a paper that makes three points:
> >
> >     +    First point
> >     +    Second point
> >     +    Third point
> >
> >     The first point shows that @item1 [p. 22] is mistaken.^[But see my
> > discussion in @item2.]
> > ...
> >
> > I currently use multiline markdown-rich metadata in jekyll to specify
> > excerpts of long posts, to be used in post index templates. I've grown
> > fond of the power it enables.
> >
> > I don't know how this interacts with the question raised in previous
> > threads about metadata type. I suspect it is a mess.
> >
> > Another reason to go with YAML is the possibility of embedding other
> > kinds of structured data. For example, people have suggested that it
> > would be nice to be able to embed the name of a specific bibliography
> > file in metadata. But go a step further: imagine the ability to embed
> > a YAML representation of the MODS bibliography data needed to parse
> > all the citations in a paper. (Is YAML expressively powerful enough to
> > do this? Can this be implemented via citeproc just by writing an
> > appropriate CSL file that generates a YAML "works cited" list?) And
> > suppose then that pandoc could read this data directly from the the
> > markdown file itself and pass it as XML to citeproc-hs, so that you
> > have completely self-contained file: good both for portability and for
> > archival purposes.
> 
> FWIW, the CSL community has been gravitating towards a common input
> format based on JSON (e.g. one that closely parallels the CSL model),
> which is a more widely used standard that largely overlaps with YAML.
> I believe citeproc-hs supports this already, or will soon.

I like David's suggestion of YAML; that's where I'm leaning.
YAML is a superset of JSON, so you could include a JSON bibliography
in the YAML metadata without change, I believe.  YAML data structures
are pretty much parallel to JSON structures, as I understand things,
only the syntax is less restrictive.  I think it's important to make
the metadata as easy as possible to write and read, so it's nice
to be able to write

title:  My title

instead of

"title":  "My title"

etc.

John

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to pandoc-discuss+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pandoc-discuss?hl=en.

Re: Multimarkdown Meta Data

[dsanson]

Neat. I suppose the ideal would be that a pandoc document with
embedded metadata is also a valid document in the appropriate metadata
format. I wonder if the JSON format that the CSL community is moving
toward includes (or could be nudged to include) support for delimited
embedded JSON.

- David

Re: Multimarkdown Meta Data

[Ryan Gray]

(A little late to this thread, but...)

Seems several suggestions for the metadata are just as MultiMarkdown
does it:

* Headers at the very top until a blank line.

* You start the file with a blank line if you have no headers so that
some automated process could add header lines to the top.

* Unrecognized header keywords are passed through the XHTML header as
meta tags and into LaTeX as \VerbatimFootnotes \def\foo{bar}

I can put two spaces at the end of each header line so that a plain
Markdown processor doesn't wrap them. [It just made me think that if
MultiMarkdown supported it, you could indent these, and a plain
processor would treat them as code, which would give you line breaks
as well]

I like the idea that a localization file could translate the metadata
strings in the user's file since it would keep the whole document in
their language.

I really would want the option of having the metadata in either the
header and/or the command line. Clearly, some people like it one way
or the other, and Fletcher Penny clearly likes it all in the document,
but I want to be able to leave some up to the processor script so that
I can take one source and process it several ways. But then, I don't
put literal target markup in my documents. If I was putting in literal
XHTML or LaTeX, I'm probably only going to those target formats, so
having metadata in the header is fine for that, but even still, why
should I have to boilerplate several header lines into each document
just to get it to come out right? Yes, that's MultiMarkdown and not
Pandoc, but is also what you could be in for if you move the metadata
to purely the document header.

I'm sure there would be debates over whether a value in the header
should override one defined as a command parameter, but maybe that can
be solved with command parameters that can set defaults and ones that
override values set in the document.

Of course, it would be *convenient* if the header format between
Pandoc and MultiMarkdown was the same, or at least compatible at say
the "Title", "Author", "Date" level and ignored unrecognized tags.