pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

Currently pandoc inserts the list of works cited at the end
of the document.  (You can provide a label like

    # References

at the end if you like; the references will be placed after
this.)

This doesn't offer much flexibility. In particular:

(1)  The writer doesn't know which part is a bibliography and
can't style it differently or even offer the option to do so through
CSS, etc.  (Unlike the footnotes...)

(2)  You can't choose not to include the works cited.

(3)  You can't have per-section bibliographies in your document;
the bibliography only comes at the very end, even if you've got
a long book.

(4) There's currently no way to include items you didn't cite,
or suppress items you did cite.

There's a proposal at

    http://gitit.net/PandocCitationGrammar

that would address (2-4) by requiring the user to put an explicit

    <references />

tag where the bibliography goes; this would then turn into a list of
works cited since the last <references />.  It would include attributes
to "omit" or "include" citation keys.  (I suppose we could even have
a "source" tag that specifies the bib file to use, but this would
only work if we no longer had to read the bib file prior to parsing
the markdown -- see the previous "issue".)

We could address (1) as well by adding a special Bibliography block
to pandoc.  This would have some cost, as each writer would have to
be told how to treat it, but it would give more flexibility. For
example, in HTML, the bibliography could be put in a special
<div>. (Note that with the current system, if you want your bibliography in a
div, you can put it in a section and use the --section-divs option.)

I'd like to get some feedback on whether we want this flexibility,
or whether the current simple behavior sufficies.

John

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Bruce]

On Nov 21, 2:32 pm, John MacFarlane <fiddlosop...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Currently pandoc inserts the list of works cited at the end
> of the document.  (You can provide a label like
>
>     # References
>
> at the end if you like; the references will be placed after
> this.)
>
> This doesn't offer much flexibility. In particular:
>
> (1)  The writer doesn't know which part is a bibliography and
> can't style it differently or even offer the option to do so through
> CSS, etc.  (Unlike the footnotes...)
>
> (2)  You can't choose not to include the works cited.
>
> (3)  You can't have per-section bibliographies in your document;
> the bibliography only comes at the very end, even if you've got
> a long book.
>
> (4) There's currently no way to include items you didn't cite,
> or suppress items you did cite.

There's another issue that will inevitably come up, so I should
mention it now:

(5) You can't have multiple sections per bibliography. Common use case
is you need a section for legal cases, another for primary documents,
and a third (which we might call default) of the secondary academic
literature.

I believe there are one or more LaTeX packages that support this sort
of thing, but I don't recall their solution. And IIRC, discussion
around this in the CSL world has tended to center around bib groups,
where you might optionally assign references to a named group, which
would correspond to an equivalent names section.

> There's a proposal at
>
>    http://gitit.net/PandocCitationGrammar
>
> that would address (2-4) by requiring the user to put an explicit
>
>     <references />
>
> tag where the bibliography goes; this would then turn into a list of
> works cited since the last <references />.  It would include attributes
> to "omit" or "include" citation keys.  (I suppose we could even have
> a "source" tag that specifies the bib file to use, but this would
> only work if we no longer had to read the bib file prior to parsing
> the markdown -- see the previous "issue".)
>
> We could address (1) as well by adding a special Bibliography block
> to pandoc.  This would have some cost, as each writer would have to
> be told how to treat it, but it would give more flexibility. For
> example, in HTML, the bibliography could be put in a special
> <div>. (Note that with the current system, if you want your bibliography in a
> div, you can put it in a section and use the --section-divs option.)
>
> I'd like to get some feedback on whether we want this flexibility,
> or whether the current simple behavior sufficies.

I don't have specific feedback on your proposal ATM, but generally
think the use cases are important.

Bruce

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to pandoc-discuss+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pandoc-discuss?hl=en.

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Tillmann Rendel]

John MacFarlane wrote:
> There's a proposal at
>
>      http://gitit.net/PandocCitationGrammar
>
> that would address (2-4) by requiring the user to put an explicit
>
>      <references />
>
> tag where the bibliography goes;[...]

I like the idea of explicitly specifying where the bibliography goes. I 
don't like that syntax too much, however, because

  (a) it is HTML-focused, so it might look awkward in LaTeX-target
      markdown files with lots of literal LaTeX code

  (b) it looks like literal HTML, but is actually (pandoc extended)
      markdown, which might be confusing


I had a similar problem in my use of pandoc. I want to insert 
system-generated content, such as a table of contents, at a specific 
point in the document. So I decided to write a script which replaces a 
whole section (or subsection etc.) with system-generated content, based 
on the name of the section. So for example, the snippet:

     Table of Contents
     =================

     It doesn't matter what you write here

     Introduction
     ============

would get translated to something like:

     \tableofcontents
     \section{Introduction}

I think this would work well for references, too.


The main benefit is that one can use the input format's syntax for 
headings to specify "special sections" like table of contents or 
references. The input document therefore looks natural. (One could even 
have one's editor and pandoc automatically fill in the section content 
with a textual version of the system-generated data, for some kind of 
semi-WYSIWYG effect).

The main drawback is that the name of the section is relevant, so this 
introduces a language dependency. For my scripts, I just have a 
command-line flag for the various special sections:

   "--toc=Table of Contents"


Of course, I can easily extend my script to translate a 
"References"-section into a "<references/>" marker, so I don't the mind 
the concrete syntax too much.

Just wanted to share my thoughts.

   Tillmann

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Nathan Gass]

On 21.11.10 22:46, Bruce wrote:
>
>
> On Nov 21, 2:32 pm, John MacFarlane<fiddlosop...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>  wrote:
>> Currently pandoc inserts the list of works cited at the end
>> of the document.  (You can provide a label like
>>
>>      # References
>>
>> at the end if you like; the references will be placed after
>> this.)
>>
>> This doesn't offer much flexibility. In particular:
>>
>> (1)  The writer doesn't know which part is a bibliography and
>> can't style it differently or even offer the option to do so through
>> CSS, etc.  (Unlike the footnotes...)
>>
>> (2)  You can't choose not to include the works cited.
>>
>> (3)  You can't have per-section bibliographies in your document;
>> the bibliography only comes at the very end, even if you've got
>> a long book.
>>
>> (4) There's currently no way to include items you didn't cite,
>> or suppress items you did cite.
>
> There's another issue that will inevitably come up, so I should
> mention it now:
>
> (5) You can't have multiple sections per bibliography. Common use case
> is you need a section for legal cases, another for primary documents,
> and a third (which we might call default) of the secondary academic
> literature.
>
> I believe there are one or more LaTeX packages that support this sort
> of thing, but I don't recall their solution. And IIRC, discussion
> around this in the CSL world has tended to center around bib groups,
> where you might optionally assign references to a named group, which
> would correspond to an equivalent names section.

As I understand the common use case, the section a reference belongs 
into is given by the type of the reference? So this info could actually 
be encoded in the bibliography? Pandoc then only needs a way to add all 
needed bibliography sections, and not a way to mark references belonging 
to a specific section. How individual are this sections named? I ask 
this because currently I consider attaching the bibliographic info to 
specific named sections the best solution. For this to work we need to 
cover most/all possible names for such a section.

This way we don't need any new syntax to support this use case. Legal 
cases would be marked as such in the bibliography file, and pandoc would 
attach the rendered bibliographic info to a differently named section if 
it finds such a section in the text.

Nathan

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

+++ Nathan Gass [Nov 22 10 17:02 ]:
> On 21.11.10 22:46, Bruce wrote:
> >
> >
> >On Nov 21, 2:32 pm, John MacFarlane<fiddlosop...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>  wrote:
> >>Currently pandoc inserts the list of works cited at the end
> >>of the document.  (You can provide a label like
> >>
> >>     # References
> >>
> >>at the end if you like; the references will be placed after
> >>this.)
> >>
> >>This doesn't offer much flexibility. In particular:
> >>
> >>(1)  The writer doesn't know which part is a bibliography and
> >>can't style it differently or even offer the option to do so through
> >>CSS, etc.  (Unlike the footnotes...)
> >>
> >>(2)  You can't choose not to include the works cited.
> >>
> >>(3)  You can't have per-section bibliographies in your document;
> >>the bibliography only comes at the very end, even if you've got
> >>a long book.
> >>
> >>(4) There's currently no way to include items you didn't cite,
> >>or suppress items you did cite.
> >
> >There's another issue that will inevitably come up, so I should
> >mention it now:
> >
> >(5) You can't have multiple sections per bibliography. Common use case
> >is you need a section for legal cases, another for primary documents,
> >and a third (which we might call default) of the secondary academic
> >literature.
> >
> >I believe there are one or more LaTeX packages that support this sort
> >of thing, but I don't recall their solution. And IIRC, discussion
> >around this in the CSL world has tended to center around bib groups,
> >where you might optionally assign references to a named group, which
> >would correspond to an equivalent names section.
> 
> As I understand the common use case, the section a reference belongs
> into is given by the type of the reference? So this info could
> actually be encoded in the bibliography? Pandoc then only needs a
> way to add all needed bibliography sections, and not a way to mark
> references belonging to a specific section. How individual are this
> sections named? I ask this because currently I consider attaching
> the bibliographic info to specific named sections the best solution.
> For this to work we need to cover most/all possible names for such a
> section.
> 
> This way we don't need any new syntax to support this use case.
> Legal cases would be marked as such in the bibliography file, and
> pandoc would attach the rendered bibliographic info to a differently
> named section if it finds such a section in the text.

I'm not sure from Bruce's message whether CSL yet supports named
groups, but that seems the way to go in the future.  So the
bibliography file would say that the entry belongs to the "primary"
group, and then in your pandoc document you'd have something like:

<references filter-group="primary" />

I agree that we don't want to clutter up the actual in-document
citation syntax with syntax for groups.

John

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Andrea Rossato]

On Sun, Nov 21, 2010 at 11:32:29AM -0800, John MacFarlane wrote:
> There's a proposal at
> 
>     http://gitit.net/PandocCitationGrammar
> 
> that would address (2-4) by requiring the user to put an explicit
> 
>     <references />
> 
> tag where the bibliography goes; this would then turn into a list of
> works cited since the last <references />.  It would include attributes
> to "omit" or "include" citation keys.

I think it would be nice if we could follow the way citeproc-js deals
with this problem:

http://gsl-nagoya-u.net/http/pub/citeproc-doc.html#selective-output

This is basically a list of ("field","value") tuples for selecting
entries include in or exclude from the bibliography (there are four
commands: include/select or exclude/quash depending on whether all or
any of the tuples are to be matched).

What do you think?

> (I suppose we could even have a "source" tag that specifies the
> bib file to use, but this would only work if we no longer had to
> read the bib file prior to parsing the markdown -- see the previous
> "issue".)

What about parsing the markdown text for the reference field, if
present read the source file with the bibliographic data and then
parse the actual markdown text?

> We could address (1) as well by adding a special Bibliography block
> to pandoc.  This would have some cost, as each writer would have to
> be told how to treat it, but it would give more flexibility. For
> example, in HTML, the bibliography could be put in a special
> <div>. (Note that with the current system, if you want your bibliography in a
> div, you can put it in a section and use the --section-divs option.)
> 
> I'd like to get some feedback on whether we want this flexibility,
> or whether the current simple behavior sufficies.

There would be also some specific needs for bibliographic formatting
to be taken into account, like second field alignment or per-author
listing. See more about it here:
http://citationstyles.org/downloads/specification.html#id73

I must admit I didn't have yet the time to dig into those issues and
I'd prefer to leave them for future enhancements. I think we should
focus on getting the citation part right, first, since it involves
syntax modification. The bibliography, instead, is a matter which does
not affect the way documents are written.

Andrea

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

+++ Andrea Rossato [Nov 23 10 00:21 ]:
> On Sun, Nov 21, 2010 at 11:32:29AM -0800, John MacFarlane wrote:
> > There's a proposal at
> > 
> >     http://gitit.net/PandocCitationGrammar
> > 
> > that would address (2-4) by requiring the user to put an explicit
> > 
> >     <references />
> > 
> > tag where the bibliography goes; this would then turn into a list of
> > works cited since the last <references />.  It would include attributes
> > to "omit" or "include" citation keys.
> 
> I think it would be nice if we could follow the way citeproc-js deals
> with this problem:
> 
> http://gsl-nagoya-u.net/http/pub/citeproc-doc.html#selective-output
> 
> This is basically a list of ("field","value") tuples for selecting
> entries include in or exclude from the bibliography (there are four
> commands: include/select or exclude/quash depending on whether all or
> any of the tuples are to be matched).
> 
> What do you think?

That sounds very flexible.  How would the syntax work?
Maybe:

<references src="mybib.bib">
  <select type="book" categories="1990s" />
  <quash author="Smith" />
</references>

> > (I suppose we could even have a "source" tag that specifies the
> > bib file to use, but this would only work if we no longer had to
> > read the bib file prior to parsing the markdown -- see the previous
> > "issue".)
> 
> What about parsing the markdown text for the reference field, if
> present read the source file with the bibliographic data and then
> parse the actual markdown text?

This could be done, but there'd be a significant performance
penalty, so I'd rather avoid it.

> > We could address (1) as well by adding a special Bibliography block
> > to pandoc.  This would have some cost, as each writer would have to
> > be told how to treat it, but it would give more flexibility. For
> > example, in HTML, the bibliography could be put in a special
> > <div>. (Note that with the current system, if you want your bibliography in a
> > div, you can put it in a section and use the --section-divs option.)
> > 
> > I'd like to get some feedback on whether we want this flexibility,
> > or whether the current simple behavior sufficies.
> 
> There would be also some specific needs for bibliographic formatting
> to be taken into account, like second field alignment or per-author
> listing. See more about it here:
> http://citationstyles.org/downloads/specification.html#id73
> 
> I must admit I didn't have yet the time to dig into those issues and
> I'd prefer to leave them for future enhancements. I think we should
> focus on getting the citation part right, first, since it involves
> syntax modification. The bibliography, instead, is a matter which does
> not affect the way documents are written.

Yes, it seems fine to wait on refining the formatting of the
bibliography. But we ought to settle the issues above about how
a writer indicates where the bibliography goes, what its source
is, etc.

John

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Nathan Gass]

On 23.11.10 07:30, John MacFarlane wrote:
> +++ Andrea Rossato [Nov 23 10 00:21 ]:
>> On Sun, Nov 21, 2010 at 11:32:29AM -0800, John MacFarlane wrote:
>>
>> What about parsing the markdown text for the reference field, if
>> present read the source file with the bibliographic data and then
>> parse the actual markdown text?
>
> This could be done, but there'd be a significant performance
> penalty, so I'd rather avoid it.

Do you have any benchmarks for pandoc? Performance seems to be a very 
important concern for you, while in my usage it never was an issue. Of 
course I neither have large documents to parse nor use pandoc in any 
high-traffic web environment, but I would be interested in which region 
pandoc currently is and how big this penalties are compared to that.

>> There would be also some specific needs for bibliographic formatting
>> to be taken into account, like second field alignment or per-author
>> listing. See more about it here:
>> http://citationstyles.org/downloads/specification.html#id73
>>
>> I must admit I didn't have yet the time to dig into those issues and
>> I'd prefer to leave them for future enhancements. I think we should
>> focus on getting the citation part right, first, since it involves
>> syntax modification. The bibliography, instead, is a matter which does
>> not affect the way documents are written.
>
> Yes, it seems fine to wait on refining the formatting of the
> bibliography. But we ought to settle the issues above about how
> a writer indicates where the bibliography goes, what its source
> is, etc.

For whats it worth, in my opinion the formatting of the bibliography 
belongs into the style, or some separate pandoc configuration, and not 
in the document itself.

Nathan

>
> John
>

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

+++ Nathan Gass [Nov 23 10 11:04 ]:
> On 23.11.10 07:30, John MacFarlane wrote:
> >+++ Andrea Rossato [Nov 23 10 00:21 ]:
> >>On Sun, Nov 21, 2010 at 11:32:29AM -0800, John MacFarlane wrote:
> >>
> >>What about parsing the markdown text for the reference field, if
> >>present read the source file with the bibliographic data and then
> >>parse the actual markdown text?
> >
> >This could be done, but there'd be a significant performance
> >penalty, so I'd rather avoid it.
> 
> Do you have any benchmarks for pandoc? Performance seems to be a
> very important concern for you, while in my usage it never was an
> issue. Of course I neither have large documents to parse nor use
> pandoc in any high-traffic web environment, but I would be
> interested in which region pandoc currently is and how big this
> penalties are compared to that.

I want pandoc to be as fast as possible.  It may not matter if
you're just converting some documents at your desktop, but if
pandoc is used in producing web pages, it can matter a lot,
in terms of response time and server load.

Benchmarks vary a lot depending on your source file.  Here
are some rough figures on one fairly long (178K) markdown file:

Markdown.pl         5 sec
php markdown extra  1.7 sec
pandoc              1 sec
lunamark            0.6 sec
peg-markdown        0.15 sec
discount            0.03 sec

Pandoc does a lot more than lunamark, peg-markdown, and discount,
but still, I think there's lots of room for improvement here, and
the differences are certainly important to people considering
using pandoc in a web app.

> >>There would be also some specific needs for bibliographic formatting
> >>to be taken into account, like second field alignment or per-author
> >>listing. See more about it here:
> >>http://citationstyles.org/downloads/specification.html#id73
> >>
> >>I must admit I didn't have yet the time to dig into those issues and
> >>I'd prefer to leave them for future enhancements. I think we should
> >>focus on getting the citation part right, first, since it involves
> >>syntax modification. The bibliography, instead, is a matter which does
> >>not affect the way documents are written.
> >
> >Yes, it seems fine to wait on refining the formatting of the
> >bibliography. But we ought to settle the issues above about how
> >a writer indicates where the bibliography goes, what its source
> >is, etc.
> 
> For whats it worth, in my opinion the formatting of the bibliography
> belongs into the style, or some separate pandoc configuration, and
> not in the document itself.

Yes, I think we all agree on that!  But issues about where the
bibliography is placed and what gets put into it need to be addressed
in the document.

JOhn

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Bruce]

On Nov 23, 5:04 am, Nathan Gass <xa...-8UOIJiGH10pyDzI6CaY1VQ@public.gmane.org> wrote:
> On 23.11.10 07:30, John MacFarlane wrote:
>
> > +++ Andrea Rossato [Nov 23 10 00:21 ]:
> >> On Sun, Nov 21, 2010 at 11:32:29AM -0800, John MacFarlane wrote:
>
> >> What about parsing the markdown text for the reference field, if
> >> present read the source file with the bibliographic data and then
> >> parse the actual markdown text?
>
> > This could be done, but there'd be a significant performance
> > penalty, so I'd rather avoid it.
>
> Do you have any benchmarks for pandoc? Performance seems to be a very
> important concern for you, while in my usage it never was an issue. Of
> course I neither have large documents to parse nor use pandoc in any
> high-traffic web environment, but I would be interested in which region
> pandoc currently is and how big this penalties are compared to that.
>
> >> There would be also some specific needs for bibliographic formatting
> >> to be taken into account, like second field alignment or per-author
> >> listing. See more about it here:
> >>http://citationstyles.org/downloads/specification.html#id73
>
> >> I must admit I didn't have yet the time to dig into those issues and
> >> I'd prefer to leave them for future enhancements. I think we should
> >> focus on getting the citation part right, first, since it involves
> >> syntax modification. The bibliography, instead, is a matter which does
> >> not affect the way documents are written.
>
> > Yes, it seems fine to wait on refining the formatting of the
> > bibliography. But we ought to settle the issues above about how
> > a writer indicates where the bibliography goes, what its source
> > is, etc.
>
> For whats it worth, in my opinion the formatting of the bibliography
> belongs into the style, or some separate pandoc configuration, and not
> in the document itself.

FWIW from the CSL perspective, there are some areas over overlap
between the domain of CSL and the domain of, say, a document styling
system (line-spacing, some font details, even whether you call a
reference list a "Bibliography" or "Works Cited"). We cover what we
can (say line spacing between bib entries), but defer otherwise.

As a user of pandoc particularly interested in(X)HTML, I'd consider it
valuable at least if the bib was wrapped in a div that I can style.

Bruce

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to pandoc-discuss+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pandoc-discuss?hl=en.

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

+++ Bruce [Nov 23 10 10:39 ]:
> 
> 
> On Nov 23, 5:04 am, Nathan Gass <xa...-8UOIJiGH10pyDzI6CaY1VQ@public.gmane.org> wrote:
> > On 23.11.10 07:30, John MacFarlane wrote:
> >
> > > +++ Andrea Rossato [Nov 23 10 00:21 ]:
> > >> On Sun, Nov 21, 2010 at 11:32:29AM -0800, John MacFarlane wrote:
> >
> > >> What about parsing the markdown text for the reference field, if
> > >> present read the source file with the bibliographic data and then
> > >> parse the actual markdown text?
> >
> > > This could be done, but there'd be a significant performance
> > > penalty, so I'd rather avoid it.
> >
> > Do you have any benchmarks for pandoc? Performance seems to be a very
> > important concern for you, while in my usage it never was an issue. Of
> > course I neither have large documents to parse nor use pandoc in any
> > high-traffic web environment, but I would be interested in which region
> > pandoc currently is and how big this penalties are compared to that.
> >
> > >> There would be also some specific needs for bibliographic formatting
> > >> to be taken into account, like second field alignment or per-author
> > >> listing. See more about it here:
> > >>http://citationstyles.org/downloads/specification.html#id73
> >
> > >> I must admit I didn't have yet the time to dig into those issues and
> > >> I'd prefer to leave them for future enhancements. I think we should
> > >> focus on getting the citation part right, first, since it involves
> > >> syntax modification. The bibliography, instead, is a matter which does
> > >> not affect the way documents are written.
> >
> > > Yes, it seems fine to wait on refining the formatting of the
> > > bibliography. But we ought to settle the issues above about how
> > > a writer indicates where the bibliography goes, what its source
> > > is, etc.
> >
> > For whats it worth, in my opinion the formatting of the bibliography
> > belongs into the style, or some separate pandoc configuration, and not
> > in the document itself.
> 
> FWIW from the CSL perspective, there are some areas over overlap
> between the domain of CSL and the domain of, say, a document styling
> system (line-spacing, some font details, even whether you call a
> reference list a "Bibliography" or "Works Cited"). We cover what we
> can (say line spacing between bib entries), but defer otherwise.
> 
> As a user of pandoc particularly interested in(X)HTML, I'd consider it
> valuable at least if the bib was wrapped in a div that I can style.

My current plan will make this happen.

The plan is to introduce a new Bibliography block element. This will carry
information about the bibliography source file, excludes & includes (see
Andrea's earlier message), and a formatted Block list produced by citeproc.
Writers can do what they like with it. If the writer is producing latex using
natbib, for example, it will just write "\bibliography{foo}" or something like
that. The default, though, will be to write the formatted blocks produced
by citeproc. In the HTML writer, these can be put in a div with class
"bibliography".

John

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to pandoc-discuss+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pandoc-discuss?hl=en.

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Nathan Gass]

On 22.11.10 00:26, Tillmann Rendel wrote:
> John MacFarlane wrote:
>> There's a proposal at
>>
>> http://gitit.net/PandocCitationGrammar
>>
>> that would address (2-4) by requiring the user to put an explicit
>>
>> <references />
>>
>> tag where the bibliography goes;[...]
>
> I like the idea of explicitly specifying where the bibliography goes. I
> don't like that syntax too much, however, because
>
> (a) it is HTML-focused, so it might look awkward in LaTeX-target
> markdown files with lots of literal LaTeX code
>
> (b) it looks like literal HTML, but is actually (pandoc extended)
> markdown, which might be confusing

I just wanted to add that I completely agree here. I find syntax which 
looks like html, but actually is not pass-through html but special 
markdown syntax confusing.

>
>
> I had a similar problem in my use of pandoc. I want to insert
> system-generated content, such as a table of contents, at a specific
> point in the document. So I decided to write a script which replaces a
> whole section (or subsection etc.) with system-generated content, based
> on the name of the section. So for example, the snippet:
>
> Table of Contents
> =================
>
> It doesn't matter what you write here
>
> Introduction
> ============
>
> would get translated to something like:
>
> \tableofcontents
> \section{Introduction}
>
> I think this would work well for references, too.
>
>
> The main benefit is that one can use the input format's syntax for
> headings to specify "special sections" like table of contents or
> references. The input document therefore looks natural. (One could even
> have one's editor and pandoc automatically fill in the section content
> with a textual version of the system-generated data, for some kind of
> semi-WYSIWYG effect).
>
> The main drawback is that the name of the section is relevant, so this
> introduces a language dependency. For my scripts, I just have a
> command-line flag for the various special sections:
>
> "--toc=Table of Contents"
>

Other possibilities would be to add some label syntax, similar to the 
\label{} of latex, to just recognize most or even all possible headers 
for such a section, or use a special code block, which is at least 
clearly markdown syntax. Actually, when I think about it, the last one 
strikes me as a very good alternative. So instead of

     <reference>
     some config
     </reference>

we simply use

    ~~~~~~~~~~ {.reference}
    some config
    ~~~~~~~~~~

Nathan

>
> Of course, I can easily extend my script to translate a
> "References"-section into a "<references/>" marker, so I don't the mind
> the concrete syntax too much.
>
> Just wanted to share my thoughts.
>
> Tillmann
>

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

+++ Nathan Gass [Nov 24 10 02:29 ]:
> On 22.11.10 00:26, Tillmann Rendel wrote:
> >John MacFarlane wrote:
> >>There's a proposal at
> >>
> >>http://gitit.net/PandocCitationGrammar
> >>
> >>that would address (2-4) by requiring the user to put an explicit
> >>
> >><references />
> >>
> >>tag where the bibliography goes;[...]
> >
> >I like the idea of explicitly specifying where the bibliography goes. I
> >don't like that syntax too much, however, because
> >
> >(a) it is HTML-focused, so it might look awkward in LaTeX-target
> >markdown files with lots of literal LaTeX code
> >
> >(b) it looks like literal HTML, but is actually (pandoc extended)
> >markdown, which might be confusing
> 
> I just wanted to add that I completely agree here. I find syntax
> which looks like html, but actually is not pass-through html but
> special markdown syntax confusing.
> 
> >
> >
> >I had a similar problem in my use of pandoc. I want to insert
> >system-generated content, such as a table of contents, at a specific
> >point in the document. So I decided to write a script which replaces a
> >whole section (or subsection etc.) with system-generated content, based
> >on the name of the section. So for example, the snippet:
> >
> >Table of Contents
> >=================
> >
> >It doesn't matter what you write here
> >
> >Introduction
> >============
> >
> >would get translated to something like:
> >
> >\tableofcontents
> >\section{Introduction}
> >
> >I think this would work well for references, too.
> >
> >
> >The main benefit is that one can use the input format's syntax for
> >headings to specify "special sections" like table of contents or
> >references. The input document therefore looks natural. (One could even
> >have one's editor and pandoc automatically fill in the section content
> >with a textual version of the system-generated data, for some kind of
> >semi-WYSIWYG effect).
> >
> >The main drawback is that the name of the section is relevant, so this
> >introduces a language dependency. For my scripts, I just have a
> >command-line flag for the various special sections:
> >
> >"--toc=Table of Contents"
> >
> 
> Other possibilities would be to add some label syntax, similar to
> the \label{} of latex, to just recognize most or even all possible
> headers for such a section, or use a special code block, which is at
> least clearly markdown syntax. Actually, when I think about it, the
> last one strikes me as a very good alternative. So instead of
> 
>     <reference>
>     some config
>     </reference>
> 
> we simply use
> 
>    ~~~~~~~~~~ {.reference}
>    some config
>    ~~~~~~~~~~


I'm open to suggestions other than the HTMLish syntax.  But I don't
think this is obviously better.  For one thing, it looks like a code
block.  And we'd have to implement some mini-language inside the
block to specify the includes, excludes, source, etc.  If we use
XML, we're using something that's already familiar.

One possibility would be a special attribute on a header:

# Works cited {.bibliography src="mybib.json"}

# References {.bibliography src="foo.bib" include="item2,item3"
               omit="item4" if-year="1999" only-if-type="primary"}

John

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

+++ John MacFarlane [Nov 23 10 19:33 ]:
> +++ Nathan Gass [Nov 24 10 02:29 ]:
> > On 22.11.10 00:26, Tillmann Rendel wrote:
> > >John MacFarlane wrote:
> > >>There's a proposal at
> > >>
> > >>http://gitit.net/PandocCitationGrammar
> > >>
> > >>that would address (2-4) by requiring the user to put an explicit
> > >>
> > >><references />
> > >>
> > >>tag where the bibliography goes;[...]
> > >
> > >I like the idea of explicitly specifying where the bibliography goes. I
> > >don't like that syntax too much, however, because
> > >
> > >(a) it is HTML-focused, so it might look awkward in LaTeX-target
> > >markdown files with lots of literal LaTeX code
> > >
> > >(b) it looks like literal HTML, but is actually (pandoc extended)
> > >markdown, which might be confusing
> > 
> > I just wanted to add that I completely agree here. I find syntax
> > which looks like html, but actually is not pass-through html but
> > special markdown syntax confusing.
> > 
> > >
> > >
> > >I had a similar problem in my use of pandoc. I want to insert
> > >system-generated content, such as a table of contents, at a specific
> > >point in the document. So I decided to write a script which replaces a
> > >whole section (or subsection etc.) with system-generated content, based
> > >on the name of the section. So for example, the snippet:
> > >
> > >Table of Contents
> > >=================
> > >
> > >It doesn't matter what you write here
> > >
> > >Introduction
> > >============
> > >
> > >would get translated to something like:
> > >
> > >\tableofcontents
> > >\section{Introduction}
> > >
> > >I think this would work well for references, too.
> > >
> > >
> > >The main benefit is that one can use the input format's syntax for
> > >headings to specify "special sections" like table of contents or
> > >references. The input document therefore looks natural. (One could even
> > >have one's editor and pandoc automatically fill in the section content
> > >with a textual version of the system-generated data, for some kind of
> > >semi-WYSIWYG effect).
> > >
> > >The main drawback is that the name of the section is relevant, so this
> > >introduces a language dependency. For my scripts, I just have a
> > >command-line flag for the various special sections:
> > >
> > >"--toc=Table of Contents"
> > >
> > 
> > Other possibilities would be to add some label syntax, similar to
> > the \label{} of latex, to just recognize most or even all possible
> > headers for such a section, or use a special code block, which is at
> > least clearly markdown syntax. Actually, when I think about it, the
> > last one strikes me as a very good alternative. So instead of
> > 
> >     <reference>
> >     some config
> >     </reference>
> > 
> > we simply use
> > 
> >    ~~~~~~~~~~ {.reference}
> >    some config
> >    ~~~~~~~~~~
> 
> 
> I'm open to suggestions other than the HTMLish syntax.  But I don't
> think this is obviously better.  For one thing, it looks like a code
> block.  And we'd have to implement some mini-language inside the
> block to specify the includes, excludes, source, etc.  If we use
> XML, we're using something that's already familiar.
> 
> One possibility would be a special attribute on a header:
> 
> # Works cited {.bibliography src="mybib.json"}
> 
> # References {.bibliography src="foo.bib" include="item2,item3"
>                omit="item4" if-year="1999" only-if-type="primary"}

PS.  I'm sorely tempted to put off implementing these complexities
til later, and release a simple version of pandoc/citeproc that just
constructs a bibliography of works cited in the document and puts
them at the end, more or less the way it does now.  The one twist
I'd add would be this:  the bibliography is inserted only if the
document ends with a Header.  That way you could suppress the
bibliography by not ending with a Header.

We could keep this convention later even if we implemented a
<references /> element. So existing documents would not break.

What we'd be missing:

- ability to have multiple bibliographies
- ability to include works not cited in bibliographies
- ability to suppress works cited in bibbliographies
- ability to filter bibliographies by fields (e.g. source=primary)

These things would be nice for some users, but they add a lot of
complexity and could be implemented later.

Note:  users who want to style the bibliography could still do so.
Just specify --section-divs and the section containing the bibliography
(the section at the end of the document) will be put in a div with
its own id.

John

Re: Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Andrea Rossato]

On Tue, Nov 23, 2010 at 09:06:32PM -0800, John MacFarlane wrote:
> PS.  I'm sorely tempted to put off implementing these complexities
> til later, and release a simple version of pandoc/citeproc that just
> constructs a bibliography of works cited in the document and puts
> them at the end, more or less the way it does now.  The one twist
> I'd add would be this:  the bibliography is inserted only if the
> document ends with a Header.  That way you could suppress the
> bibliography by not ending with a Header.

If you do not want a bibliography you can just remove the
<bibliography> element from a style. This is the cleaner way to do
it I think.

> We could keep this convention later even if we implemented a
> <references /> element. So existing documents would not break.
> 
> What we'd be missing:
> 
> - ability to have multiple bibliographies
> - ability to include works not cited in bibliographies
> - ability to suppress works cited in bibbliographies
> - ability to filter bibliographies by fields (e.g. source=primary)
> 
> These things would be nice for some users, but they add a lot of
> complexity and could be implemented later.

I'm working to get the filtering options done. citeproc type signature
must be changed and I'd prefer to do it before a new release (since
the API has already been changed).

citeproc would become:

citeproc :: ProcOpts -> Style -> [Reference] -> Citations -> BiblioData

with
data ProcOpts
    = ProcOpts
      { bibOpts :: [BibOpts]
      }

(the record could be extended at a later time). And:

data BibOpts
    = Select  [(String, String)]
    | Include [(String, String)]
    | Exclude [(String, String)]
    | Quash   [(String, String)]

(the tuple is (field,value))

Multiple bibliographies would require calling citeproc for everyone.
This means splitting the pandoc document - in other words, partially
rewriting Pandoc.Biblio.

Can we do it now? If no I'm not going to work on the citeproc side any
longer for the time being.

For the syntax: I like the xml syntax you proposed.

Andrea

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

+++ Andrea Rossato [Nov 24 10 21:51 ]:
> On Tue, Nov 23, 2010 at 09:06:32PM -0800, John MacFarlane wrote:
> > PS.  I'm sorely tempted to put off implementing these complexities
> > til later, and release a simple version of pandoc/citeproc that just
> > constructs a bibliography of works cited in the document and puts
> > them at the end, more or less the way it does now.  The one twist
> > I'd add would be this:  the bibliography is inserted only if the
> > document ends with a Header.  That way you could suppress the
> > bibliography by not ending with a Header.
> 
> If you do not want a bibliography you can just remove the
> <bibliography> element from a style. This is the cleaner way to do
> it I think.

Makes sense.

> > We could keep this convention later even if we implemented a
> > <references /> element. So existing documents would not break.
> > 
> > What we'd be missing:
> > 
> > - ability to have multiple bibliographies
> > - ability to include works not cited in bibliographies
> > - ability to suppress works cited in bibbliographies
> > - ability to filter bibliographies by fields (e.g. source=primary)
> > 
> > These things would be nice for some users, but they add a lot of
> > complexity and could be implemented later.
> 
> I'm working to get the filtering options done. citeproc type signature
> must be changed and I'd prefer to do it before a new release (since
> the API has already been changed).
> 
> citeproc would become:
> 
> citeproc :: ProcOpts -> Style -> [Reference] -> Citations -> BiblioData
> 
> with
> data ProcOpts
>     = ProcOpts
>       { bibOpts :: [BibOpts]
>       }
> 
> (the record could be extended at a later time). And:
> 
> data BibOpts
>     = Select  [(String, String)]
>     | Include [(String, String)]
>     | Exclude [(String, String)]
>     | Quash   [(String, String)]
> 
> (the tuple is (field,value))
> 
> Multiple bibliographies would require calling citeproc for everyone.
> This means splitting the pandoc document - in other words, partially
> rewriting Pandoc.Biblio.
> 
> Can we do it now? If no I'm not going to work on the citeproc side any
> longer for the time being.

There's at least one thing that needs to be done anyway -- the
spacing issue between multiple locators.  (See the failing
tests when you do 'make test'.)

> For the syntax: I like the xml syntax you proposed.

I'm a bit uncertain how to proceed, because the xml syntax seems
unpopular (judging from responses on the list), and I don't have
a good alternative in mind.

Also, I'm not *sure* that the select/include/exclude/quash system
is the best way to go on filtering.  It works pretty well with
the xml syntax, but less well with some of the other options.

Hence the temptation to avoid solving all these problems and
just release something workable. But let me think on it some more.

John

Re: Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Andrea Rossato]

On Wed, Nov 24, 2010 at 11:42:40PM -0800, John MacFarlane wrote:
> +++ Andrea Rossato [Nov 24 10 21:51 ]:
> > Can we do it now? If no I'm not going to work on the citeproc side any
> > longer for the time being.
> 
> There's at least one thing that needs to be done anyway -- the
> spacing issue between multiple locators.  (See the failing
> tests when you do 'make test'.)

I mean that I'm not going to work on this specific issue anymore (I'll
leave it for the next release), to concentrate on the remaining issues
(page ranges, which involves also the problem you reported, style
version recognition, rich text formatting and flip-flopping, etc.).

> > For the syntax: I like the xml syntax you proposed.
> 
> I'm a bit uncertain how to proceed, because the xml syntax seems
> unpopular (judging from responses on the list), and I don't have
> a good alternative in mind.
> 
> Also, I'm not *sure* that the select/include/exclude/quash system
> is the best way to go on filtering.  It works pretty well with
> the xml syntax, but less well with some of the other options.
> 
> Hence the temptation to avoid solving all these problems and
> just release something workable. But let me think on it some more.

If you want to wait I have no objections. Still I would like to change
the type signature of citeproc to include the ProcOpts parameter, so
that we don't need to change it at a latter time.

Related to that: in pandoc.cabal the citeproc-hs dependency has been
set to:

citeproc-hs >= 0.3 && < 0.4

Do we want to drop the upper bound?

Andrea

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

+++ Andrea Rossato [Nov 25 10 09:09 ]:
> On Wed, Nov 24, 2010 at 11:42:40PM -0800, John MacFarlane wrote:
> > +++ Andrea Rossato [Nov 24 10 21:51 ]:
> > > Can we do it now? If no I'm not going to work on the citeproc side any
> > > longer for the time being.
> > 
> > There's at least one thing that needs to be done anyway -- the
> > spacing issue between multiple locators.  (See the failing
> > tests when you do 'make test'.)
> 
> I mean that I'm not going to work on this specific issue anymore (I'll
> leave it for the next release), to concentrate on the remaining issues
> (page ranges, which involves also the problem you reported, style
> version recognition, rich text formatting and flip-flopping, etc.).
> 
> > > For the syntax: I like the xml syntax you proposed.
> > 
> > I'm a bit uncertain how to proceed, because the xml syntax seems
> > unpopular (judging from responses on the list), and I don't have
> > a good alternative in mind.
> > 
> > Also, I'm not *sure* that the select/include/exclude/quash system
> > is the best way to go on filtering.  It works pretty well with
> > the xml syntax, but less well with some of the other options.
> > 
> > Hence the temptation to avoid solving all these problems and
> > just release something workable. But let me think on it some more.
> 
> If you want to wait I have no objections. Still I would like to change
> the type signature of citeproc to include the ProcOpts parameter, so
> that we don't need to change it at a latter time.
> 
> Related to that: in pandoc.cabal the citeproc-hs dependency has been
> set to:
> 
> citeproc-hs >= 0.3 && < 0.4
> 
> Do we want to drop the upper bound?

I think these upper bounds are generally a good idea.  This way you
can release a new version of citeproc-hs with an incompatible API,
and pandoc will still build.

I was looking a bit at the code in citeproc that parses locators.

    parseLocator :: String -> (String, String)
    parseLocator s
        | "b"    `isPrefixOf` formatField s = mk "book"
        | "ch"   `isPrefixOf` formatField s = mk "chapter"
        | "co"   `isPrefixOf` formatField s = mk "column"
        | "fi"   `isPrefixOf` formatField s = mk "figure"
        | "fo"   `isPrefixOf` formatField s = mk "folio"
        | "i"    `isPrefixOf` formatField s = mk "issue"
        | "l"    `isPrefixOf` formatField s = mk "line"
        | "n"    `isPrefixOf` formatField s = mk "note"
        | "o"    `isPrefixOf` formatField s = mk "opus"
        | "para" `isPrefixOf` formatField s = mk "paragraph"
        | "part" `isPrefixOf` formatField s = mk "part"
        | "p"    `isPrefixOf` formatField s = mk "page"
        | "sec"  `isPrefixOf` formatField s = mk "section"
        | "sub"  `isPrefixOf` formatField s = mk "sub verbo"
        | "ve"   `isPrefixOf` formatField s = mk "verse"
        | "v"    `isPrefixOf` formatField s = mk "volume"
        | otherwise                         =    ([], [])
        where
          mk c = if null s then ([], []) else (,) c . concat . tail . words $ s

Am I right that this will work for English and Italian, but not
Swedish and Chinese? Is there a way to localize this and make it less
language-centric?

John

Re: Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Andrea Rossato]

On Sat, Nov 27, 2010 at 07:23:52AM -0800, John MacFarlane wrote:
> I was looking a bit at the code in citeproc that parses locators.
> 
>     parseLocator :: String -> (String, String)
>     parseLocator s
> 
> Am I right that this will work for English and Italian, but not
> Swedish and Chinese? Is there a way to localize this and make it less
> language-centric?


Possibly other European languages should be covered too. I do not see
any simple solution. I'll have a look at what citeproc-js does (Frank
works in Japan and is very committed to multi-language support, so I
hope he'll be able to help me).

Andrea

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

+++ Andrea Rossato [Nov 27 10 20:01 ]:
> On Sat, Nov 27, 2010 at 07:23:52AM -0800, John MacFarlane wrote:
> > I was looking a bit at the code in citeproc that parses locators.
> > 
> >     parseLocator :: String -> (String, String)
> >     parseLocator s
> > 
> > Am I right that this will work for English and Italian, but not
> > Swedish and Chinese? Is there a way to localize this and make it less
> > language-centric?
> 
> 
> Possibly other European languages should be covered too. I do not see
> any simple solution. I'll have a look at what citeproc-js does (Frank
> works in Japan and is very committed to multi-language support, so I
> hope he'll be able to help me).
> 

It seems to me that the needed information is already in the locale
files.  E.g. in the US english file:

      <!-- SHORT LOCATOR FORMS -->
      <term name="book" form="short">bk.</term>
      <term name="chapter" form="short">chap.</term>
      <term name="column" form="short">col.</term>
      <term name="figure" form="short">fig.</term>
      <term name="folio" form="short">f.</term>
      <term name="issue" form="short">no.</term>
      <term name="opus" form="short">op.</term>
      <term name="page" form="short">
         <single>p.</single>
         <multiple>pp.</multiple>
      </term>
      <term name="paragraph" form="short">para.</term>
      <term name="part" form="short">pt.</term>
      <term name="section" form="short">sec.</term>
      <term name="sub verbo" form="short">
         <single>s.v.</single>
         <multiple>s.vv.</multiple>
      </term>
      <term name="verse" form="short">
         <single>v.</single>
         <multiple>vv.</multiple>
      </term>
      <term name="volume" form="short">
         <single>vol.</single>
         <multiple>vols.</multiple>
      </term>

And there are counterparts in the others.  Couldn't you just
extract this information and use it instead of the hard-coded
values?

John

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Nathan Gass]

On 24.11.10 06:06, John MacFarlane wrote:
> +++ John MacFarlane [Nov 23 10 19:33 ]:
>> +++ Nathan Gass [Nov 24 10 02:29 ]:
>>>
>>> Other possibilities would be to add some label syntax, similar to
>>> the \label{} of latex, to just recognize most or even all possible
>>> headers for such a section, or use a special code block, which is at
>>> least clearly markdown syntax. Actually, when I think about it, the
>>> last one strikes me as a very good alternative. So instead of
>>>
>>>      <reference>
>>>      some config
>>>      </reference>
>>>
>>> we simply use
>>>
>>>     ~~~~~~~~~~ {.reference}
>>>     some config
>>>     ~~~~~~~~~~
>>
>>
>> I'm open to suggestions other than the HTMLish syntax.  But I don't
>> think this is obviously better.  For one thing, it looks like a code
>> block.  And we'd have to implement some mini-language inside the
>> block to specify the includes, excludes, source, etc.  If we use
>> XML, we're using something that's already familiar.

I see the possibilities of markdown quite a bit broader than XML. I will 
want to show my markdown documents to people and encourage them to use 
it themself, who will not know XML at all and not be very comfortable 
editing it. Most of them will only need one Bibliography with all cited 
works and so not use any special syntax for the more complex use-cases.

For me code blocks are a very specialized feature, which is absolutely 
useless in many usages of markdown and pandoc. I therefore tend to steal 
this syntax for arbitrary plugins. That is why I considered this a good 
idea. But I like your idea below better anyway.

>>
>> One possibility would be a special attribute on a header:
>>
>> # Works cited {.bibliography src="mybib.json"}
>>
>> # References {.bibliography src="foo.bib" include="item2,item3"
>>                 omit="item4" if-year="1999" only-if-type="primary"}

I like this, especially as this syntax would allow the addition of other 
features without adding new syntax later on.

>
> PS.  I'm sorely tempted to put off implementing these complexities
> til later, and release a simple version of pandoc/citeproc that just
> constructs a bibliography of works cited in the document and puts
> them at the end, more or less the way it does now.

+1 one to that

The only problem I see with that plan is the following:
The sensible default now would be to always add a bibliography. On the 
other hand, if we have a syntax to  include the bibliography where ever 
we want, the better default would be to not include any bibliography per 
default.

But I think an appropriate warning for future incompatible changes would 
be enough in this case, as the documents will be easy enough to convert.

Nathan

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

+++ Nathan Gass [Nov 30 10 00:45 ]:
> >>One possibility would be a special attribute on a header:
> >>
> >># Works cited {.bibliography src="mybib.json"}
> >>
> >># References {.bibliography src="foo.bib" include="item2,item3"
> >>                omit="item4" if-year="1999" only-if-type="primary"}
> 
> I like this, especially as this syntax would allow the addition of
> other features without adding new syntax later on.
> 
> >
> >PS.  I'm sorely tempted to put off implementing these complexities
> >til later, and release a simple version of pandoc/citeproc that just
> >constructs a bibliography of works cited in the document and puts
> >them at the end, more or less the way it does now.
> 
> +1 one to that
> 
> The only problem I see with that plan is the following:
> The sensible default now would be to always add a bibliography. On
> the other hand, if we have a syntax to  include the bibliography
> where ever we want, the better default would be to not include any
> bibliography per default.
> 
> But I think an appropriate warning for future incompatible changes
> would be enough in this case, as the documents will be easy enough
> to convert.

I'd like to avoid having documents work now and then fail later,
even if they're easy to convert.

Abstracting from the details, and from the question of whether we're
going to implement fancy filtering now, the main questions we need to decide
are:

1. whether the bibliography file should be specified in the text or as
a command-line option.  (I think having both possibilities would be
confusing -- it should be one or the other, I think.)

2. whether multiple bibliographies should be allowed, or just one
bibliography at the end.

The current implementation answers 1 by "command-line option."
If we want the bibliography file to be specified in the text,
then we need

(a)  a syntax for specifying this in the file
(b)  some architectural changes to account for the fact that
the bibliography file won't be known until after the markdown
parser has been called. (More on this below.)

For (a), I currently favor either the XML syntax or the attribute syntax after
a header that would serve as the header for the bibliography. Example:

    # References { bib="mybib.bib" }

(Though this forces you to have a header before the bibliography,
as the XML doesn't.)

This would also go well with multiple bibliographies (question 2).
And it could be extended to allow filtering when this is available
(it might already be available, in the latest darcs citeproc):

    # Primary sources { bib="mybib.bib" include-type="primary" }

Another alternative (and the only way to go if we don't allow
multiple bibliographies) would be to add a metadata block and
put the bibliography filename there.  I'm probably going to add
a metadata block at some point, but it would be nice to avoid
doing it just yet, as it would require many more decisions.

If we allow multiple bibliographies, we'll also need to change
the code in Text.Pandoc.Biblio, but I think this should be
fairly straightforward.

I guess I'm tempted to:
- allow multiple bibliographies
- specify the source file in the markdown text, as above

I'm not positive this will work, though. As I mentioned, there are technical
hurdles to having the bibliography file specified in the text. The markdown
reader itself can't do IO, so it can't read the file. So we can't read the
bibliography (or even verify that it can be found) until we've parsed the
markdown source. That means that we can't check potential citations as we
parse, to see if they're in the bibliography; instead, we have to parse
everything that might be a citation as a citation. But what if we have a
citation with id "foo" (from "@foo" in the text), but the bibliography
contains no corresponding item? Then we need to reinsert the literal text.
It's easy enough to generate "@foo" from a Cite inline, but what if the Cite
was generated by parsing latex + bibtex? This gets complicated.

I did think of one solution, which I'd like to get Andrea's feedback on (as
it might require citeproc changes). Currently when we parse a citation, we
just leave the [Inline] part empty; this gets filled in by citeproc when it
processes the citations with the bibliography.

My suggestion: instead of leaving it empty, fill in the [Inline] part of the
Cite with the literal text that should be included in the document if the
citation isn't found in the database. This way, if citeproc doesn't find the
item, it can simply leave the Cite alone (rather than raising an error). That
seems simple; it just adds a bit of complexity to the parsers, which have to
generate the "replacement text" when they parse a Cite.

Thoughts?

John

Re: Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Andrea Rossato]

On Tue, Nov 30, 2010 at 07:25:56PM -0800, John MacFarlane wrote:
> I guess I'm tempted to:
> - allow multiple bibliographies

That would be easy, indeed.

> - specify the source file in the markdown text, as above

keep in mind that multiple bibliographic databases should be allowed
(for instance to mix bibtex and json data).

> I'm not positive this will work, though. As I mentioned, there are technical
> hurdles to having the bibliography file specified in the text. The markdown
> reader itself can't do IO, so it can't read the file. So we can't read the
> bibliography (or even verify that it can be found) until we've parsed the
> markdown source. That means that we can't check potential citations as we
> parse, to see if they're in the bibliography; instead, we have to parse
> everything that might be a citation as a citation. But what if we have a
> citation with id "foo" (from "@foo" in the text), but the bibliography
> contains no corresponding item? Then we need to reinsert the literal text.
> It's easy enough to generate "@foo" from a Cite inline, but what if the Cite
> was generated by parsing latex + bibtex? This gets complicated.
> 
> I did think of one solution, which I'd like to get Andrea's feedback on (as
> it might require citeproc changes). Currently when we parse a citation, we
> just leave the [Inline] part empty; this gets filled in by citeproc when it
> processes the citations with the bibliography.
> 
> My suggestion: instead of leaving it empty, fill in the [Inline] part of the
> Cite with the literal text that should be included in the document if the
> citation isn't found in the database. This way, if citeproc doesn't find the
> item, it can simply leave the Cite alone (rather than raising an error). That
> seems simple; it just adds a bit of complexity to the parsers, which have to
> generate the "replacement text" when they parse a Cite.
> 
> Thoughts?

That would be fine with me. But I just wanted to let you know, though,
that I modified citeproc so that, when a citation is not found in the
bibliographic data, an error is emitted:

   "[CSL BIBLIOGRAPHIC DATA ERROR: reference " ++ show (citationId c) ++ " not found.]"

I don't know if this is going to solve your problem.

More generally I'd prefer the metadata approach you were talking
about. If this requires more time to get mature I think we could now
stick with the present approach: a single bibliography automatically
added at the end of the Pandoc document, and multiple bibliographic
databases via the command line --bibliography option.

That would leave us free to take every possible path for implementing
multiple bibliographies and for setting the source biblio data in the
source document.

These are my preferences but I'll just followed your directions.

Andrea

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

+++ Andrea Rossato [Dec 01 10 14:03 ]:
> On Tue, Nov 30, 2010 at 07:25:56PM -0800, John MacFarlane wrote:
> > I guess I'm tempted to:
> > - allow multiple bibliographies
> 
> That would be easy, indeed.
> 
> > - specify the source file in the markdown text, as above
> 
> keep in mind that multiple bibliographic databases should be allowed
> (for instance to mix bibtex and json data).
> 
> > I'm not positive this will work, though. As I mentioned, there are technical
> > hurdles to having the bibliography file specified in the text. The markdown
> > reader itself can't do IO, so it can't read the file. So we can't read the
> > bibliography (or even verify that it can be found) until we've parsed the
> > markdown source. That means that we can't check potential citations as we
> > parse, to see if they're in the bibliography; instead, we have to parse
> > everything that might be a citation as a citation. But what if we have a
> > citation with id "foo" (from "@foo" in the text), but the bibliography
> > contains no corresponding item? Then we need to reinsert the literal text.
> > It's easy enough to generate "@foo" from a Cite inline, but what if the Cite
> > was generated by parsing latex + bibtex? This gets complicated.
> > 
> > I did think of one solution, which I'd like to get Andrea's feedback on (as
> > it might require citeproc changes). Currently when we parse a citation, we
> > just leave the [Inline] part empty; this gets filled in by citeproc when it
> > processes the citations with the bibliography.
> > 
> > My suggestion: instead of leaving it empty, fill in the [Inline] part of the
> > Cite with the literal text that should be included in the document if the
> > citation isn't found in the database. This way, if citeproc doesn't find the
> > item, it can simply leave the Cite alone (rather than raising an error). That
> > seems simple; it just adds a bit of complexity to the parsers, which have to
> > generate the "replacement text" when they parse a Cite.
> > 
> > Thoughts?
> 
> That would be fine with me. But I just wanted to let you know, though,
> that I modified citeproc so that, when a citation is not found in the
> bibliographic data, an error is emitted:
> 
>    "[CSL BIBLIOGRAPHIC DATA ERROR: reference " ++ show (citationId c) ++ " not found.]"

This is fine the way the code currently works, since pandoc *shouldn't*
be passing citations that aren't in the bibliography to citeproc.
But it would have to be changed if we did things the way I sketched
above.  With this approach, citeproc would simply leave Citations alone
if the entry was not found. (There might be difficulties I'm not
anticipating; I don't know the code.)

> I don't know if this is going to solve your problem.
> 
> More generally I'd prefer the metadata approach you were talking
> about. If this requires more time to get mature I think we could now
> stick with the present approach: a single bibliography automatically
> added at the end of the Pandoc document, and multiple bibliographic
> databases via the command line --bibliography option.
> 
> That would leave us free to take every possible path for implementing
> multiple bibliographies and for setting the source biblio data in the
> source document.

The only drawback I see to proceeding this way is that these future
changes would require changes in existing documents, unless we kept
the --bibliography option in addition to whatever mechanism we come
up with for multiply bibliographies and in-text specification of
the source files.

But maybe we could just do that:  keep --bibliography for legacy
files.  That would allow us to do a release sooner, without settling
all of these issues.

John

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[siram.berlin-Re5JQEeQqe8AvxtiuMwx3w]

[-- Attachment #1.1: Type: text/plain, Size: 1351 bytes --]

I know this has been discussed many years ago, but I couldn't find any more 
information about this on github or anywhere else. 

Has something like this ever been implemented?

I'm currently writing a paper (md+latex) and using pandoc to create a pdf. 
I would like to have two bibliographies, one for sources and one for 
references (with a #title and a \pagebreak in between). 
I don't really care whether I would need to separate my bibliography into 
two different .bib files or just apply some sort of class.. 

I know I can pass in two bib files, but they just end up in the same 
bibliography. Can I somehow adjust my template or my .md files to seperate 
them out as you were considering here? Are there any best practices? Could 
you hint me to something?

Best,
Simon

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To post to this group, send email to pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/9802cbc1-fa98-43f9-b42b-b7781066f375%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[-- Attachment #1.2: Type: text/html, Size: 1899 bytes --]

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane]

Unfortunately, I don't think there's any good way of doing
this currently.  It would be nice to be able to do this.

+++ siram.berlin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org [Jul 27 17 01:18 ]:
>   I know this has been discussed many years ago, but I couldn't find any
>   more information about this on github or anywhere else.
>   Has something like this ever been implemented?
>   I'm currently writing a paper (md+latex) and using pandoc to create a
>   pdf.
>   I would like to have two bibliographies, one for sources and one for
>   references (with a #title and a \pagebreak in between).
>   I don't really care whether I would need to separate my bibliography
>   into two different .bib files or just apply some sort of class..
>   I know I can pass in two bib files, but they just end up in the same
>   bibliography. Can I somehow adjust my template or my .md files to
>   seperate them out as you were considering here? Are there any best
>   practices? Could you hint me to something?
>   Best,
>   Simon
>
>   --
>   You received this message because you are subscribed to the Google
>   Groups "pandoc-discuss" group.
>   To unsubscribe from this group and stop receiving emails from it, send
>   an email to [1]pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>   To post to this group, send email to
>   [2]pandoc-discuss-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>   To view this discussion on the web visit
>   [3]https://groups.google.com/d/msgid/pandoc-discuss/9802cbc1-fa98-43f9-
>   b42b-b7781066f375%40googlegroups.com.
>   For more options, visit [4]https://groups.google.com/d/optout.
>
>References
>
>   1. mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
>   2. mailto:pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
>   3. https://groups.google.com/d/msgid/pandoc-discuss/9802cbc1-fa98-43f9-b42b-b7781066f375-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org?utm_medium=email&utm_source=footer
>   4. https://groups.google.com/d/optout

Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[Pranesh Prakash]

[-- Attachment #1.1: Type: text/plain, Size: 1554 bytes --]

Dear Simon,
In case you're still looking for a solution, the multiple-bibliographies 
lua filter allows you do to exactly this:
https://github.com/pandoc/lua-filters/tree/master/multiple-bibliographies

Regards,
Pranesh


On Thursday, 27 July, 2017 at 1:48:44 pm UTC+5:30 siram....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:

> I know this has been discussed many years ago, but I couldn't find any 
> more information about this on github or anywhere else. 
>
> Has something like this ever been implemented?
>
> I'm currently writing a paper (md+latex) and using pandoc to create a pdf. 
> I would like to have two bibliographies, one for sources and one for 
> references (with a #title and a \pagebreak in between). 
> I don't really care whether I would need to separate my bibliography into 
> two different .bib files or just apply some sort of class.. 
>
> I know I can pass in two bib files, but they just end up in the same 
> bibliography. Can I somehow adjust my template or my .md files to seperate 
> them out as you were considering here? Are there any best practices? Could 
> you hint me to something?
>
> Best,
> Simon
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/19e3868a-6e4e-42a3-8636-75a95e925946n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2257 bytes --]

AW: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[denis.maier-FfwAq0itz3ofv37vnLkPlQ]

[-- Attachment #1: Type: text/plain, Size: 2763 bytes --]

HI Pranesh,

thanks for bringing this up again. While this filter is certainly nice, I still think a pandoc-native solution would have its merits. (And could be much more flexible…)

Best,
Denis

Von: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> Im Auftrag von Pranesh Prakash
Gesendet: Donnerstag, 19. November 2020 14:14
An: pandoc-discuss <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>
Betreff: Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

Dear Simon,
In case you're still looking for a solution, the multiple-bibliographies lua filter allows you do to exactly this:
https://github.com/pandoc/lua-filters/tree/master/multiple-bibliographies

Regards,
Pranesh


On Thursday, 27 July, 2017 at 1:48:44 pm UTC+5:30 siram....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org<mailto:siram....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
I know this has been discussed many years ago, but I couldn't find any more information about this on github or anywhere else.

Has something like this ever been implemented?

I'm currently writing a paper (md+latex) and using pandoc to create a pdf.
I would like to have two bibliographies, one for sources and one for references (with a #title and a \pagebreak in between).
I don't really care whether I would need to separate my bibliography into two different .bib files or just apply some sort of class..

I know I can pass in two bib files, but they just end up in the same bibliography. Can I somehow adjust my template or my .md files to seperate them out as you were considering here? Are there any best practices? Could you hint me to something?

Best,
Simon
--
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org<mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/19e3868a-6e4e-42a3-8636-75a95e925946n%40googlegroups.com<https://groups.google.com/d/msgid/pandoc-discuss/19e3868a-6e4e-42a3-8636-75a95e925946n%40googlegroups.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/001a76dbc7d145008abc3b792e2a5e5c%40ub.unibe.ch.

[-- Attachment #2: Type: text/html, Size: 6998 bytes --]