Re: pandoc/citeproc issues: multiple bibliographies, nocite, citeonly

[John MacFarlane <fiddlosopher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]

+++ Nathan Gass [Nov 30 10 00:45 ]:
> >>One possibility would be a special attribute on a header:
> >>
> >># Works cited {.bibliography src="mybib.json"}
> >>
> >># References {.bibliography src="foo.bib" include="item2,item3"
> >>                omit="item4" if-year="1999" only-if-type="primary"}
> 
> I like this, especially as this syntax would allow the addition of
> other features without adding new syntax later on.
> 
> >
> >PS.  I'm sorely tempted to put off implementing these complexities
> >til later, and release a simple version of pandoc/citeproc that just
> >constructs a bibliography of works cited in the document and puts
> >them at the end, more or less the way it does now.
> 
> +1 one to that
> 
> The only problem I see with that plan is the following:
> The sensible default now would be to always add a bibliography. On
> the other hand, if we have a syntax to  include the bibliography
> where ever we want, the better default would be to not include any
> bibliography per default.
> 
> But I think an appropriate warning for future incompatible changes
> would be enough in this case, as the documents will be easy enough
> to convert.

I'd like to avoid having documents work now and then fail later,
even if they're easy to convert.

Abstracting from the details, and from the question of whether we're
going to implement fancy filtering now, the main questions we need to decide
are:

1. whether the bibliography file should be specified in the text or as
a command-line option.  (I think having both possibilities would be
confusing -- it should be one or the other, I think.)

2. whether multiple bibliographies should be allowed, or just one
bibliography at the end.

The current implementation answers 1 by "command-line option."
If we want the bibliography file to be specified in the text,
then we need

(a)  a syntax for specifying this in the file
(b)  some architectural changes to account for the fact that
the bibliography file won't be known until after the markdown
parser has been called. (More on this below.)

For (a), I currently favor either the XML syntax or the attribute syntax after
a header that would serve as the header for the bibliography. Example:

    # References { bib="mybib.bib" }

(Though this forces you to have a header before the bibliography,
as the XML doesn't.)

This would also go well with multiple bibliographies (question 2).
And it could be extended to allow filtering when this is available
(it might already be available, in the latest darcs citeproc):

    # Primary sources { bib="mybib.bib" include-type="primary" }

Another alternative (and the only way to go if we don't allow
multiple bibliographies) would be to add a metadata block and
put the bibliography filename there.  I'm probably going to add
a metadata block at some point, but it would be nice to avoid
doing it just yet, as it would require many more decisions.

If we allow multiple bibliographies, we'll also need to change
the code in Text.Pandoc.Biblio, but I think this should be
fairly straightforward.

I guess I'm tempted to:
- allow multiple bibliographies
- specify the source file in the markdown text, as above

I'm not positive this will work, though. As I mentioned, there are technical
hurdles to having the bibliography file specified in the text. The markdown
reader itself can't do IO, so it can't read the file. So we can't read the
bibliography (or even verify that it can be found) until we've parsed the
markdown source. That means that we can't check potential citations as we
parse, to see if they're in the bibliography; instead, we have to parse
everything that might be a citation as a citation. But what if we have a
citation with id "foo" (from "@foo" in the text), but the bibliography
contains no corresponding item? Then we need to reinsert the literal text.
It's easy enough to generate "@foo" from a Cite inline, but what if the Cite
was generated by parsing latex + bibtex? This gets complicated.

I did think of one solution, which I'd like to get Andrea's feedback on (as
it might require citeproc changes). Currently when we parse a citation, we
just leave the [Inline] part empty; this gets filled in by citeproc when it
processes the citations with the bibliography.

My suggestion: instead of leaving it empty, fill in the [Inline] part of the
Cite with the literal text that should be included in the document if the
citation isn't found in the database. This way, if citeproc doesn't find the
item, it can simply leave the Cite alone (rather than raising an error). That
seems simple; it just adds a bit of complexity to the parsers, which have to
generate the "replacement text" when they parse a Cite.

Thoughts?

John