ANN: pandoc 2.11

ANN: pandoc 2.11

[John MacFarlane]

I'm pleased to announce the release of pandoc 2.11,
available in the usual places:

Binary packages & changelog:
  https://github.com/jgm/pandoc/releases/tag/2.11

Source & API documentation:
  http://hackage.haskell.org/package/pandoc-2.11

The biggest new feature is built-in citation processing (using
the new citeproc library).  It is no longer necessary to use
the external filter pandoc-citeproc to process citations (and
we no longer provide pandoc-citeproc in the binary packages).
Instead, just use the --citeproc option where you previously
would have used --filter pandoc-citeproc.

The new citation processing is faster and more accurate than
pandoc-citeproc.  In most cases it should serve as a drop-in
replacement, but note the following:

- Whereas pandoc-citeproc searches the `~/.csl` directory for CSL
  styles, pandoc searches the `csl` subdirectory of
  the pandoc user data directory (e.g., `~/.pandoc/csl` or
  `~/.local/share/pandoc/csl`).  Users who already keep
  CSL styles in `~/.csl` may find it convenient to add a
  symlink.

- Copac, EndNote, ISI, MEDLINE, MODS, and RIS bibliographies
  are no longer supported.  If you use one of these formats,
  you may use the `bibutils` utility to convert to BibLaTeX.
  We now support only BibTeX, BibLaTeX, CSL JSON,
  and pandoc's YAML/Markdown analogue of CSL JSON.

- Using the `--bibliography` option (or including
  `bibliography` in YAML metadata) no longer triggers
  citation processing implicitly: one must always use the
  `--citeproc` option if citation processing is wanted.

- Instead of using pandoc-citeproc with the --bib2json or
  --bib2yaml option to convert BibTeX and BibLaTeX bibliographies
  to CSL JSON or YAML, one can now use pandoc directly.

In addition to this feature, there are many bug fixes and
small improvements: see the changelog for details.

Thanks to everyone who contributed, and especially to
new pandoc contributors Blake Eryx, Diego Balseiro, Dmitry Volodin,
Leonard Rosenthol, Michael Hoffmann, a-vrma, argent0, and
the-solipsist.

Re: ANN: pandoc 2.11

['Nick Bart' via pandoc-discuss]

Great news, in particular about the new citeproc library.

Just one small observation: `pandoc -F pandoc-citeproc test.md` (where `pandoc` is 2.11, latest dev compiled from source, and `pandoc-citeproc` either 0.17.0.2, compiled from the latest available source, or 0.17, installed via homebrew) emits a warning:

`[WARNING] Deprecated: pandoc-citeproc filter. Use --citeproc instead.`

... followed immediately by an error message:

```
pandoc-citeproc: Error in $: Incompatible API versions: encoded with [1,22] but attempted to decode with [1,21].
CallStack (from HasCallStack):
  error, called at ./Text/Pandoc/JSON.hs:112:48 in pandoc-types-1.21-Lb7evYKD6jv46JnhbWrkOK:Text.Pandoc.JSON
Error running filter pandoc-citeproc:
Filter returned error status 1
```

So unless I’m missing something, pandoc 2.11 just isn’t compatible with pandoc-citeproc at all.

If that’s true, wouldn’t it be clearer to terminate with an error message if `-F pandoc-citeproc` is used instead of merely emitting a warning?

Or, since I suspect many users not closely following pandoc’s development are likely to be caught off guard by this, automatically replace `-F pandoc-citeproc` by `-C` if encountered on the command line, still emitting a warning of course that `-F pandoc-citeproc` will become fully obsolete in a later version?

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/QuQHd9MObO1zEpJdu9u4ggRmzn62E3sfkexsbaKCMQqekwKJ7wFNBuQeNrAt7VXDHZ0WneOzd0qodQoYfEEqOzYPzsxsPi0bIid32lrCHcc%3D%40protonmail.com.

Re: ANN: pandoc 2.11

[Pranesh Prakash]

[-- Attachment #1.1: Type: text/plain, Size: 1114 bytes --]

Thanks a lot for the amazing work, John & al.

On Sunday, 11 October, 2020 at 10:06:29 pm UTC+5:30 John MacFarlane wrote:

> - Using the `--bibliography` option (or including 
> `bibliography` in YAML metadata) no longer triggers 
> citation processing implicitly: one must always use the 
> `--citeproc` option if citation processing is wanted. 
>

Could you explain why this change was necessary? Because this seems to 
remove a smart assumption (that if `bibliography` is there in the YAML or 
as an option, then citation processing needs to be done).

Also, is there a YAML metadata option (either in-file or in the defaults 
file) to indicate that citation processing needs to be done?

~ Pranesh


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/004e40ea-0329-4dde-824d-3d650e58c791n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1810 bytes --]

Re: ANN: pandoc 2.11

[John MacFarlane]

"'Nick Bart' via pandoc-discuss"
<pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> writes:

> So unless I’m missing something, pandoc 2.11 just isn’t compatible with pandoc-citeproc at all.

This is not quite true.  Pandoc 2.11 isn't compatible with
pandoc-citeproc binaries that were compiled against earlier
versions of pandoc-types.  And since we don't distribute a new
binary, you'd have to compile it yourself.  But it should still be
possible to use a self-compiled version.

> If that’s true, wouldn’t it be clearer to terminate with an error message if `-F pandoc-citeproc` is used instead of merely emitting a warning?
>
> Or, since I suspect many users not closely following pandoc’s development are likely to be caught off guard by this, automatically replace `-F pandoc-citeproc` by `-C` if encountered on the command line, still emitting a warning of course that `-F pandoc-citeproc` will become fully obsolete in a later version?

I thought about doing this but worried that there might be some
folks who want to keep using pandoc-citeproc (due to small
differences in the output, Copac support, or whatever).  But
curious what people think.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m2blh7ecbc.fsf%40MacBook-Pro.hsd1.ca.comcast.net.

Re: ANN: pandoc 2.11

['Denis Maier' via pandoc-discuss]

Am 12.10.2020 um 16:41 schrieb John MacFarlane:
> "'Nick Bart' via pandoc-discuss"
> <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> writes:
>
>> So unless I’m missing something, pandoc 2.11 just isn’t compatible with pandoc-citeproc at all.
> This is not quite true.  Pandoc 2.11 isn't compatible with
> pandoc-citeproc binaries that were compiled against earlier
> versions of pandoc-types.  And since we don't distribute a new
> binary, you'd have to compile it yourself.  But it should still be
> possible to use a self-compiled version.
>
>> If that’s true, wouldn’t it be clearer to terminate with an error message if `-F pandoc-citeproc` is used instead of merely emitting a warning?
>>
>> Or, since I suspect many users not closely following pandoc’s development are likely to be caught off guard by this, automatically replace `-F pandoc-citeproc` by `-C` if encountered on the command line, still emitting a warning of course that `-F pandoc-citeproc` will become fully obsolete in a later version?
> I thought about doing this but worried that there might be some
> folks who want to keep using pandoc-citeproc (due to small
> differences in the output, Copac support, or whatever).  But
> curious what people think.

I will certainly just switch over to the new citeproc since it's much 
more accurate. Also, of course, because I hope that future development 
(e.g. CSL 1.1, multiple bibliographies, and such) will happen here.

Best,
Denis

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/b7eee3a8-223e-06fa-2d82-e3eead5b0214%40mailbox.org.

Re: ANN: pandoc 2.11

[John MacFarlane]

Pranesh Prakash <the.solipsist-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> Thanks a lot for the amazing work, John & al.
>
> On Sunday, 11 October, 2020 at 10:06:29 pm UTC+5:30 John MacFarlane wrote:
>
>> - Using the `--bibliography` option (or including 
>> `bibliography` in YAML metadata) no longer triggers 
>> citation processing implicitly: one must always use the 
>> `--citeproc` option if citation processing is wanted. 
>>
>
> Could you explain why this change was necessary? Because this seems to 
> remove a smart assumption (that if `bibliography` is there in the YAML or 
> as an option, then citation processing needs to be done).

There are some cases where you wouldn't want citation processing
to happen -- e.g. if you're reformatting using `pandoc -f
markdown -t markdown`.  I have a feeling there was another
reason that I'm not remembering.  Anyway, comments welcome on
this issue too.  We could potentially change back.

> Also, is there a YAML metadata option (either in-file or in the defaults 
> file) to indicate that citation processing needs to be done?

Not in metadata, but in defaults, yes, you can either do

citeproc: true

or put citeproc in your list of filters (allowing you control
over order).

Re: ANN: pandoc 2.11

['Denis Maier' via pandoc-discuss]

Am 12.10.2020 um 16:45 schrieb John MacFarlane:
> Pranesh Prakash <the.solipsist-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
>> Thanks a lot for the amazing work, John & al.
>>
>> On Sunday, 11 October, 2020 at 10:06:29 pm UTC+5:30 John MacFarlane wrote:
>>
>>> - Using the `--bibliography` option (or including
>>> `bibliography` in YAML metadata) no longer triggers
>>> citation processing implicitly: one must always use the
>>> `--citeproc` option if citation processing is wanted.
>>>
>> Could you explain why this change was necessary? Because this seems to
>> remove a smart assumption (that if `bibliography` is there in the YAML or
>> as an option, then citation processing needs to be done).
> There are some cases where you wouldn't want citation processing
> to happen -- e.g. if you're reformatting using `pandoc -f
> markdown -t markdown`.  I have a feeling there was another
> reason that I'm not remembering.  Anyway, comments welcome on
> this issue too.  We could potentially change back.

If I remember correctly, you'd have to enable citation processing 
explicity when going to markdown by disabling the citation extension, 
right? Or has that changed? Anyway, I've always found this a bit awkward.

Denis

Re: ANN: pandoc 2.11

[John MacFarlane]

Denis Maier <denismaier-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:


> If I remember correctly, you'd have to enable citation processing 
> explicity when going to markdown by disabling the citation extension, 
> right? Or has that changed? Anyway, I've always found this a bit awkward.

If you do -t markdown+citations, then citation processing may
still be done, but pandoc ignores the results.  This needlessly
slows things down, so that's one reason to avoid making
citation processing automatic.  Another reason is that there
was always an awkward inconsistency: if you specified your
bibliographic data using a `references` field in the YAML
metadata, you needed to do `--filter pandoc-citeproc` explicitly,
but if you used a bibliography, you didn't. The implicit
triggering of the pandoc-citeproc filter was something I added
for backwards-compatibility a long, long time ago and always
intended to sunset eventually.

Re: ANN: pandoc 2.11

[Pranesh Prakash]

[-- Attachment #1.1: Type: text/plain, Size: 1711 bytes --]



On Monday, 12 October, 2020 at 8:16:11 pm UTC+5:30 John MacFarlane wrote:

> There are some cases where you wouldn't want citation processing 
> to happen -- e.g. if you're reformatting using `pandoc -f 
> markdown -t markdown`. I have a feeling there was another 
> reason that I'm not remembering. Anyway, comments welcome on 
> this issue too. We could potentially change back. 


Makes sense. I don't have a strong view one way or the other.  The one 
advantage is that it would help confused users like this,[1] whereas it can 
be overridden by "--metadata bibliography=FILE",[2] by those who don't want 
citeproc to do the citation processing.

[1]: https://github.com/jgm/pandoc/issues/6713
[2]: https://github.com/jgm/pandoc/issues/1849
  

> > Also, is there a YAML metadata option (either in-file or in the defaults 
> > file) to indicate that citation processing needs to be done? 
>
> Not in metadata, but in defaults, yes, you can either do 
>
> citeproc: true 
>
> or put citeproc in your list of filters (allowing you control 
> over order). 
>

"citeproc: true" would work well for me. 

I just checked, and the manual already reflects both the "citeproc" method, 
as well as the "filters: - citeproc"/"filters: - type: citeproc" method.  
I'm sorry for not having checked earlier.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/e72b4312-d64f-41ed-938d-8ba73b88eb98n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2531 bytes --]

Re: ANN: pandoc 2.11

[Pranesh Prakash]

[-- Attachment #1.1: Type: text/plain, Size: 1523 bytes --]

On Sunday, 11 October, 2020 at 10:06:29 pm UTC+5:30 John MacFarlane wrote:

> - Copac, EndNote, ISI, MEDLINE, MODS, and RIS bibliographies 
> are no longer supported. If you use one of these formats, 
> you may use the `bibutils` utility to convert to BibLaTeX. 
> We now support only BibTeX, BibLaTeX, CSL JSON, 
> and pandoc's YAML/Markdown analogue of CSL JSON. 
>

Given a choice between those four formats which is preferable?  I'm 
guessing CSL JSON is the "native" format used by Pandoc, but I wanted to 
confirm.

I use Zotero with the BetterBibTex add-on.  That add-on allows me to export 
to:
1. BibLaTex
2. BibTeX
3. CSL JSON
4. BibLaTeX JSON
5. CSL YAML

And separately, I tried to convert a CSL JSON file produced using Zotero's 
internal CSL JSON export format as well as using BetterBibTex's CSL JSON 
export format.

I got this error:
$ pandoc coc.bbtcsl.json -s -t markdown
JSON parse error: Error in $: mempty

$ pandoc coc.plaincsl.json -s -t markdown
JSON parse error: Error in $: mempty

$ pandoc --version
pandoc 2.11
Compiled with pandoc-types 1.22, texmath 0.12.0.3, skylighting 0.10.0.2

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/0bc97e34-8feb-4b14-84c9-92c064e303e2n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2320 bytes --]

Re: ANN: pandoc 2.11

['Nick Bart' via pandoc-discuss]

[-- Attachment #1: Type: text/plain, Size: 2681 bytes --]

There is no such thing as "BibLaTeX JSON". "Better BibTeX JSON" is a format used by BBT for its internal purposes only, and not supported by pandoc.

The "native" bibliography data formats for pandoc when using citeproc are CSL JSON and CSL YAML, and should be preferred. Both are on a par, though CSL YAML offers a few more options concerning in-field markup.

BibLaTeX is a close third. Its data model differs from CSL, but pandoc’s conversion to CSL is usually all but lossless.

BibTeX on the other hand should be avoided if at all possible.

As to converting CSL JSON files using pandoc, I noticed this too:

You need to add `-f csljson` to the command line.

@jgm, I think some examples in the documentation need to be fixed to include `-f csljson`.

Or could pandoc be made to automatically distinguish CSL JSON files from other .json files?

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, October 13, 2020 11:30 PM, Pranesh Prakash <the.solipsist@gmail.com> wrote:

> On Sunday, 11 October, 2020 at 10:06:29 pm UTC+5:30 John MacFarlane wrote:
>
>> - Copac, EndNote, ISI, MEDLINE, MODS, and RIS bibliographies
>> are no longer supported. If you use one of these formats,
>> you may use the `bibutils` utility to convert to BibLaTeX.
>> We now support only BibTeX, BibLaTeX, CSL JSON,
>> and pandoc's YAML/Markdown analogue of CSL JSON.
>
> Given a choice between those four formats which is preferable? I'm guessing CSL JSON is the "native" format used by Pandoc, but I wanted to confirm.
>
> I use Zotero with the BetterBibTex add-on. That add-on allows me to export to:
> 1. BibLaTex
> 2. BibTeX
> 3. CSL JSON
> 4. BibLaTeX JSON
> 5. CSL YAML
>
> And separately, I tried to convert a CSL JSON file produced using Zotero's internal CSL JSON export format as well as using BetterBibTex's CSL JSON export format.
>
> I got this error:
> $ pandoc coc.bbtcsl.json -s -t markdown
> JSON parse error: Error in $: mempty
>
> $ pandoc coc.plaincsl.json -s -t markdown
> JSON parse error: Error in $: mempty
>
> $ pandoc --version
> pandoc 2.11
> Compiled with pandoc-types 1.22, texmath 0.12.0.3, skylighting 0.10.0.2

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/EWgxgNikoC-UWOI6W369l9CRlH-wjU7zrqeloOKmlw_ehTHnmjhCUTUsE2s_geAKy1QCUxXYwvqo3ydaqs6y0NrwoDEw8QIUk20zys3zhGk%3D%40protonmail.com.

[-- Attachment #2: Type: text/html, Size: 4302 bytes --]

Re: ANN: pandoc 2.11

[John MacFarlane]

Pranesh Prakash <the.solipsist-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> On Sunday, 11 October, 2020 at 10:06:29 pm UTC+5:30 John MacFarlane wrote:
>
>> - Copac, EndNote, ISI, MEDLINE, MODS, and RIS bibliographies 
>> are no longer supported. If you use one of these formats, 
>> you may use the `bibutils` utility to convert to BibLaTeX. 
>> We now support only BibTeX, BibLaTeX, CSL JSON, 
>> and pandoc's YAML/Markdown analogue of CSL JSON. 
>>
>
> Given a choice between those four formats which is preferable?  I'm 
> guessing CSL JSON is the "native" format used by Pandoc, but I wanted to 
> confirm.

It depends on your needs.  CSL JSON has limited support for
formatting.  (Italics, bold, super/subscript, smallcaps.)
If you need formatting that goes beyond that, in particular math,
then you may want to use pandoc YAML/Markdown or BibLaTeX.
(BibLaTeX is strictly better than BibTeX.)

Bottom line:  if you have math in your titles, go for BibLaTeX
or pandoc YAML/Markdown.  If you have a very large bibliography
so that speed becomes an issue, CSL JSON will be by far the
fastest to parse.  Otherwise, personal preference.

> And separately, I tried to convert a CSL JSON file produced using Zotero's 
> internal CSL JSON export format as well as using BetterBibTex's CSL JSON 
> export format.
>
> I got this error:
> $ pandoc coc.bbtcsl.json -s -t markdown
> JSON parse error: Error in $: mempty

Can you upload coc.bbtcsl.json so we can diagnose this?

Re: ANN: pandoc 2.11

[John MacFarlane]

"'Nick Bart' via pandoc-discuss"
<pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> writes:

> As to converting CSL JSON files using pandoc, I noticed this too:
>
> You need to add `-f csljson` to the command line.
>
> @jgm, I think some examples in the documentation need to be fixed to include `-f csljson`.

Yes, sorry -- fixed now.

> Or could pandoc be made to automatically distinguish CSL JSON files from other .json files?

In principle, we could peek into the contents of the json file
and determine this.  But that goes beyond what we do now for any
format (which is just default based on the extension), so I'd
favor requiring -f csljson.

Re: ANN: pandoc 2.11

[John MacFarlane]

John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:

>> I got this error:
>> $ pandoc coc.bbtcsl.json -s -t markdown
>> JSON parse error: Error in $: mempty
>
> Can you upload coc.bbtcsl.json so we can diagnose this?

Never mind -- as Nick Bart points out, the problem is
that you need to specify -f cslsjon.