Update on new citation processing -- please help test!

Update on new citation processing -- please help test!

[John MacFarlane]

I'm getting close to a releasable version of the new citeproc
library.  I encourage those who use pandoc-citeproc to try
downloading the lastest nightly of pandoc and using that instead,
to test how well the new library works.

https://github.com/jgm/pandoc/actions/runs/287527580

It should work the same as before, only instead of doing
'--filter pandoc-citeproc' you'll use '--citeproc'.
You won't need pandoc-citeproc at all with the new pandoc.

I think you'll notice quite a few improvements in citation
processing, both in accuracy and speed.

The new citeproc library also includes an optional executable,
citeproc, which can do citation processing independently of pandoc.
It takes in JSON and returns a result in JSON. Its behavior is
described more fully here:

https://github.com/jgm/citeproc/blob/master/man/citeproc.1.md

It would also be great if people could help test that.
To do that you'll need to compile from source:

    stack install --flag citeproc:executable

John

Re: Update on new citation processing -- please help test!

[Joseph Reagle]

On 10/4/20 11:42 AM, John MacFarlane wrote:
> I think you'll notice quite a few improvements in citation
> processing, both in accuracy and speed.

In terms of speed, it's about 40% slower when run with my big YAML bibliography and typical parameters.

```
─➤  time /usr/local/bin/pandoc -f markdown+mmd_title_block+yaml_metadata_block+smart+implicit_header_references+superscript+subscript+tex_math_dollars+autolink_bare_uris -w html -s --tab-stop 4 --email-obfuscation=references -c https://reagle.org/joseph/2003/papers.css -o /Users/reagle/joseph/2016/04/min-eg/test.html --mathjax --csl=chicago-fullnote-bibliography.csl --bibliography=/Users/reagle/joseph/readings.yaml --filter pandoc-citeproc /Users/reagle/joseph/2016/04/min-eg/test.md

35.14s user 4.59s system 97% cpu 40.592 total

#######################

─➤  time ~/tmp/p/pandoc -f markdown+mmd_title_block+yaml_metadata_block+smart+implicit_header_references+superscript+subscript+tex_math_dollars+autolink_bare_uris -w html -s --tab-stop 4 --email-obfuscation=references -c https://reagle.org/joseph/2003/papers.css -o /Users/reagle/joseph/2016/04/min-eg/test.html --mathjax --csl=chicago-fullnote-bibliography.csl --bibliography=/Users/reagle/joseph/readings.yaml --citeproc /Users/reagle/joseph/2016/04/min-eg/test.md

50.40s user 5.46s system 96% cpu 57.893 total

```

Also, something is different with respect to locating the CSL file.

```
╭─reagle@hom ~/joseph/2003/cv
╰─➤  ~/tmp/p/pandoc -f markdown+mmd_title_block+yaml_metadata_block+smart+implicit_header_references+superscript+subscript+tex_math_dollars+autolink_bare_uris -w html -s --tab-stop 4 --email-obfuscation=references -c https://reagle.org/joseph/2003/papers.css -o /Users/reagle/joseph/2003/cv/cv-new.html --mathjax --csl=apa-cv.csl --bibliography=/Users/reagle/joseph/2003/cv/cv.yaml --citeproc /Users/reagle/joseph/2003/cv/cv-2.md
File apa-cv.csl not found in resource path

╭─reagle@hom ~/joseph/2003/cv
╰─➤  ~/tmp/p/pandoc -f markdown+mmd_title_block+yaml_metadata_block+smart+implicit_header_references+superscript+subscript+tex_math_dollars+autolink_bare_uris -w html -s --tab-stop 4 --email-obfuscation=references -c https://reagle.org/joseph/2003/papers.css -o /Users/reagle/joseph/2003/cv/cv-new.html --mathjax --csl=~/.csl/apa-cv.csl --bibliography=/Users/reagle/joseph/2003/cv/cv.yaml --citeproc /Users/reagle/joseph/2003/cv/cv-2.md
File ~/.csl/apa-cv.csl not found in resource path

╭─reagle@hom ~/joseph/2003/cv
╰─➤  head ~/.csl/apa-cv.csl                                                                  99 ↵
<?xml version="1.0" encoding="utf-8"?>
<style xmlns="http://purl.org/net/xbiblio/csl" class="in-text" version="1.0" demote-non-dropping-particle="never" page-range-format="expanded">
  <info>
    <title>American Psychological Association 7th edition (curriculum vitae, sorted by descending date)</title>
    <title-short>APA (CV)</title-short>
    <id>http://www.zotero.org/styles/apa-cv</id>
    <link href="http://www.zotero.org/styles/apa-cv" rel="self"/>
    <link href="http://www.zotero.org/styles/apa" rel="template"/>
    <link href="https://apastyle.apa.org/style-grammar-guidelines/references/examples" rel="documentation"/>
    <author>
```

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/40579bf4-8842-f778-98f7-7769cb64a470%40reagle.org.

Re: Update on new citation processing -- please help test!

[John MacFarlane]

Joseph Reagle <joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org> writes:

> On 10/4/20 11:42 AM, John MacFarlane wrote:
>> I think you'll notice quite a few improvements in citation
>> processing, both in accuracy and speed.
>
> In terms of speed, it's about 40% slower when run with my big YAML bibliography and typical parameters.

Hm, what version of pandoc are you running normally? I know that
our shift a little while back from the (C library) yaml to the
(pure Haskell) HsYAML for YAML metadata parsing had a big cost
for large bibliographies.  See

https://github.com/jgm/pandoc/issues/6084

My guess is that the slowdown you're seeing is from pandoc's YAML
metadata parsing (not part of the new library) and not the
library itself.  You could test this hyphothesis by trying both
versions with a CSL JSON bibliography.  (Conveniently, you can
use pandoc to convert the YAML to csljson.)

> Also, something is different with respect to locating the CSL file.

Yes. I should have mentioned this.  It now looks the csl
subdirectory of your pandoc user data directory (~/.pandoc or
~/.local/share/pandoc) instead of in ~/.csl.  You can just
symlink one to the other for now, so both versions will work.

    ln -s ~/.csl ~/.pandoc/csl

Re: Update on new citation processing -- please help test!

[John MacFarlane]

Joseph Reagle <joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org> writes:

> In terms of speed, it's about 40% slower when run with my big YAML bibliography and typical parameters.

One more thing to note:  the nightly isn't compiled with
optimizations, so it should be expected to parse markdown
slower than the released version. This may also be what's
going on.

Re: Update on new citation processing -- please help test!

[John MacFarlane]

> One more thing to note:  the nightly isn't compiled with
> optimizations, so it should be expected to parse markdown
> slower than the released version. This may also be what's
> going on.

Sorry, this is false. The nightly *is* compiled with
optimizations after all.

Re: Update on new citation processing -- please help test!

[Joseph Reagle]

On 10/5/20 1:05 PM, John MacFarlane wrote:
> Hm, what version of pandoc are you running normally? 

╰─➤  /usr/local/bin/pandoc --version
pandoc 2.10.1
Compiled with pandoc-types 1.21, texmath 0.12.0.2, skylighting 0.8.5

╰─➤  ~/tmp/p/pandoc --version
pandoc 2.11
Compiled with pandoc-types 1.22, texmath 0.12.0.3, skylighting 0.10.0.2
```

--------------

> Conveniently, you can use pandoc to convert the YAML to csljson

Do you mean `pandoc-citeproc` to convert between the two? I can do that -- and it takes forever -- but `pandoc` 2.11 doesn't seem to know yaml as in input format.

--------------

I had a json version from the last time we discussed YAML parsing slowness and 2.10.1 took about 1 sec and 2.11 took about half a sec. At some point, I'll have to change my tool to follow pandoc's preference for JSON.

--------------

> It now looks the csl subdirectory of your pandoc user data directory (~/.pandoc or ~/.local/share/pandoc) instead of in ~/.csl.

Okay. BTW: How do I specify a specific CSL file, perhaps in the local working directory? (I used to do this.) Does this recurse if their are subdirectories? (I don't think so, but there is a "dependent" subdirectory [1].)

[1]: https://github.com/citation-style-language/styles/tree/master/dependent


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/7a090173-6b56-7dc1-05f1-5210d17391d9%40reagle.org.

Re: Update on new citation processing -- please help test!

[John MacFarlane]

Joseph Reagle <joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org> writes:

> On 10/5/20 1:05 PM, John MacFarlane wrote:
>> Hm, what version of pandoc are you running normally?
>
> ╰─➤  /usr/local/bin/pandoc --version
> pandoc 2.10.1
> Compiled with pandoc-types 1.21, texmath 0.12.0.2, skylighting 0.8.5
>
> ╰─➤  ~/tmp/p/pandoc --version
> pandoc 2.11
> Compiled with pandoc-types 1.22, texmath 0.12.0.3, skylighting 0.10.0.2
> ```

Hm, I'm not sure how to explain why YAML metadata parsing is
slower; there aren't any changes I can think of that would
explain that.  Your other test shows that this isn't due to
citation processing.  But I'll look to see whether
pandoc-citeproc was doing some filtering of the references
that pandoc isn't doing.

>> Conveniently, you can use pandoc to convert the YAML to csljson
>
> Do you mean `pandoc-citeproc` to convert between the two? I can do that -- and it takes forever -- but `pandoc` 2.11 doesn't seem to know yaml as in input format.

Assuming your YAML file starts with --- and ends with ... (and
thus is also a valid pandoc markdown  file), you should be able
to do

pandoc -f markdown -t csljosn biblio.yaml

If it doesn't have the delimiters around it, you can do

echo "" | pandoc --metadata-file=biblio.yaml -f markdown -t csljson


> Okay. BTW: How do I specify a specific CSL file, perhaps in the local working directory? (I used to do this.) Does this recurse if their are subdirectories? (I don't think so, but there is a "dependent" subdirectory [1].)
>
> [1]: https://github.com/citation-style-language/styles/tree/master/dependent

In the working directory, just as before: --csl mystyle.csl

It will not recurse into subdirectories (but you can specify
--resource-path and include the subdirectory).

It does not recurse.
If the style is dependent, pandoc will do an HTTP request to
fetch the independent parent.

I should change the searching of the csl subdirectory of
the data dir so that it automatically checks dependent/.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m2a6x0qnou.fsf%40MacBook-Pro.hsd1.ca.comcast.net.

Re: Update on new citation processing -- please help test!

[John MacFarlane]

John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:

> Hm, I'm not sure how to explain why YAML metadata parsing is
> slower...

Here's a guess. pandoc-citeproc uses the same YAML parsing
library.  But it handles it differently.  First, it parses this
as an Aeson Value, then it applies the selectEntries filter
to remove entries not cited in the document, and only the
remaining entries are parsed as Markdown.

Pandoc, by contrast, parses all the entries in the YAML file as
Markdown  -- for pandoc, this is just a regular YAML metadata
file -- and then applies the selectEntries filter.  As a result,
it may be doing a lot of unnecessary Markdown parsing.  (In
principle, because of Haskell's laziness, it should avoid
doing unnecessary work, but maybe we're doing something to
force these values?)

Re: Update on new citation processing -- please help test!

[John MacFarlane]

OK, I've just pushed a commit that sped things up considerably:
from 36s to 16s with a 3 megabyte YAML bibliography.  Try
again with tonight's nightly after it is built!

Re: Update on new citation processing -- please help test!

[John MacFarlane]

Sorry. Bad test.  It doesn't actually speed things up.

John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:

> OK, I've just pushed a commit that sped things up considerably:
> from 36s to 16s with a 3 megabyte YAML bibliography.  Try
> again with tonight's nightly after it is built!

Re: Update on new citation processing -- please help test!

[John MacFarlane]

OK, now I think I've got it.  With the latest commit
(6a32ea71ea9af8613b8585e874fef8d5f29665fa), pandoc --citeproc
should handle YAML bibliographies with the same speed as
pandoc-citeproc.


> Sorry. Bad test.  It doesn't actually speed things up.
>
> John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:
>
>> OK, I've just pushed a commit that sped things up considerably:
>> from 36s to 16s with a 3 megabyte YAML bibliography.  Try
>> again with tonight's nightly after it is built!

Re: Update on new citation processing -- please help test!

[John MacFarlane]

Argh.  Speed is there but it's not working.  More later!

John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:

> OK, now I think I've got it.  With the latest commit
> (6a32ea71ea9af8613b8585e874fef8d5f29665fa), pandoc --citeproc
> should handle YAML bibliographies with the same speed as
> pandoc-citeproc.
>
>
>> Sorry. Bad test.  It doesn't actually speed things up.
>>
>> John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:
>>
>>> OK, I've just pushed a commit that sped things up considerably:
>>> from 36s to 16s with a 3 megabyte YAML bibliography.  Try
>>> again with tonight's nightly after it is built!

Re: Update on new citation processing -- please help test!

[Joseph Reagle]

On 10/5/20 7:04 PM, John MacFarlane wrote:
>> BTW: How do I specify a specific CSL file, perhaps in the local working directory? 
>> ...
> In the working directory, just as before: 
> --csl mystyle.csl 

What about an arbitrary path, such as `--csl=~/tmp/mystyle.csl`? (Perhaps it never did this?)

Re: Update on new citation processing -- please help test!

[Joseph Reagle]

On 10/5/20 7:04 PM, John MacFarlane wrote:
> echo "" | pandoc --metadata-file=biblio.yaml -f markdown -t csljson

BTW: This on yesterday's pandoc is faster (53s) than pandoc-citeproc (97s) -- though too slow for my daily use.

```
─➤  time echo "" | ~/tmp/p/pandoc --metadata-file=readings.yaml -f markdown -t csljson -o readings.json
echo ""  0.00s user 0.00s system 36% cpu 0.001 total
~/tmp/p/pandoc --metadata-file=readings.yaml -f markdown -t csljson -o   50.26s user 2.65s system 99% cpu 53.268 total
```

vs.

```
╰─➤  time /usr/local/bin/pandoc-citeproc -j readings.yaml > readings-citeproc.json
/usr/local/bin/pandoc-citeproc -j readings.yaml > readings-citeproc.json  97.25s user 11.36s system 97% cpu 1:51.62 total
```

Also, readings.json (new) and readings-citeproc.json (old), though substantively the same, are not identical. pandoc includes a "custom2" field that citeproc dropped, and it does quotes-within-quotes differently.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/579252e9-aaab-33e4-e1f9-6470008a8bb8%40reagle.org.

Re: Update on new citation processing -- please help test!

[Albert Krewinkel]

Joseph Reagle writes:

> On 10/5/20 7:04 PM, John MacFarlane wrote:
>>> BTW: How do I specify a specific CSL file, perhaps in the local working directory?
>>> ...
>> In the working directory, just as before:
>> --csl mystyle.csl
>
> What about an arbitrary path, such as `--csl=~/tmp/mystyle.csl`? (Perhaps it never did this?)

Try dropping the equal sign: expansion of the tilde to the user's home
directory is done by the shell, not pandoc, but only if the tilde is the
first character. So with `--csl=~/tmp/mystyle.csl` pandoc will look for
`~/tmp/mystyle.csl` with a literal tilde character, while `--csl
~/tmp/mystyle.csl` should make it use `$HOME/tmp/mystyle.csl`.

--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Update on new citation processing -- please help test!

[John MacFarlane]

Joseph Reagle <joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org> writes:

> On 10/5/20 7:04 PM, John MacFarlane wrote:
>> echo "" | pandoc --metadata-file=biblio.yaml -f markdown -t csljson
>
> BTW: This on yesterday's pandoc is faster (53s) than pandoc-citeproc (97s) -- though too slow for my daily use.

Good to hear.  The underlying issue with YAML is due to
performance issues with the HsYAML library, which would
need to be fixed upstream.

> Also, readings.json (new) and readings-citeproc.json (old), though substantively the same, are not identical. pandoc includes a "custom2" field that citeproc dropped, and it does quotes-within-quotes differently.

I'd expect this.  Including unofficial fields is one of the
changes.  Are the changes in quotes-within-quotes good or bad?

Re: Update on new citation processing -- please help test!

[Joseph Reagle]

On 10/6/20 1:11 PM, John MacFarlane wrote:
> Are the changes in quotes-within-quotes good or bad?

I think neutral so far. A little quote-nesting bug I've never bothered with seems to persist. (I wasn't sure if it was with my YAML, with pandoc, or with citeproc/CSL.)

In the title fields below you can see the YAML representation of a quote (two single-quotes) within a quote (double-quote).

```yaml
- id: Reagle2013fsf
  type: article-journal
  author:
  - family: "Reagle"
    given: "Joseph"
  custom2: "pubs.mm"
  issued:
    year: 2013
    month: 01
  container-title: "First Monday"
  note: "acceptance rate 15%"
  issue: "1"
  title-short: "''Free as in sexist?''"
  title: "''Free as in sexist?'': Free culture and the gender gap"
  URL: "https://reagle.org/joseph/2012/fas/free-as-in-sexist.html"
  volume: "18"
```

Both pandoc-citeproc and the new pandoc do confuse the nesting when converting to smart quotes, which manifests in the outputted HTML as well.

```
    "title": "“’Free as in sexist?”’: Free culture and the gender gap",
```

But maybe I'm doing quotes within quotes wrong in YAML?

-----------

I do see a bug in the new pandoc with respect to the encoding of ampersands in the resulting HTML -- both version generate the same JSON.

```yaml
  container-title: "New Media & Society"
```

```json
    "container-title": "New Media & Society",
```

```html
<em>New Media &amp;#38; Society</em>
```

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/1f98a4af-d866-d411-a716-e79244c7ace0%40reagle.org.

Re: Update on new citation processing -- please help test!

[John MacFarlane]

Joseph Reagle <joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org> writes:

> Both pandoc-citeproc and the new pandoc do confuse the nesting when converting to smart quotes, which manifests in the outputted HTML as well.
>
> ```
>     "title": "“’Free as in sexist?”’: Free culture and the gender gap",
> ```

Sorry, I wasn't really clear about the problem.  You provided a
YAML reference, but how did you generate the above line from
that?

> I do see a bug in the new pandoc with respect to the encoding of ampersands in the resulting HTML -- both version generate the same JSON.
>
> ```yaml
>   container-title: "New Media & Society"
> ```
>
> ```json
>     "container-title": "New Media &#38; Society",
> ```
>
> ```html
> <em>New Media &amp;#38; Society</em>
> ```

OK, I guess I wrongly assumed that & would need to be
encoded this way in CSL JSON (since HTML-like tags are
allowed).  I need to figure that out.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m2pn5uoe8z.fsf%40MacBook-Pro.hsd1.ca.comcast.net.

Re: Update on new citation processing -- please help test!

[John MacFarlane]

John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:
>> I do see a bug in the new pandoc with respect to the encoding of ampersands in the resulting HTML -- both version generate the same JSON.
>>
>> ```yaml
>>   container-title: "New Media & Society"
>> ```
>>
>> ```json
>>     "container-title": "New Media &#38; Society",
>> ```
>>
>> ```html
>> <em>New Media &amp;#38; Society</em>
>> ```

I think this should be fixed in tonight's nightly.

Re: Update on new citation processing -- please help test!

[William Lupton]

[-- Attachment #1: Type: text/plain, Size: 943 bytes --]

With regard to this:

On Sun, 4 Oct 2020 at 16:42, John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:

> I'm getting close to a releasable version of the new citeproc library.
> [...]
>
> It should work the same as before, only instead of doing '--filter
> pandoc-citeproc' you'll use '--citeproc'. You won't need pandoc-citeproc at
> all with the new pandoc.
>

I have a lua filter that runs _after_ pandoc-citeproc (it does some
hyperlink processing). Will this be possible with the new library?

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAEe_xxgUSgtOZLkKcwLGKNvTfKcqUMkZg-8t%3DQjEpzWq%3D95esA%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 1597 bytes --]

Re: Update on new citation processing -- please help test!

['Denis Maier' via pandoc-discuss]

[-- Attachment #1: Type: text/plain, Size: 1158 bytes --]



Am 07.10.2020 um 15:55 schrieb William Lupton:
> With regard to this:
>
> On Sun, 4 Oct 2020 at 16:42, John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org 
> <mailto:jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org>> wrote:
>
>     I'm getting close to a releasable version of the new citeproc
>     library.  [...]
>
>     It should work the same as before, only instead of doing '--filter
>     pandoc-citeproc' you'll use '--citeproc'. You won't need
>     pandoc-citeproc at all with the new pandoc.
>
>
> I have a lua filter that runs _after_ pandoc-citeproc (it does some 
> hyperlink processing). Will this be possible with the new library?

Yes, just use the `--citeproc` switch before your filter.
Best,
Denis

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/5322f662-bd0e-41d2-90a4-eca07ddf0271%40mailbox.org.

[-- Attachment #2: Type: text/html, Size: 2467 bytes --]

Re: Update on new citation processing -- please help test!

[William Lupton]

[-- Attachment #1: Type: text/plain, Size: 1215 bytes --]

Great! Tx, W.

On Wed, 7 Oct 2020 at 14:57, Denis Maier <denismaier-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> wrote:

>
>
> Am 07.10.2020 um 15:55 schrieb William Lupton:
>
> With regard to this:
>
> On Sun, 4 Oct 2020 at 16:42, John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
>
>> I'm getting close to a releasable version of the new citeproc library.
>> [...]
>>
>> It should work the same as before, only instead of doing '--filter
>> pandoc-citeproc' you'll use '--citeproc'. You won't need pandoc-citeproc at
>> all with the new pandoc.
>>
>
> I have a lua filter that runs _after_ pandoc-citeproc (it does some
> hyperlink processing). Will this be possible with the new library?
>
>
> Yes, just use the `--citeproc` switch before your filter.
> Best,
> Denis
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAEe_xxjoDgjkUAnBCt4kEcnz3NSpcxt_jkn5Z7%3D6rJ%3DOksA_PA%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 2632 bytes --]

Re: Update on new citation processing -- please help test!

[Joseph Reagle]

[-- Attachment #1: Type: text/plain, Size: 1414 bytes --]

On 10/7/20 12:23 AM, John MacFarlane wrote:
> Sorry, I wasn't really clear about the problem.  You provided a
> YAML reference, but how did you generate the above line from
> that?

Here's a cleaner example:

https://reagle.org/joseph/2016/04/min-eg/test-nested-quotes.md

to:

https://reagle.org/joseph/2016/04/min-eg/test-nested-quotes.html

via:

```
 ~/tmp/p/pandoc -f markdown+mmd_title_block+yaml_metadata_block+smart -w html -s --tab-stop 4 --email-obfuscation=references -c https://reagle.org/joseph/2003/papers.css -o test-nested-quotes.html --mathjax --csl=chicago-fullnote-bibliography.csl --citeproc test-nested-quotes.md
```

(The YAML is generated via my python script from a Freeplane mindmap.)

BTW: There are other differences, but I don't *think* they are substantive. I can share my HTML CV via 2.10.1 and 2.11 (or nightly) and the diffs might be worthwhile looking at. (I was reminded of the nested quote issue by the placement of the span in the attached diff.)

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/eb0c24b5-c3cb-1fb9-7213-95fda828684b%40reagle.org.

[-- Attachment #2: Screen Shot 2020-10-07 at 10.02.49 AM.png --]
[-- Type: image/png, Size: 25613 bytes --]

Re: Update on new citation processing -- please help test!

[John MacFarlane]

You have

  title: "''Free as in sexist?'': Free culture and the gender gap"

I think you should only have single ':

  title: "'Free as in sexist?': Free culture and the gender gap"

Or did you really intend to have single quotes within single
quotes?

Note that in YAML single quoted strings, you need to escape
' as ''. But not in double quoted strings.

Re: Update on new citation processing -- please help test!

[Joseph Reagle]

On 10/7/20 7:08 PM, John MacFarlane wrote:
> Note that in YAML single quoted strings, you need to escape
> ' as ''. But not in double quoted strings.

Okay, a bug on my part. How would you represent the following title in YAML?

"A quote with 'square quotes'": Can't believe this ugly title


I don't think the following is right...

title: "'A quote with 'square quotes'': Can't believe this ugly title"

Re: Update on new citation processing -- please help test!

[John MacFarlane]

Pandoc's smart quote parser doesn't work too well on this
case, so you might just use curly quotes.

Another alternative is to use the commonmark format
(commonmark_x gives you a lot of pandoc extensions, though
not all of them).  The commonmark smart quote parser gets
this case right.

Joseph Reagle <joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org> writes:

> On 10/7/20 7:08 PM, John MacFarlane wrote:
>> Note that in YAML single quoted strings, you need to escape
>> ' as ''. But not in double quoted strings.
>
> Okay, a bug on my part. How would you represent the following title in YAML?
>
> "A quote with 'square quotes'": Can't believe this ugly title
>
>
> I don't think the following is right...
>
> title: "'A quote with 'square quotes'': Can't believe this ugly title"
>
>
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/8901e820-71d6-28b1-a1ca-65b73022e706%40reagle.org.

Re: Update on new citation processing -- please help test!

[Joseph Reagle]

On 10/8/20 2:00 PM, John MacFarlane wrote:
> Another alternative is to use the commonmark format
> (commonmark_x gives you a lot of pandoc extensions, though
> not all of them).  The commonmark smart quote parser gets
> this case right.

Any chance you'll be adopting that for markdown? I ask because it looks
like commonmark_x doesn't support `mmd_title_block` or
`yaml_metadata_block` -- both of which I've used and the latter is what
I'm trying to move toward uniformly.

I've only used straight quotes in my data and relied on the seeming
magic of pandoc to get it right. (I can't envision going back and
adding smart quotes to every thing or bettering pandoc's parsing,
though I'll have to look a bit more closely at my code to see if I can
improve anything.)

Re: Update on new citation processing -- please help test!

[John MacFarlane]

Joseph Reagle <joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org> writes:

> On 10/8/20 2:00 PM, John MacFarlane wrote:
>> Another alternative is to use the commonmark format
>> (commonmark_x gives you a lot of pandoc extensions, though
>> not all of them).  The commonmark smart quote parser gets
>> this case right.
>
> Any chance you'll be adopting that for markdown? I ask because it looks
> like commonmark_x doesn't support `mmd_title_block` or
> `yaml_metadata_block` -- both of which I've used and the latter is what
> I'm trying to move toward uniformly.

The long-term plan is to move to this, but I need to add support
for more extensions.  (I'd want to support YAML metadata blocks
for sure.)

> I've only used straight quotes in my data and relied on the seeming
> magic of pandoc to get it right. (I can't envision going back and
> adding smart quotes to every thing or bettering pandoc's parsing,
> though I'll have to look a bit more closely at my code to see if I can
> improve anything.)

You probably only need it in a few odd cases.