WIP: better citation processing

WIP: better citation processing

[John MacFarlane]

I've been working this summer on a replacement to pandoc-citeproc.

Part of this is a new library, citeproc, which is
not specific to pandoc. This already passes a larger portion
of the CSL test suite than pandoc-citeproc, and it should be
straightforward to improve it further. It will be easier to
maintain than pandoc-citeproc, more accurate, and faster.

I've used this library to create a new filter,
new-pandoc-citeproc, which now passes most of the pandoc-citeproc
test suite but runs around 6 times faster.

I'm inclined, though, not to release this as a new filter,
but instead to depend on the citeproc library and build the
citation processing capabilities into pandoc itself. This will
cut down the binaries we need to distribute from two to one, and
it will simplify things for users, who won't have to worry about
filters. It will also be more performant, as we'll avoid the
overhead of JSON serialization and deserialization.

I'm not quite ready to release any of this code, but I hope
to do so in the next month or two.  This is just a teaser.

The new library is pure Haskell and won't depend on bibutils
(a wrapper around a C library).  That means we'll only support
bibtex/biblatex, pandoc yaml, and CSL JSON as bibliography
formats.  Those who need others will have to convert them using
standalone bibutils.  But support for other formats was never
great, so I don't think this is a big loss.

Re: WIP: better citation processing

[James]

This sounds awesome. Thank you John wish I had the coding skills to help 

Sent from my iPhone

> On 16 Aug 2020, at 23:59, John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
> 
> 
> I've been working this summer on a replacement to pandoc-citeproc.
> 
> Part of this is a new library, citeproc, which is
> not specific to pandoc. This already passes a larger portion
> of the CSL test suite than pandoc-citeproc, and it should be
> straightforward to improve it further. It will be easier to
> maintain than pandoc-citeproc, more accurate, and faster.
> 
> I've used this library to create a new filter,
> new-pandoc-citeproc, which now passes most of the pandoc-citeproc
> test suite but runs around 6 times faster.
> 
> I'm inclined, though, not to release this as a new filter,
> but instead to depend on the citeproc library and build the
> citation processing capabilities into pandoc itself. This will
> cut down the binaries we need to distribute from two to one, and
> it will simplify things for users, who won't have to worry about
> filters. It will also be more performant, as we'll avoid the
> overhead of JSON serialization and deserialization.
> 
> I'm not quite ready to release any of this code, but I hope
> to do so in the next month or two.  This is just a teaser.
> 
> The new library is pure Haskell and won't depend on bibutils
> (a wrapper around a C library).  That means we'll only support
> bibtex/biblatex, pandoc yaml, and CSL JSON as bibliography
> formats.  Those who need others will have to convert them using
> standalone bibutils.  But support for other formats was never
> great, so I don't think this is a big loss.
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m2mu2udwo9.fsf%40johnmacfarlane.net.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/6F6F5A78-7473-473E-927F-46E2382FE979%40gmail.com.

Re: WIP: better citation processing

[Priv.-Doz. Dr. Maria Shinoto]

> Am 17.08.2020 um 23:31 schrieb James <james.londonsw15-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> 
> This sounds awesome. Thank you John wish I had the coding skills to help 

I can only second that!





> Sent from my iPhone
> 
>> On 16 Aug 2020, at 23:59, John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
>> 
>> 
>> I've been working this summer on a replacement to pandoc-citeproc.
>> 
>> Part of this is a new library, citeproc, which is
>> not specific to pandoc. This already passes a larger portion
>> of the CSL test suite than pandoc-citeproc, and it should be
>> straightforward to improve it further. It will be easier to
>> maintain than pandoc-citeproc, more accurate, and faster.
>> 
>> I've used this library to create a new filter,
>> new-pandoc-citeproc, which now passes most of the pandoc-citeproc
>> test suite but runs around 6 times faster.
>> 
>> I'm inclined, though, not to release this as a new filter,
>> but instead to depend on the citeproc library and build the
>> citation processing capabilities into pandoc itself. This will
>> cut down the binaries we need to distribute from two to one, and
>> it will simplify things for users, who won't have to worry about
>> filters. It will also be more performant, as we'll avoid the
>> overhead of JSON serialization and deserialization.
>> 
>> I'm not quite ready to release any of this code, but I hope
>> to do so in the next month or two.  This is just a teaser.
>> 
>> The new library is pure Haskell and won't depend on bibutils
>> (a wrapper around a C library).  That means we'll only support
>> bibtex/biblatex, pandoc yaml, and CSL JSON as bibliography
>> formats.  Those who need others will have to convert them using
>> standalone bibutils.  But support for other formats was never
>> great, so I don't think this is a big loss.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m2mu2udwo9.fsf%40johnmacfarlane.net.
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/6F6F5A78-7473-473E-927F-46E2382FE979%40gmail.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/BAE1E668-397C-4B9B-A3BA-38F5CC010506%40zaw.uni-heidelberg.de.

Re: WIP: better citation processing

[Denis Maier]

Great to hear we'll soon be able to test the new citeproc.

 >  it should be straightforward to improve it further. It will be 
easier to maintain than pandoc-citeproc, more accurate, and faster.

That sounds amazing. As you know CSL 1.0.2 is about to appear soon, and 
1.1 is also already pretty advanced. It would be great if it were easy 
to integrate the new features in your library.

Best,
Denis

Am 17.08.2020 um 00:59 schrieb John MacFarlane:
> I've been working this summer on a replacement to pandoc-citeproc.
>
> Part of this is a new library, citeproc, which is
> not specific to pandoc. This already passes a larger portion
> of the CSL test suite than pandoc-citeproc, and it should be
> straightforward to improve it further. It will be easier to
> maintain than pandoc-citeproc, more accurate, and faster.
>
> I've used this library to create a new filter,
> new-pandoc-citeproc, which now passes most of the pandoc-citeproc
> test suite but runs around 6 times faster.
>
> I'm inclined, though, not to release this as a new filter,
> but instead to depend on the citeproc library and build the
> citation processing capabilities into pandoc itself. This will
> cut down the binaries we need to distribute from two to one, and
> it will simplify things for users, who won't have to worry about
> filters. It will also be more performant, as we'll avoid the
> overhead of JSON serialization and deserialization.
>
> I'm not quite ready to release any of this code, but I hope
> to do so in the next month or two.  This is just a teaser.
>
> The new library is pure Haskell and won't depend on bibutils
> (a wrapper around a C library).  That means we'll only support
> bibtex/biblatex, pandoc yaml, and CSL JSON as bibliography
> formats.  Those who need others will have to convert them using
> standalone bibutils.  But support for other formats was never
> great, so I don't think this is a big loss.
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/360dbe5b-f1ef-17f6-32e6-8c9f85204844%40mailbox.org.

Re: WIP: better citation processing

[jcr]

[-- Attachment #1.1: Type: text/plain, Size: 3264 bytes --]

When I saw the title of this thread, I was hoping that "better citation 
processing" might mean new features, so news of 1.0.2 and 1.1 is quite 
welcome. I thought  CSL development ended years ago with 1.0.1. It seemed 
that they weren't interested in supporting a broad range of domains and in 
non-English citation formats—95% is good enough. Since CSL could never 
format all my citations, I switched to BibLaTeX and stopped reading the 
forums. I still use CSL with pandoc for smaller works that I have to 
deliver in word processor format, manually writing the citations it can't 
handle. A CSL (or at least pandoc) equivalent to BibLaTeX's \mancite would 
be useful.

On Tuesday, August 18, 2020 at 11:58:12 AM UTC+2, Denis Maier wrote:
>
> Great to hear we'll soon be able to test the new citeproc. 
>
>  >  it should be straightforward to improve it further. It will be 
> easier to maintain than pandoc-citeproc, more accurate, and faster. 
>
> That sounds amazing. As you know CSL 1.0.2 is about to appear soon, and 
> 1.1 is also already pretty advanced. It would be great if it were easy 
> to integrate the new features in your library. 
>
> Best, 
> Denis 
>
> Am 17.08.2020 um 00:59 schrieb John MacFarlane: 
> > I've been working this summer on a replacement to pandoc-citeproc. 
> > 
> > Part of this is a new library, citeproc, which is 
> > not specific to pandoc. This already passes a larger portion 
> > of the CSL test suite than pandoc-citeproc, and it should be 
> > straightforward to improve it further. It will be easier to 
> > maintain than pandoc-citeproc, more accurate, and faster. 
> > 
> > I've used this library to create a new filter, 
> > new-pandoc-citeproc, which now passes most of the pandoc-citeproc 
> > test suite but runs around 6 times faster. 
> > 
> > I'm inclined, though, not to release this as a new filter, 
> > but instead to depend on the citeproc library and build the 
> > citation processing capabilities into pandoc itself. This will 
> > cut down the binaries we need to distribute from two to one, and 
> > it will simplify things for users, who won't have to worry about 
> > filters. It will also be more performant, as we'll avoid the 
> > overhead of JSON serialization and deserialization. 
> > 
> > I'm not quite ready to release any of this code, but I hope 
> > to do so in the next month or two.  This is just a teaser. 
> > 
> > The new library is pure Haskell and won't depend on bibutils 
> > (a wrapper around a C library).  That means we'll only support 
> > bibtex/biblatex, pandoc yaml, and CSL JSON as bibliography 
> > formats.  Those who need others will have to convert them using 
> > standalone bibutils.  But support for other formats was never 
> > great, so I don't think this is a big loss. 
> > 
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/46e97135-ea1f-469a-898a-eb24876c0708o%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3899 bytes --]

Re: WIP: better citation processing

[Denis Maier]

[-- Attachment #1: Type: text/plain, Size: 280 bytes --]

For manhcite you might want to check out my filter. https://github.com/denismaier/pandoc-lua-mancite
It's a super simple hack, but it works.

Regarding csl: what feature do you think are missing? If you take a look at the github repos you'll see that a lot has happened recently.

Re: WIP: better citation processing

[FI Apps]

Thank you for the filter. I wouldn’t have expected that to work. I see it’s a bit tricky to use, but it works. I wonder if it will still work when pandoc-citeproc is replaced by the new citeproc library. 

As for missing CSL features, classical citations are near the top of the list, and I’m glad to see they’ve been added to 1.0.2.

Another BibLaTeX feature I use frequently is shorthands, where the first citation introduces an abbreviation, and subsequent citations use the abbreviation instead of the author/editor name and short title. I could probably write a filter to do that, perhaps sticking the mapping from citation keys to abbreviations in the text’s YAML metadata. Knowing that your lua-mancite filter works, by running a shorthand filter in the right place I could get ibid. right as well.

One thing was a Zotero problem more than a CSL problem: it wouldn’t let me use a volume and page for a locator. They told me to use volume:page, which is fine in English, but I also write in another language in which I’ve never seen that notation.

A less common need is for a volume title in addition to the main title. I was told that this would be solved by allowing bibliography items to be hierarchical, but I don’t think that feature ever appeared. I didn’t see it mentioned among the 1.0.2 and 1.1 GitHub issues. I do see that volume-title made it into 1.0.2. Hierarchical items allow for more clever behavior like that of biblatex-chicago, which can include a book in the bibliography if a certain number of chapters in it have been cited.

> On 18 Aug 2020, at 18:59, Denis Maier <maier.de-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> For manhcite you might want to check out my filter. https://github.com/denismaier/pandoc-lua-mancite
> It's a super simple hack, but it works.
> 
> Regarding csl: what feature do you think are missing? If you take a look at the github repos you'll see that a lot has happened recently.
> 
> -- 
> You received this message because you are subscribed to a topic in the Google Groups "pandoc-discuss" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/pandoc-discuss/LaPbYbP8GU4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/1ee49c20-ee12-4984-abdf-e00a9e4414e6o%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/D2ED4685-A7ED-4BB4-B651-D362B2564F62%40gmail.com.

Re: WIP: better citation processing

[Denis Maier]

Am 18.08.2020 um 20:13 schrieb FI Apps:
> Thank you for the filter. I wouldn’t have expected that to work. I see it’s a bit tricky to use, but it works. I wonder if it will still work when pandoc-citeproc is replaced by the new citeproc library.
I don't know. The filter exploits the way how pandoc-citeproc inserts 
rendered citations and non-renderable citations into the AST. That may 
change with the new library, but at this moment only John might knows 
about this.

> As for missing CSL features, classical citations are near the top of the list, and I’m glad to see they’ve been added to 1.0.2.
Yep. A new item type `classic` will be available with 1.0.2.

> Another BibLaTeX feature I use frequently is shorthands, where the first citation introduces an abbreviation, and subsequent citations use the abbreviation instead of the author/editor name and short title. I could probably write a filter to do that, perhaps sticking the mapping from citation keys to abbreviations in the text’s YAML metadata. Knowing that your lua-mancite filter works, by running a shorthand filter in the right place I could get ibid. right as well.
No need to write a filter. The variable `citation-label` will be 
available for this purpose (among others). There's no way to print a 
list of shorthands yet, though, but having multiple bibliographical 
lists is considered to be primarily a question of how calling 
applications and citeprocs interact. So, maybe, with the new citeproc 
library, someone (John?) might want to consider adding such a feature...

> One thing was a Zotero problem more than a CSL problem: it wouldn’t let me use a volume and page for a locator. They told me to use volume:page, which is fine in English, but I also write in another language in which I’ve never seen that notation.
CSL 1.1 will change locators to an array: this should give you 
`\volcite` like features. You will be able to freely combine multiple 
locators. Think of  "Some classical author, Title, Bk. 1, Ch. 4, Sec. 3, 
Para 7, Line 2".
> A less common need is for a volume title in addition to the main title. I was told that this would be solved by allowing bibliography items to be hierarchical, but I don’t think that feature ever appeared. I didn’t see it mentioned among the 1.0.2 and 1.1 GitHub issues. I do see that volume-title made it into 1.0.2. Hierarchical items allow for more clever behavior like that of biblatex-chicago, which can include a book in the bibliography if a certain number of chapters in it have been cited.
Yeah, volume-title is coming. Also, volume-editor and volume-translator. 
Hierarchical items are being discussed as well, and we've added a 
mechanism to specify relations. Currently, the use case are reviews and
Sidenote: This "clever" chicago is actually quite hostile to both 
readers and writers, but yes, it's common in some disciplines, so this 
is being discussed as well. (But it won't make it into 1.1 I think.)
>
>> On 18 Aug 2020, at 18:59, Denis Maier <maier.de-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>
>> For manhcite you might want to check out my filter. https://github.com/denismaier/pandoc-lua-mancite
>> It's a super simple hack, but it works.
>>
>> Regarding csl: what feature do you think are missing? If you take a look at the github repos you'll see that a lot has happened recently.
>>
>> -- 
>> You received this message because you are subscribed to a topic in the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/pandoc-discuss/LaPbYbP8GU4/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/1ee49c20-ee12-4984-abdf-e00a9e4414e6o%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/d59896f4-b360-9ef4-e751-d9dc912ee700%40gmail.com.

Re: WIP: better citation processing

[FI Apps]

[-- Attachment #1: Type: text/plain, Size: 2491 bytes --]


> On 18 Aug 2020, at 23:14, Denis Maier <maier.de-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
…
>> Another BibLaTeX feature I use frequently is shorthands, where the first citation introduces an abbreviation, and subsequent citations use the abbreviation instead of the author/editor name and short title. I could probably write a filter to do that, perhaps sticking the mapping from citation keys to abbreviations in the text’s YAML metadata. Knowing that your lua-mancite filter works, by running a shorthand filter in the right place I could get ibid. right as well.
> No need to write a filter. The variable `citation-label` will be available for this purpose (among others). There's no way to print a list of shorthands yet, though, but having multiple bibliographical lists is considered to be primarily a question of how calling applications and citeprocs interact. So, maybe, with the new citeproc library, someone (John?) might want to consider adding such a feature…

I’ve occasionally seen shorthand lists in articles, but personally I would only put one in a book, so shorthand list generation isn’t something I would use often enough to want it as a core pandoc feature. Instead, what I’d like is access to the bibliography data through a Lua API, e.g., a metadata item on the document that maps citation keys to entries from the combined bibliography files. That could be useful for other things as well, like indexing the persons or works cited.

>> One thing was a Zotero problem more than a CSL problem: it wouldn’t let me use a volume and page for a locator. They told me to use volume:page, which is fine in English, but I also write in another language in which I’ve never seen that notation.
> CSL 1.1 will change locators to an array: this should give you `\volcite` like features. You will be able to freely combine multiple locators. Think of  "Some classical author, Title, Bk. 1, Ch. 4, Sec. 3, Para 7, Line 2".

This would be very nice, but it raises the question of what syntax pandoc might use for specifying this array.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/1F6EE047-E26A-49C8-A9D5-3E6E38048792%40gmail.com.

[-- Attachment #2: Type: text/html, Size: 5275 bytes --]

Re: WIP: better citation processing

[James P. Ascher]

[-- Attachment #1.1: Type: text/plain, Size: 854 bytes --]



> Regarding csl: what feature do you think are missing? If you take a look 
> at the github repos you'll see that a lot has happened recently.
>

CSL isn't fully compatible with either BibTeX, BibLaTex, or even a 
mainstream
English-language style like the Chicago Manual of Style:

https://github.com/citation-style-language/styles/pull/4578#issuecomment-593047568

So, I join everyone in strongest applause for pandoc going its own way.
 

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/1edca1a5-bb73-42b7-a61a-b02d2a1ec5dco%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1391 bytes --]

Re: WIP: better citation processing

[Denis Maier]

[-- Attachment #1: Type: text/plain, Size: 1662 bytes --]

Am 19.08.2020 um 14:47 schrieb James P. Ascher:
>
>     Regarding csl: what feature do you think are missing? If you take
>     a look at the github repos you'll see that a lot has happened
>     recently.
>
>
> CSL isn't fully compatible with either BibTeX, BibLaTex, or even a 
> mainstream
> English-language style like the Chicago Manual of Style:
>
> https://github.com/citation-style-language/styles/pull/4578#issuecomment-593047568
>
> So, I join everyone in strongest applause for pandoc going its own way.

  Sure it isn't compatible with BibTeX and BibLaTeX. It's a different 
metadata format, if that is what you're talking about. And styles aren't 
compatible either, but how should they be?

Regarding CMoS: Chicago is so utterly complex, I doubt there is any 
solution that is fully compatible with it. Biblatex-chicago is obviously 
to closest you might get ...

Anyway, you seem to be misunderstanding what this thread is about: 
John's currently working on a new CSL citeproc. It will replace 
pandoc-citeproc, but it will nevertheless be based on CSL. So it's not 
about pandoc going its own way. Or is someone working on a 
pandoc-specific citation management tool? That would be an interesting 
project.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/5d4b596a-b3ef-744c-bbae-564e490eb598%40gmail.com.

[-- Attachment #2: Type: text/html, Size: 2904 bytes --]

Re: WIP: better citation processing

[James P. Ascher]

>  Sure it isn't compatible with BibTeX and BibLaTeX. It's a different
> metadata format, if that is what you're talking about. And styles
> aren't compatible either, but how should they be?

Oh, I think I'm being confusing.  I'm thinking in terms of crosswalks:
as in, do the metadata categories map from one format to another, or
does some intermediate program rename or drop them along the way?

> Regarding CMoS: Chicago is so utterly complex, I doubt there is any
> solution that is fully compatible with it. Biblatex-chicago is
> obviously to closest you might get ...

Yup.  There are two related problems I was trying to solve (sorta solved):

- getting the metadata I needed into pandoc-citeproc/CSL/etc.  "note" is
  the right field, but objectionable to the CSL folks: I ended up just
  ignoring their use in favor of mine for my project.

- using pandoc-citeproc/CSL/etc. to process that metadata in a way that
  works for me: i.e. customizing and disciplining my own use of CMoS
  and BibLaTeX so that it renders mostly correct for most of my situations.

I agree you'd have to be crazy to claim that you cover CMoS fully--I
don't think anyone has the time or motivation to do every part before
a new version comes out.  Also, CMoS explicitly relies on the writer's
judgment about their purpose to make certain decisions.  Yet, I can make
the parts I care about mostly work for me.  I just fantasize about
a world where everyone else using CMoS can do that too.

Like, why am I maintaining a custom CSL for my writing?  Are other
people having to do that?  Shouldn't there be a way to share?

> Anyway, you seem to be misunderstanding what this thread is about:
> John's currently working on a new CSL citeproc. It will replace 
> pandoc-citeproc, but it will nevertheless be based on CSL. So it's not
> about pandoc going its own way. Or is someone working on a 
> pandoc-specific citation management tool? That would be an
> interesting project.

Drat.  Looking back, I think you're right and my fantasy distracted me.
I just got so darn excited that I jumped the gun.  Thanks for your
gentle pointing out.

Re: WIP: better citation processing

[John MacFarlane]

> That sounds amazing. As you know CSL 1.0.2 is about to appear soon, and 
> 1.1 is also already pretty advanced. It would be great if it were easy 
> to integrate the new features in your library.

Is there a list of changes in CSL 1.0.2 somewhere, so I can see
what will need to be supported when it comes out?

Re: WIP: better citation processing

[Denis Maier]

Am 21.08.2020 um 21:41 schrieb John MacFarlane:
>> That sounds amazing. As you know CSL 1.0.2 is about to appear soon, and
>> 1.1 is also already pretty advanced. It would be great if it were easy
>> to integrate the new features in your library.
> Is there a list of changes in CSL 1.0.2 somewhere, so I can see
> what will need to be supported when it comes out?
>

Yeah, there's a summary of changes here: 
https://docs.google.com/document/d/1wY1cOOamDYYh8VNW7h_uleqieBDGOa_LYsRiVdQy1RI/edit#heading=h.wsywjzy5t4j6

As you can see it's a terms, types and variables release. That should be 
trivial to support. (There were some changes to that list due to 
comments during the comments period so that's not the final list of 
changes.)

1.1 will be more complex, but we plan to document the new features with 
a detailled changelog and new tests in the test-suite.

Re: WIP: better citation processing

[John MacFarlane]

Progress report:

I have put the library at jgm/citeproc on GitHub.  It still
doesn't pass all the CSL tests, and it needs some work, but I
think it's already quite a bit better than pandoc-citeproc.

I've also been working on a version of pandoc that integrates the
library (so that pandoc-citeproc won't be needed).  That work is
happening in the citeproc branch at jgm/pandoc.  When I get a bit
farther, I'll make a binary available so people can try it out.

Citation processing will happen automatically if your document
contains `bibliography` or `references` in the metadata, or if
you use `--bibliography` on the command line.

This is _much_ faster than using pandoc-citeproc.

The pandoc-citeproc executable could also be used to  convert
between bibliographic formats.  I'm planning to integrate that
capability into pandoc, too.  So, you can convert a bibtex
bibliography to CSL json using `pandoc -f bibtex -t csljson`.
You'll also be able to do, e.g., `pandoc -f bibtex -t html` to
get a formatted HTML version of your bibliography.


Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:

> Am 21.08.2020 um 21:41 schrieb John MacFarlane:
>>> That sounds amazing. As you know CSL 1.0.2 is about to appear soon, and
>>> 1.1 is also already pretty advanced. It would be great if it were easy
>>> to integrate the new features in your library.
>> Is there a list of changes in CSL 1.0.2 somewhere, so I can see
>> what will need to be supported when it comes out?
>>
>
> Yeah, there's a summary of changes here: 
> https://docs.google.com/document/d/1wY1cOOamDYYh8VNW7h_uleqieBDGOa_LYsRiVdQy1RI/edit#heading=h.wsywjzy5t4j6
>
> As you can see it's a terms, types and variables release. That should be 
> trivial to support. (There were some changes to that list due to 
> comments during the comments period so that's not the final list of 
> changes.)
>
> 1.1 will be more complex, but we plan to document the new features with 
> a detailled changelog and new tests in the test-suite.

Re: WIP: better citation processing

[Gabriel Nützi]

Awesome, happy to try it out in https://github.com/gabyx/TechnicalMarkdown.

Best Regards.

Am 08.09.2020 um 20:15 schrieb John MacFarlane:
> Progress report:
>
> I have put the library at jgm/citeproc on GitHub.  It still
> doesn't pass all the CSL tests, and it needs some work, but I
> think it's already quite a bit better than pandoc-citeproc.
>
> I've also been working on a version of pandoc that integrates the
> library (so that pandoc-citeproc won't be needed).  That work is
> happening in the citeproc branch at jgm/pandoc.  When I get a bit
> farther, I'll make a binary available so people can try it out.
>
> Citation processing will happen automatically if your document
> contains `bibliography` or `references` in the metadata, or if
> you use `--bibliography` on the command line.
>
> This is _much_ faster than using pandoc-citeproc.
>
> The pandoc-citeproc executable could also be used to  convert
> between bibliographic formats.  I'm planning to integrate that
> capability into pandoc, too.  So, you can convert a bibtex
> bibliography to CSL json using `pandoc -f bibtex -t csljson`.
> You'll also be able to do, e.g., `pandoc -f bibtex -t html` to
> get a formatted HTML version of your bibliography.
>
>
> Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>
>> Am 21.08.2020 um 21:41 schrieb John MacFarlane:
>>>> That sounds amazing. As you know CSL 1.0.2 is about to appear soon, and
>>>> 1.1 is also already pretty advanced. It would be great if it were easy
>>>> to integrate the new features in your library.
>>> Is there a list of changes in CSL 1.0.2 somewhere, so I can see
>>> what will need to be supported when it comes out?
>>>
>> Yeah, there's a summary of changes here:
>> https://docs.google.com/document/d/1wY1cOOamDYYh8VNW7h_uleqieBDGOa_LYsRiVdQy1RI/edit#heading=h.wsywjzy5t4j6
>>
>> As you can see it's a terms, types and variables release. That should be
>> trivial to support. (There were some changes to that list due to
>> comments during the comments period so that's not the final list of
>> changes.)
>>
>> 1.1 will be more complex, but we plan to document the new features with
>> a detailled changelog and new tests in the test-suite.

Re: WIP: better citation processing

[FI Apps]

[-- Attachment #1: Type: text/plain, Size: 3557 bytes --]

Will it be possible to control when citation processing happens relative to
filters? Earlier in this thread, I saw that mancite could be emulated by
running a filter after  pandoc-citeproc. I can imagine other people might
want to generate citations in a filter and have them processed.

On Tue, Sep 8, 2020 at 8:16 PM John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:

>
>
> Progress report:
>
>
>
> I have put the library at jgm/citeproc on GitHub.  It still
>
> doesn't pass all the CSL tests, and it needs some work, but I
>
> think it's already quite a bit better than pandoc-citeproc.
>
>
>
> I've also been working on a version of pandoc that integrates the
>
> library (so that pandoc-citeproc won't be needed).  That work is
>
> happening in the citeproc branch at jgm/pandoc.  When I get a bit
>
> farther, I'll make a binary available so people can try it out.
>
>
>
> Citation processing will happen automatically if your document
>
> contains `bibliography` or `references` in the metadata, or if
>
> you use `--bibliography` on the command line.
>
>
>
> This is _much_ faster than using pandoc-citeproc.
>
>
>
> The pandoc-citeproc executable could also be used to  convert
>
> between bibliographic formats.  I'm planning to integrate that
>
> capability into pandoc, too.  So, you can convert a bibtex
>
> bibliography to CSL json using `pandoc -f bibtex -t csljson`.
>
> You'll also be able to do, e.g., `pandoc -f bibtex -t html` to
>
> get a formatted HTML version of your bibliography.
>
>
>
>
>
> Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>
>
>
> > Am 21.08.2020 um 21:41 schrieb John MacFarlane:
>
> >>> That sounds amazing. As you know CSL 1.0.2 is about to appear soon, and
>
> >>> 1.1 is also already pretty advanced. It would be great if it were easy
>
> >>> to integrate the new features in your library.
>
> >> Is there a list of changes in CSL 1.0.2 somewhere, so I can see
>
> >> what will need to be supported when it comes out?
>
> >>
>
> >
>
> > Yeah, there's a summary of changes here:
>
> >
> https://docs.google.com/document/d/1wY1cOOamDYYh8VNW7h_uleqieBDGOa_LYsRiVdQy1RI/edit#heading=h.wsywjzy5t4j6
>
> >
>
> > As you can see it's a terms, types and variables release. That should be
>
> > trivial to support. (There were some changes to that list due to
>
> > comments during the comments period so that's not the final list of
>
> > changes.)
>
> >
>
> > 1.1 will be more complex, but we plan to document the new features with
>
> > a detailled changelog and new tests in the test-suite.
>
>
>
> --
>
> You received this message because you are subscribed to a topic in the
> Google Groups "pandoc-discuss" group.
>
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/pandoc-discuss/LaPbYbP8GU4/unsubscribe.
>
> To unsubscribe from this group and all its topics, send an email to
> pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/m2bligt9ra.fsf%40MacBook-Pro.hsd1.ca.comcast.net
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAGOSsd%3DXs6u6Xup9YugR-zd6FC_29QFs1asEcARA4m2UPts_vQ%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 5036 bytes --]

Re: WIP: better citation processing

[John MacFarlane]

Currently it's hardcoded that the citation processing occurs
before the filters, as if it's part of the reader.

FI Apps <ffi.appdev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> Will it be possible to control when citation processing happens relative to
> filters? Earlier in this thread, I saw that mancite could be emulated by
> running a filter after  pandoc-citeproc. I can imagine other people might
> want to generate citations in a filter and have them processed.
>
> On Tue, Sep 8, 2020 at 8:16 PM John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
>
>>
>>
>> Progress report:
>>
>>
>>
>> I have put the library at jgm/citeproc on GitHub.  It still
>>
>> doesn't pass all the CSL tests, and it needs some work, but I
>>
>> think it's already quite a bit better than pandoc-citeproc.
>>
>>
>>
>> I've also been working on a version of pandoc that integrates the
>>
>> library (so that pandoc-citeproc won't be needed).  That work is
>>
>> happening in the citeproc branch at jgm/pandoc.  When I get a bit
>>
>> farther, I'll make a binary available so people can try it out.
>>
>>
>>
>> Citation processing will happen automatically if your document
>>
>> contains `bibliography` or `references` in the metadata, or if
>>
>> you use `--bibliography` on the command line.
>>
>>
>>
>> This is _much_ faster than using pandoc-citeproc.
>>
>>
>>
>> The pandoc-citeproc executable could also be used to  convert
>>
>> between bibliographic formats.  I'm planning to integrate that
>>
>> capability into pandoc, too.  So, you can convert a bibtex
>>
>> bibliography to CSL json using `pandoc -f bibtex -t csljson`.
>>
>> You'll also be able to do, e.g., `pandoc -f bibtex -t html` to
>>
>> get a formatted HTML version of your bibliography.
>>
>>
>>
>>
>>
>> Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>>
>>
>>
>> > Am 21.08.2020 um 21:41 schrieb John MacFarlane:
>>
>> >>> That sounds amazing. As you know CSL 1.0.2 is about to appear soon, and
>>
>> >>> 1.1 is also already pretty advanced. It would be great if it were easy
>>
>> >>> to integrate the new features in your library.
>>
>> >> Is there a list of changes in CSL 1.0.2 somewhere, so I can see
>>
>> >> what will need to be supported when it comes out?
>>
>> >>
>>
>> >
>>
>> > Yeah, there's a summary of changes here:
>>
>> >
>> https://docs.google.com/document/d/1wY1cOOamDYYh8VNW7h_uleqieBDGOa_LYsRiVdQy1RI/edit#heading=h.wsywjzy5t4j6
>>
>> >
>>
>> > As you can see it's a terms, types and variables release. That should be
>>
>> > trivial to support. (There were some changes to that list due to
>>
>> > comments during the comments period so that's not the final list of
>>
>> > changes.)
>>
>> >
>>
>> > 1.1 will be more complex, but we plan to document the new features with
>>
>> > a detailled changelog and new tests in the test-suite.
>>
>>
>>
>> --
>>
>> You received this message because you are subscribed to a topic in the
>> Google Groups "pandoc-discuss" group.
>>
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/pandoc-discuss/LaPbYbP8GU4/unsubscribe.
>>
>> To unsubscribe from this group and all its topics, send an email to
>> pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/pandoc-discuss/m2bligt9ra.fsf%40MacBook-Pro.hsd1.ca.comcast.net
>> .
>>
>>
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAGOSsd%3DXs6u6Xup9YugR-zd6FC_29QFs1asEcARA4m2UPts_vQ%40mail.gmail.com.

Re: WIP: better citation processing

[Albert Krewinkel]

John MacFarlane writes:

> Currently it's hardcoded that the citation processing occurs
> before the filters, as if it's part of the reader.

I believe that will break a couple of filters, including
pandoc-crossref. Maybe there could be a `--citeproc` option which acts
as a special filter?

I'll take a look and see if we can expose the function to Lua filters.
There are a couple of filters in pandoc/lua-filters which call out to
pandoc-citeproc and rely on a freely choosable filter order.

--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: WIP: better citation processing

[John MacFarlane]

It would be good to get clear on this.  We could expose
a --citeproc and require it to be used to process citations.
That would also give control of order; it could be treated as
a built-in filter.  So maybe it's the thing to do -- but I did
like the idea of doing it automatically.

Note that the unprocessed citations would still be accessible
to pandoc-crossref after citations wer processed; so the only
drawback I can see is that you'd get spurious warnings about
citations not found.  (Maybe this is serious enough, though,
by itself.)

It should be pretty straightforward to expose this to Lua
filters.  (But again, I'd be curious to hear why calling
manually from Lua filters would be needed.) At heart it's just a
Pandoc -> m Pandoc transformation in PandocMonad.

Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> writes:

> John MacFarlane writes:
>
>> Currently it's hardcoded that the citation processing occurs
>> before the filters, as if it's part of the reader.
>
> I believe that will break a couple of filters, including
> pandoc-crossref. Maybe there could be a `--citeproc` option which acts
> as a special filter?
>
> I'll take a look and see if we can expose the function to Lua filters.
> There are a couple of filters in pandoc/lua-filters which call out to
> pandoc-citeproc and rely on a freely choosable filter order.
>
> --
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/87imckl2tg.fsf%40zeitkraut.de.

Re: WIP: better citation processing

[Albert Krewinkel]

John MacFarlane writes:

> It would be good to get clear on this.  We could expose
> a --citeproc and require it to be used to process citations.
> That would also give control of order; it could be treated as
> a built-in filter.  So maybe it's the thing to do -- but I did
> like the idea of doing it automatically.

I agree, that's much nicer. Would an additional `--no-citeproc`
option make sense? It could suppress the automatic run.

> It should be pretty straightforward to expose this to Lua
> filters.  (But again, I'd be curious to hear why calling
> manually from Lua filters would be needed.) At heart it's just a
> Pandoc -> m Pandoc transformation in PandocMonad.

Indeed, that should be really easy then. :)

Lua filters which call out to pandoc-citeproc, or which rely on a
special filter order:

- cito: allows to add information about the semantic relation of the
  cited work to the current document. Plugs into the citation syntax and
  requires citation IDs to be rewritten before pandoc-citeproc is
  called.

- section-refs: adds "References" for all first level sections; creates
  temporary documents from sections, then calls `pandoc-citeproc` on
  the temporary docs.

- multiple-bibliographies: allows to partition references into
  categories; implementation is similar to section-refs.

--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

proposed --citeproc option

[John MacFarlane]

(Changing the subject to keep better organized.)

I think I'd prefer just requiring --citeproc for citation
processing, rather than making it automatic and providing
a --no-citeproc escape.  It's just conceptually simpler, and
I think people might find it confusing that they can omit
--citeproc, but it's not irrelevant, since it matters where
it is placed.

To allow --citeproc to be inserted at an arbitrary
position in a sequence of filters, we'd
have to add a new constructor for Filter, CiteprocFilter.
Then we could just add the transformation to applyFilters
(in T.P.Filter).

It would also be good to add a deprecation warning
for people who continue to use `--filter pandoc-citeproc`,
suggesting that they use `--citeproc` instead.
(This will still work, for the moment, but I don't plan
to keep maintaining pandoc-citeproc after the new lib
is published.)

Currently the pandoc-citeproc filter is implicitly
used when someone specifies --bibliography on the command
line.  That's a legacy feature going way back, which
I'd planned to phase out.  So maybe this is the time to
do so.

Re: proposed --citeproc option

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 2134 bytes --]

Den fre 11 sep. 2020 20:52John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> skrev:

>
> (Changing the subject to keep better organized.)
>
> I think I'd prefer just requiring --citeproc for citation
> processing,


Yes. The alternative is very confusing.


rather than making it automatic and providing
> a --no-citeproc escape.  It's just conceptually simpler, and
> I think people might find it confusing that they can omit
> --citeproc, but it's not irrelevant, since it matters where
> it is placed.
>
> To allow --citeproc to be inserted at an arbitrary
> position in a sequence of filters, we'd
> have to add a new constructor for Filter, CiteprocFilter.
> Then we could just add the transformation to applyFilters
> (in T.P.Filter).
>
> It would also be good to add a deprecation warning
> for people who continue to use `--filter pandoc-citeproc`,
> suggesting that they use `--citeproc` instead.
> (This will still work, for the moment, but I don't plan
> to keep maintaining pandoc-citeproc after the new lib
> is published.)
>
> Currently the pandoc-citeproc filter is implicitly
> used when someone specifies --bibliography on the command
> line.  That's a legacy feature going way back, which
> I'd planned to phase out.  So maybe this is the time to
> do so.
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/m21rj8jge2.fsf%40MacBook-Pro.hsd1.ca.comcast.net
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhC%3DtUGbaPbQYNKJeNHGdo32HXoX%3DNtLbUJ39RZVehnocQ%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 3379 bytes --]

Re: WIP: better citation processing

[Denis Maier]

Am 11.09.2020 um 18:46 schrieb Albert Krewinkel:
> 
> John MacFarlane writes:
> 
>> It would be good to get clear on this.  We could expose
>> a --citeproc and require it to be used to process citations.
>> That would also give control of order; it could be treated as
>> a built-in filter.  So maybe it's the thing to do -- but I did
>> like the idea of doing it automatically.
> 
> I agree, that's much nicer. Would an additional `--no-citeproc`
> option make sense? It could suppress the automatic run.
> 
>> It should be pretty straightforward to expose this to Lua
>> filters.  (But again, I'd be curious to hear why calling
>> manually from Lua filters would be needed.) At heart it's just a
>> Pandoc -> m Pandoc transformation in PandocMonad.
> 
> Indeed, that should be really easy then. :)
> 
> Lua filters which call out to pandoc-citeproc, or which rely on a
> special filter order:
> 
> - cito: allows to add information about the semantic relation of the
>    cited work to the current document. Plugs into the citation syntax and
>    requires citation IDs to be rewritten before pandoc-citeproc is
>    called.
> 
> - section-refs: adds "References" for all first level sections; creates
>    temporary documents from sections, then calls `pandoc-citeproc` on
>    the temporary docs.
> 
> - multiple-bibliographies: allows to partition references into
>    categories; implementation is similar to section-refs.

Adding a `-citeproc` option would be a good option.
But that aside, I think that these particular filters (or replacements 
thereof) could/should be included in the new citeproc library.

> 
> --
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
> 

Re: WIP: better citation processing

[John MacFarlane]

Yes, we could definitely consider adding support for
these things at a later date.

By-section citations would actually be pretty easy.

Multiple bibliographies is also simple IF it's just
a matter of separating items out of a formatted bibliography
into several separate ones.  But if it's going to affect
things like disambiguation then it would be trickier.

The cito thing seems more special-case, but maybe that too.

Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:

> Am 11.09.2020 um 18:46 schrieb Albert Krewinkel:
>> 
>> John MacFarlane writes:
>> 
>>> It would be good to get clear on this.  We could expose
>>> a --citeproc and require it to be used to process citations.
>>> That would also give control of order; it could be treated as
>>> a built-in filter.  So maybe it's the thing to do -- but I did
>>> like the idea of doing it automatically.
>> 
>> I agree, that's much nicer. Would an additional `--no-citeproc`
>> option make sense? It could suppress the automatic run.
>> 
>>> It should be pretty straightforward to expose this to Lua
>>> filters.  (But again, I'd be curious to hear why calling
>>> manually from Lua filters would be needed.) At heart it's just a
>>> Pandoc -> m Pandoc transformation in PandocMonad.
>> 
>> Indeed, that should be really easy then. :)
>> 
>> Lua filters which call out to pandoc-citeproc, or which rely on a
>> special filter order:
>> 
>> - cito: allows to add information about the semantic relation of the
>>    cited work to the current document. Plugs into the citation syntax and
>>    requires citation IDs to be rewritten before pandoc-citeproc is
>>    called.
>> 
>> - section-refs: adds "References" for all first level sections; creates
>>    temporary documents from sections, then calls `pandoc-citeproc` on
>>    the temporary docs.
>> 
>> - multiple-bibliographies: allows to partition references into
>>    categories; implementation is similar to section-refs.
>
> Adding a `-citeproc` option would be a good option.
> But that aside, I think that these particular filters (or replacements 
> thereof) could/should be included in the new citeproc library.
>
>> 
>> --
>> Albert Krewinkel
>> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
>> 
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/1adfb0c1-d745-a6dc-bf14-f790e3f1280f%40mailbox.org.

Re: WIP: better citation processing

[Denis Maier]

Am 11.09.2020 um 22:58 schrieb John MacFarlane:
> Yes, we could definitely consider adding support for
> these things at a later date.
>
> By-section citations would actually be pretty easy.
Sounds great!

> Multiple bibliographies is also simple IF it's just
> a matter of separating items out of a formatted bibliography
> into several separate ones.  But if it's going to affect
> things like disambiguation then it would be trickier.
As outlined on the github issue, I think a simple solution could be 
allowing multiple bibliographies, each with filters applied that either 
exclude or include specific items based on different criteria.

>
> The cito thing seems more special-case, but maybe that too.
>
> Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>
>> Am 11.09.2020 um 18:46 schrieb Albert Krewinkel:
>>> John MacFarlane writes:
>>>
>>>> It would be good to get clear on this.  We could expose
>>>> a --citeproc and require it to be used to process citations.
>>>> That would also give control of order; it could be treated as
>>>> a built-in filter.  So maybe it's the thing to do -- but I did
>>>> like the idea of doing it automatically.
>>> I agree, that's much nicer. Would an additional `--no-citeproc`
>>> option make sense? It could suppress the automatic run.
>>>
>>>> It should be pretty straightforward to expose this to Lua
>>>> filters.  (But again, I'd be curious to hear why calling
>>>> manually from Lua filters would be needed.) At heart it's just a
>>>> Pandoc -> m Pandoc transformation in PandocMonad.
>>> Indeed, that should be really easy then. :)
>>>
>>> Lua filters which call out to pandoc-citeproc, or which rely on a
>>> special filter order:
>>>
>>> - cito: allows to add information about the semantic relation of the
>>>     cited work to the current document. Plugs into the citation syntax and
>>>     requires citation IDs to be rewritten before pandoc-citeproc is
>>>     called.
>>>
>>> - section-refs: adds "References" for all first level sections; creates
>>>     temporary documents from sections, then calls `pandoc-citeproc` on
>>>     the temporary docs.
>>>
>>> - multiple-bibliographies: allows to partition references into
>>>     categories; implementation is similar to section-refs.
>> Adding a `-citeproc` option would be a good option.
>> But that aside, I think that these particular filters (or replacements
>> thereof) could/should be included in the new citeproc library.
>>
>>> --
>>> Albert Krewinkel
>>> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/1adfb0c1-d745-a6dc-bf14-f790e3f1280f%40mailbox.org.

Re: WIP: better citation processing

[Denis Maier]

Am 11.09.2020 um 23:14 schrieb Denis Maier:
>
>
> Am 11.09.2020 um 22:58 schrieb John MacFarlane:
>> Yes, we could definitely consider adding support for
>> these things at a later date.
>>
>> By-section citations would actually be pretty easy.
> Sounds great!
>
>> Multiple bibliographies is also simple IF it's just
>> a matter of separating items out of a formatted bibliography
>> into several separate ones.  But if it's going to affect
>> things like disambiguation then it would be trickier.
> As outlined on the github issue, I think a simple solution could be 
> allowing multiple bibliographies, each with filters applied that 
> either exclude or include specific items based on different criteria.
>
>>
>> The cito thing seems more special-case, but maybe that too.
>>
>> Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>>
>>> Am 11.09.2020 um 18:46 schrieb Albert Krewinkel:
>>>> John MacFarlane writes:
>>>>
>>>>> It would be good to get clear on this.  We could expose
>>>>> a --citeproc and require it to be used to process citations.
>>>>> That would also give control of order; it could be treated as
>>>>> a built-in filter.  So maybe it's the thing to do -- but I did
>>>>> like the idea of doing it automatically.
>>>> I agree, that's much nicer. Would an additional `--no-citeproc`
>>>> option make sense? It could suppress the automatic run.
>>>>
>>>>> It should be pretty straightforward to expose this to Lua
>>>>> filters.  (But again, I'd be curious to hear why calling
>>>>> manually from Lua filters would be needed.) At heart it's just a
>>>>> Pandoc -> m Pandoc transformation in PandocMonad.
>>>> Indeed, that should be really easy then. :)
>>>>
>>>> Lua filters which call out to pandoc-citeproc, or which rely on a
>>>> special filter order:
>>>>
>>>> - cito: allows to add information about the semantic relation of the
>>>>     cited work to the current document. Plugs into the citation 
>>>> syntax and
>>>>     requires citation IDs to be rewritten before pandoc-citeproc is
>>>>     called.
>>>>
>>>> - section-refs: adds "References" for all first level sections; 
>>>> creates
>>>>     temporary documents from sections, then calls `pandoc-citeproc` on
>>>>     the temporary docs.
>>>>
>>>> - multiple-bibliographies: allows to partition references into
>>>>     categories; implementation is similar to section-refs.
>>> Adding a `-citeproc` option would be a good option.
>>> But that aside, I think that these particular filters (or replacements
>>> thereof) could/should be included in the new citeproc library.
>>>
>>>> -- 
>>>> Albert Krewinkel
>>>> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
>>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "pandoc-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, 
>>> send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/pandoc-discuss/1adfb0c1-d745-a6dc-bf14-f790e3f1280f%40mailbox.org.
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/af98c088-941f-36d1-ac9b-a263fc46a219%40mailbox.org.

Re: WIP: better citation processing

[John MacFarlane]

[-- Attachment #1.1: Type: text/plain, Size: 3877 bytes --]

Progress report: I've added support for CSL display attributes to the LaTeX 
writer in the citeproc branch.   (Still TODO: docx, HTML, others?)
Some examples:

[image: Screen Shot 2020-09-17 at 10.12.43 AM.png][image: Screen Shot 
2020-09-17 at 10.13.22 AM.png]
[image: Screen Shot 2020-09-17 at 10.13.44 AM.png]
On Tuesday, September 15, 2020 at 9:43:57 AM UTC-7 Denis Maier wrote:

>
>
> Am 11.09.2020 um 22:58 schrieb John MacFarlane:
> > Yes, we could definitely consider adding support for
> > these things at a later date.
> >
> > By-section citations would actually be pretty easy.
> Sounds great!
>
> > Multiple bibliographies is also simple IF it's just
> > a matter of separating items out of a formatted bibliography
> > into several separate ones. But if it's going to affect
> > things like disambiguation then it would be trickier.
> As outlined on the github issue, I think a simple solution could be 
> allowing multiple bibliographies, each with filters applied that either 
> exclude or include specific items based on different criteria.
>
> >
> > The cito thing seems more special-case, but maybe that too.
> >
> > Denis Maier <denis.ma...-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
> >
> >> Am 11.09.2020 um 18:46 schrieb Albert Krewinkel:
> >>> John MacFarlane writes:
> >>>
> >>>> It would be good to get clear on this. We could expose
> >>>> a --citeproc and require it to be used to process citations.
> >>>> That would also give control of order; it could be treated as
> >>>> a built-in filter. So maybe it's the thing to do -- but I did
> >>>> like the idea of doing it automatically.
> >>> I agree, that's much nicer. Would an additional `--no-citeproc`
> >>> option make sense? It could suppress the automatic run.
> >>>
> >>>> It should be pretty straightforward to expose this to Lua
> >>>> filters. (But again, I'd be curious to hear why calling
> >>>> manually from Lua filters would be needed.) At heart it's just a
> >>>> Pandoc -> m Pandoc transformation in PandocMonad.
> >>> Indeed, that should be really easy then. :)
> >>>
> >>> Lua filters which call out to pandoc-citeproc, or which rely on a
> >>> special filter order:
> >>>
> >>> - cito: allows to add information about the semantic relation of the
> >>> cited work to the current document. Plugs into the citation syntax and
> >>> requires citation IDs to be rewritten before pandoc-citeproc is
> >>> called.
> >>>
> >>> - section-refs: adds "References" for all first level sections; creates
> >>> temporary documents from sections, then calls `pandoc-citeproc` on
> >>> the temporary docs.
> >>>
> >>> - multiple-bibliographies: allows to partition references into
> >>> categories; implementation is similar to section-refs.
> >> Adding a `-citeproc` option would be a good option.
> >> But that aside, I think that these particular filters (or replacements
> >> thereof) could/should be included in the new citeproc library.
> >>
> >>> --
> >>> Albert Krewinkel
> >>> GPG: 8eed e3e2 e8c5 6f18 81fe e836 388d c0b2 1f63 1124
> >>>
> >> -- 
> >> You received this message because you are subscribed to the Google 
> Groups "pandoc-discuss" group.
> >> To unsubscribe from this group and stop receiving emails from it, send 
> an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> >> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pandoc-discuss/1adfb0c1-d745-a6dc-bf14-f790e3f1280f%40mailbox.org
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/d7317035-fba6-466b-8d6e-699f82ae5445n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 5883 bytes --]

[-- Attachment #2: Screen Shot 2020-09-17 at 10.12.43 AM.png --]
[-- Type: image/png, Size: 79032 bytes --]

[-- Attachment #3: Screen Shot 2020-09-17 at 10.13.44 AM.png --]
[-- Type: image/png, Size: 64411 bytes --]

[-- Attachment #4: Screen Shot 2020-09-17 at 10.13.22 AM.png --]
[-- Type: image/png, Size: 75413 bytes --]

Re: WIP: better citation processing

[Denis Maier]

[-- Attachment #1: Type: text/plain, Size: 5105 bytes --]

Awesome. Thanks. We should also add this to the ConTeXt writer.

Side issue: Why is the Frege example "Über sinn und bedeutung" sentence 
cased?

Best,
Denis


Am 17.09.2020 um 19:18 schrieb John MacFarlane:
> Progress report: I've added support for CSL display attributes to the 
> LaTeX writer in the citeproc branch.   (Still TODO: docx, HTML, others?)
> Some examples:
>
> Screen Shot 2020-09-17 at 10.12.43 AM.pngScreen Shot 2020-09-17 at 
> 10.13.22 AM.png
> Screen Shot 2020-09-17 at 10.13.44 AM.png
> On Tuesday, September 15, 2020 at 9:43:57 AM UTC-7 Denis Maier wrote:
>
>
>
>     Am 11.09.2020 um 22:58 schrieb John MacFarlane:
>     > Yes, we could definitely consider adding support for
>     > these things at a later date.
>     >
>     > By-section citations would actually be pretty easy.
>     Sounds great!
>
>     > Multiple bibliographies is also simple IF it's just
>     > a matter of separating items out of a formatted bibliography
>     > into several separate ones. But if it's going to affect
>     > things like disambiguation then it would be trickier.
>     As outlined on the github issue, I think a simple solution could be
>     allowing multiple bibliographies, each with filters applied that
>     either
>     exclude or include specific items based on different criteria.
>
>     >
>     > The cito thing seems more special-case, but maybe that too.
>     >
>     > Denis Maier <denis.ma...-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>     >
>     >> Am 11.09.2020 um 18:46 schrieb Albert Krewinkel:
>     >>> John MacFarlane writes:
>     >>>
>     >>>> It would be good to get clear on this. We could expose
>     >>>> a --citeproc and require it to be used to process citations.
>     >>>> That would also give control of order; it could be treated as
>     >>>> a built-in filter. So maybe it's the thing to do -- but I did
>     >>>> like the idea of doing it automatically.
>     >>> I agree, that's much nicer. Would an additional `--no-citeproc`
>     >>> option make sense? It could suppress the automatic run.
>     >>>
>     >>>> It should be pretty straightforward to expose this to Lua
>     >>>> filters. (But again, I'd be curious to hear why calling
>     >>>> manually from Lua filters would be needed.) At heart it's just a
>     >>>> Pandoc -> m Pandoc transformation in PandocMonad.
>     >>> Indeed, that should be really easy then. :)
>     >>>
>     >>> Lua filters which call out to pandoc-citeproc, or which rely on a
>     >>> special filter order:
>     >>>
>     >>> - cito: allows to add information about the semantic relation
>     of the
>     >>> cited work to the current document. Plugs into the citation
>     syntax and
>     >>> requires citation IDs to be rewritten before pandoc-citeproc is
>     >>> called.
>     >>>
>     >>> - section-refs: adds "References" for all first level
>     sections; creates
>     >>> temporary documents from sections, then calls
>     `pandoc-citeproc` on
>     >>> the temporary docs.
>     >>>
>     >>> - multiple-bibliographies: allows to partition references into
>     >>> categories; implementation is similar to section-refs.
>     >> Adding a `-citeproc` option would be a good option.
>     >> But that aside, I think that these particular filters (or
>     replacements
>     >> thereof) could/should be included in the new citeproc library.
>     >>
>     >>> --
>     >>> Albert Krewinkel
>     >>> GPG: 8eed e3e2 e8c5 6f18 81fe e836 388d c0b2 1f63 1124
>     >>>
>     >> --
>     >> You received this message because you are subscribed to the
>     Google Groups "pandoc-discuss" group.
>     >> To unsubscribe from this group and stop receiving emails from
>     it, send an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>     >> To view this discussion on the web visit
>     https://groups.google.com/d/msgid/pandoc-discuss/1adfb0c1-d745-a6dc-bf14-f790e3f1280f%40mailbox.org.
>
>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org 
> <mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pandoc-discuss/d7317035-fba6-466b-8d6e-699f82ae5445n%40googlegroups.com 
> <https://groups.google.com/d/msgid/pandoc-discuss/d7317035-fba6-466b-8d6e-699f82ae5445n%40googlegroups.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/a3a6d888-3a82-e29d-d723-7a754c7b964c%40mailbox.org.

[-- Attachment #2: Type: text/html, Size: 9384 bytes --]

Re: WIP: better citation processing

[Denis Maier]

Posting back to the list... Maybe someone else has opinions regarding 
the title case question.


Am 17.09.2020 um 21:18 schrieb Denis Maier:
> Regarding ConTeXt: I'll have a look.
>
> Title-casing: I think adopting biblatex's behavior would be 
> reasonable. I.e., skip the conversion for non-English items. I'd 
> assume those to be entered according to the conventions of their 
> language. Everything else would just be a mistake. I don't think that 
> distinction exists in many languages other than English.
>
>
> Am 17.09.2020 um 19:53 schrieb John MacFarlane:
>> John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:
>>
>>> Question:  if the reference contains `language: de-DE`, should
>>> that prevent titlecase transformations automatically, even if
>>> the locale for the style is en-US?
>> Sorry, this question was confused.  Just for background:
>> in converting bibtex to CSL references, we do a sentence case
>> transformation, since CSL references are assumed to be in
>> sentence case.
>>
>> That transformation lowercases the German nouns unless
>> they're protected with {} in the bibtex.
>>
>> Perhaps the question to ask is whether the sentence-case
>> transformation should be skipped for references that have
>> language = {german} (or other non-English) in their
>> biblatex entries.  My view is that it should not be
>> skipped, because bitex bibliographies may use title case
>> even for foreign titles, but I'm not completely sure.
>>
>> As for adding to ConTeXt, I'll put that on the list too.
>> The LaTeX changes define these macros:
>>
>> ```
>> \newlength{\cslhangindent}
>> \setlength{\cslhangindent}{1.5em}
>> \newlength{\csllabelwidth}
>> \setlength{\csllabelwidth}{3em}
>> \newenvironment{CSLReferences}[3] % #1 hanging-ident, #2 entry sp
>>   {% turn on hanging indent if param 1 is 1
>>    \ifodd #1 
>> \everypar{\setlength{\hangindent}{\cslhangindent}}\ignorespaces\fi
>>    % set line spacing
>>    % set entry spacing
>>    \ifnum #2 > 0
>>    \setlength{\parskip}{#3\baselineskip}
>>    \fi
>>   }%
>>   {}
>> \usepackage{calc} % for \widthof, \maxof
>> \newcommand{\CSLBlock}[1]{#1\hfill\break}
>> \newcommand{\CSLLeftMargin}[1]{\parbox[t]{\maxof{\widthof{#1}}{\csllabelwidth}}{#1}} 
>>
>> \newcommand{\CSLRightInline}[1]{\parbox[t]{\linewidth}{#1}}
>> \newcommand{\CSLIndent}[1]{\hspace{\cslhangindent}#1}
>> ```
>>
>> So it would just be a matter of creating similar macros that
>> work in ConTeXt, and telling the ConTeXt writer to use them.
>> I can do the latter if you do the former.
> I'll have a look!
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/2e636364-deea-3db9-cb4c-ad2625632c4d%40mailbox.org.

Re: WIP: better citation processing

[Denis Maier]

[-- Attachment #1: Type: text/plain, Size: 3428 bytes --]

Am 11.09.2020 um 23:14 schrieb Denis Maier:

Am 11.09.2020 um 22:58 schrieb John MacFarlane:
> Yes, we could definitely consider adding support for
> these things at a later date.
>
> By-section citations would actually be pretty easy.
Sounds great!

> Multiple bibliographies is also simple IF it's just
> a matter of separating items out of a formatted bibliography
> into several separate ones.  But if it's going to affect
> things like disambiguation then it would be trickier.
As outlined on the github issue, I think a simple solution could be 
allowing multiple bibliographies, each with filters applied that either 
exclude or include specific items based on different criteria.

Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>> Am 11.09.2020 um 18:46 schrieb Albert Krewinkel:
>>> John MacFarlane writes:
>>>
>>>> It would be good to get clear on this.  We could expose
>>>> a --citeproc and require it to be used to process citations.
>>>> That would also give control of order; it could be treated as
>>>> a built-in filter.  So maybe it's the thing to do -- but I did
>>>> like the idea of doing it automatically.
>>> I agree, that's much nicer. Would an additional `--no-citeproc`
>>> option make sense? It could suppress the automatic run.
>>>
>>>> It should be pretty straightforward to expose this to Lua
>>>> filters.  (But again, I'd be curious to hear why calling
>>>> manually from Lua filters would be needed.) At heart it's just a
>>>> Pandoc -> m Pandoc transformation in PandocMonad.
>>> Indeed, that should be really easy then. :)
>>>
>>> Lua filters which call out to pandoc-citeproc, or which rely on a
>>> special filter order:
>>>
>>> - cito: allows to add information about the semantic relation of the
>>>     cited work to the current document. Plugs into the citation syntax and
>>>     requires citation IDs to be rewritten before pandoc-citeproc is
>>>     called.
>>>
>>> - section-refs: adds "References" for all first level sections; creates
>>>     temporary documents from sections, then calls `pandoc-citeproc` on
>>>     the temporary docs.
>>>
>>> - multiple-bibliographies: allows to partition references into
>>>     categories; implementation is similar to section-refs.
>> Adding a `-citeproc` option would be a good option.
>> But that aside, I think that these particular filters (or replacements
>> thereof) could/should be included in the new citeproc library.
>>
>>> --
>>> Albert Krewinkel
>>> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/1adfb0c1-d745-a6dc-bf14-f790e3f1280f%40mailbox.org.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/a3c8f8e1-c695-2cf7-ef74-7a6622debef8%40gmail.com.

[-- Attachment #2: Type: text/html, Size: 5376 bytes --]

Re: WIP: better citation processing

[Denis Maier]

[-- Attachment #1.1: Type: text/plain, Size: 1170 bytes --]

Sorry for crossposting; my email-setup seems to be messed up... So posting 
again via the web interface.


Am 11.09.2020 um 22:58 schrieb John MacFarlane: 
> Yes, we could definitely consider adding support for 
> these things at a later date. 

By-section citations would actually be pretty easy. 
Sounds great! 

> Multiple bibliographies is also simple IF it's just 
> a matter of separating items out of a formatted bibliography 
> into several separate ones.  But if it's going to affect 
> things like disambiguation then it would be trickier. 

As outlined on the github issue, I think a simple solution could be 
allowing multiple bibliographies, each with filters applied that either 
exclude or include specific items based on different criteria. 


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/96f69491-314f-410e-8f8a-dd4980e42224n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1749 bytes --]

Re: WIP: better citation processing

['Nick Bart' via pandoc-discuss]

[-- Attachment #1: Type: text/plain, Size: 4836 bytes --]

> I can imagine other people might want to generate citations in a filter and have them processed.

For example, in a Zotero -> zotxt -> pandoc workflow it is essential to be able to call the pandoc-zotxt.lua filter _before_ (pandoc-)citeproc (see https://github.com/egh/zotxt/).

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, September 11, 2020 3:05 PM, FI Apps <ffi.appdev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Will it be possible to control when citation processing happens relative to filters? Earlier in this thread, I saw that mancite could be emulated by running a filter after pandoc-citeproc. I can imagine other people might want to generate citations in a filter and have them processed.
>
> On Tue, Sep 8, 2020 at 8:16 PM John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> wrote:
>
>> Progress report:
>>
>> I have put the library at jgm/citeproc on GitHub. It still
>>
>> doesn't pass all the CSL tests, and it needs some work, but I
>>
>> think it's already quite a bit better than pandoc-citeproc.
>>
>> I've also been working on a version of pandoc that integrates the
>>
>> library (so that pandoc-citeproc won't be needed). That work is
>>
>> happening in the citeproc branch at jgm/pandoc. When I get a bit
>>
>> farther, I'll make a binary available so people can try it out.
>>
>> Citation processing will happen automatically if your document
>>
>> contains `bibliography` or `references` in the metadata, or if
>>
>> you use `--bibliography` on the command line.
>>
>> This is _much_ faster than using pandoc-citeproc.
>>
>> The pandoc-citeproc executable could also be used to convert
>>
>> between bibliographic formats. I'm planning to integrate that
>>
>> capability into pandoc, too. So, you can convert a bibtex
>>
>> bibliography to CSL json using `pandoc -f bibtex -t csljson`.
>>
>> You'll also be able to do, e.g., `pandoc -f bibtex -t html` to
>>
>> get a formatted HTML version of your bibliography.
>>
>> Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>>
>>> Am 21.08.2020 um 21:41 schrieb John MacFarlane:
>>
>>>>> That sounds amazing. As you know CSL 1.0.2 is about to appear soon, and
>>
>>>>> 1.1 is also already pretty advanced. It would be great if it were easy
>>
>>>>> to integrate the new features in your library.
>>
>>>> Is there a list of changes in CSL 1.0.2 somewhere, so I can see
>>
>>>> what will need to be supported when it comes out?
>>
>>>>
>>
>>>
>>
>>> Yeah, there's a summary of changes here:
>>
>>> https://docs.google.com/document/d/1wY1cOOamDYYh8VNW7h_uleqieBDGOa_LYsRiVdQy1RI/edit#heading=h.wsywjzy5t4j6
>>
>>>
>>
>>> As you can see it's a terms, types and variables release. That should be
>>
>>> trivial to support. (There were some changes to that list due to
>>
>>> comments during the comments period so that's not the final list of
>>
>>> changes.)
>>
>>>
>>
>>> 1.1 will be more complex, but we plan to document the new features with
>>
>>> a detailled changelog and new tests in the test-suite.
>>
>> --
>>
>> You received this message because you are subscribed to a topic in the Google Groups "pandoc-discuss" group.
>>
>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/pandoc-discuss/LaPbYbP8GU4/unsubscribe.
>>
>> To unsubscribe from this group and all its topics, send an email to [pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org](mailto:pandoc-discuss%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org).
>>
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m2bligt9ra.fsf%40MacBook-Pro.hsd1.ca.comcast.net.
>
> --
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit [https://groups.google.com/d/msgid/pandoc-discuss/CAGOSsd%3DXs6u6Xup9YugR-zd6FC_29QFs1asEcARA4m2UPts_vQ%40mail.gmail.com](https://groups.google.com/d/msgid/pandoc-discuss/CAGOSsd%3DXs6u6Xup9YugR-zd6FC_29QFs1asEcARA4m2UPts_vQ%40mail.gmail.com?utm_medium=email&utm_source=footer).

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/cy_zB0ryMZZFLdb4qMo7kMPfyBvIZwVJI81wd5aRWS3_SYgyYoch4zRkj-LMCAVRvnumZfgy_DG1kfG3pQem7P9HnuT7UQhPymOhc7DK-EQ%3D%40protonmail.com.

[-- Attachment #2: Type: text/html, Size: 7761 bytes --]

Re: WIP: better citation processing

[Joseph Reagle]

On 8/16/20 6:59 PM, John MacFarlane wrote:
> I've used this library to create a new filter,
> new-pandoc-citeproc, which now passes most of the pandoc-citeproc
> test suite but runs around 6 times faster.

Awesome! I've topped 10k items in my YAML file. Even though you kindly sped up some citeproc bottlenecks for me in the past, I continue to "subset" the 10k file into document-specific YAML files when building a document. That is, it's faster for me to regex my document for citations and pull their entries out of the 10k YAML file before handing it off to pandoc than just using pandoc itself.

I'd love to drop all of this! (Though, to be fair, this is fast because my YAML is super-contstrained and need not be properly parsed, only "chunked.")

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/e94c6f9d-e6d8-099a-4bf2-7aed30476a6c%40reagle.org.

Re: WIP: better citation processing

[John MacFarlane]

Joseph Reagle <joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org> writes:

> On 8/16/20 6:59 PM, John MacFarlane wrote:
>> I've used this library to create a new filter,
>> new-pandoc-citeproc, which now passes most of the pandoc-citeproc
>> test suite but runs around 6 times faster.
>
> Awesome! I've topped 10k items in my YAML file. Even though you kindly sped up some citeproc bottlenecks for me in the past, I continue to "subset" the 10k file into document-specific YAML files when building a document. That is, it's faster for me to regex my document for citations and pull their entries out of the 10k YAML file before handing it off to pandoc than just using pandoc itself.

There are two separate issues here:

1.  Parsing the YAML metadata (this is done in pandoc's markdown reader)
2.  Processing the CSL (this is done by pandoc-citeproc)

The new library will speed up 2, but it won't affect 1, and I
suspect 1 is the bottleneck for you.

For #1, we have this open issue:

https://github.com/jgm/pandoc/issues/6084

The issue could be fixed by switching back from the pure Haskell
HsYaml to the wrapped C library yaml.  I hate to do that, though,
because I've been trying to remove all pure C library
dependencies from pandoc (both for security reasons and
because they don't work with e.g. compiling to JavaScript
with ghcjs).

Here's a workaround that should work right now.  Reading CSL JSON
is fast, so you could try using pandoc-citeproc -j to convert
your YAML bibliography to CSL, then refer to the CSL bibliography
in your pandoc metadata.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m2v9hbbyyu.fsf%40johnmacfarlane.net.

Re: WIP: better citation processing

[John MacFarlane]

> Here's a workaround that should work right now.  Reading CSL JSON
> is fast, so you could try using pandoc-citeproc -j to convert
> your YAML bibliography to CSL, then refer to the CSL bibliography
> in your pandoc metadata.

Just tested this with a 10K entry bib file.
Using YAML with an empty md file, 16 seconds.
Using JSON, 1.8 seconds.

By the way, my preferred fix for the slow YAML parsing issue
would be progress on the upstream HsYaml:

https://github.com/haskell-hvr/HsYAML/issues/40

But the developers have not responded.  If anyone out there
is a whiz at diagnosing Haskell space leaks, it would be great
if you could have a look at that library and see if there's
a fix.  I wasn't able to diagnose it in the time I had to devote
to it.

Re: WIP: better citation processing

[Joseph Reagle]

On 8/21/20 3:06 PM, John MacFarlane wrote:
> The new library will speed up 2, but it won't affect 1, and I
> suspect 1 is the bottleneck for you.

I hadn't noticed the performance drop given I continued subsetting my readings.yaml into a document specific yaml file.

> Here's a workaround that should work right now.  Reading CSL JSON
> is fast, so you could try using pandoc-citeproc -j to convert
> your YAML bibliography to CSL, then refer to the CSL bibliography
> in your pandoc metadata.

It could help, but doesn't practically.

Creating a subset of my full bibliography for a document and building a single ref markdown to HTML takes 1 second, which is great for frequent edits/builds of my markdown.

Using the full YAML bibliography takes 90 seconds. 

Converting the full bibliography from YAML to JSON with pandoc-citeproc takes a similar 90 seconds and then running the simple single-ref document build with the resulting JSON never exits -- I killed it after four minutes. Perhaps this is because I typically include the bibliography file as a second input file. (I forget why I do this, there's probably something on this list's archives from years ago.)

Using the `--bibliography=FILE` approach has much better performance. My full YAML build takes 35 seconds, and the JSON takes 1 second -- similar to subsetting the YAML. However, adding 90 seconds to convert from YAML to JSON every time I edit my bibliography is a big hit. 

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/91c2cfed-7211-4194-72cf-5c01abf2315c%40reagle.org.

Re: WIP: better citation processing

[John MacFarlane]

Joseph Reagle <joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org> writes:

> Converting the full bibliography from YAML to JSON with pandoc-citeproc takes a similar 90 seconds

Yes, because pandoc is being used to read the YAML.  But you'd
only need to do this when the bibliography changes, not every
time you process the document.  (A Makefile would make it
easy to get this right.)

> and then running the simple single-ref document build with the resulting JSON never exits -- I killed it after four minutes. Perhaps this is because I typically include the bibliography file as a second input file. (I forget why I do this, there's probably something on this list's archives from years ago.)

You need to use --bibliography yourbib.json

Otherwise it will be interpret as Markdown (and pandoc is
probably getting flummoxed interpreting the JSON as Markdown).
With YAML it works because this is a valid pandoc Markdown
YAML metadata block.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m2sgcf8uvo.fsf%40johnmacfarlane.net.

Re: WIP: better citation processing

[Joseph]

[-- Attachment #1.1: Type: text/plain, Size: 1236 bytes --]

I subset my YAML bibliography into document-specific bibliographies based 
on keys used in the corresponding markdown file. Otherwise, it takes pandoc 
too long to parse the ~6MB YAML bibliography. 

Since I had to update my "find citation keys in markdown files" script, I 
thought I'd look in on performance of the YAML and  JSON parsing. 
`pandoc-citeproc` isn't around anymore, and pandoc itself chokes on 
converting from YAML to CSLJSON. Something killed it off after four 
minutes. Is there an alternative for creating CSLJSON?

```
╰─➤  time pandoc -f markdown --citeproc -t csljson readings.yaml 
[2]    28124 killed     pandoc -f markdown --citeproc -t csljson 
readings.yaml
pandoc -f markdown --citeproc -t csljson readings.yaml  265.51s user 70.23s 
system 48% cpu 11:31.15 total
```

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/30abf74d-df6f-4df8-aaee-a493331d4e92n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1724 bytes --]

Re: WIP: better citation processing

[John MacFarlane]

See https://github.com/jgm/pandoc/issues/6084

You could try grabbing an old pandoc-citeproc binary (from before
we started using HsYAML) and use it to convert your YAML to
bibtex or csljson.

Joseph <joseph.2011-T1oY19WcHSwdnm+yROfE0A@public.gmane.org> writes:

> I subset my YAML bibliography into document-specific bibliographies based 
> on keys used in the corresponding markdown file. Otherwise, it takes pandoc 
> too long to parse the ~6MB YAML bibliography. 
>
> Since I had to update my "find citation keys in markdown files" script, I 
> thought I'd look in on performance of the YAML and  JSON parsing. 
> `pandoc-citeproc` isn't around anymore, and pandoc itself chokes on 
> converting from YAML to CSLJSON. Something killed it off after four 
> minutes. Is there an alternative for creating CSLJSON?
>
> ```
> ╰─➤  time pandoc -f markdown --citeproc -t csljson readings.yaml 
> [2]    28124 killed     pandoc -f markdown --citeproc -t csljson 
> readings.yaml
> pandoc -f markdown --citeproc -t csljson readings.yaml  265.51s user 70.23s 
> system 48% cpu 11:31.15 total
> ```
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/30abf74d-df6f-4df8-aaee-a493331d4e92n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m24kdx38qp.fsf%40MacBook-Pro-2.hsd1.ca.comcast.net.

Re: WIP: better citation processing

[Joseph Reagle]

Thanks, good idea: I grabbed pandoc-citeproc 0.16.2 out of pandoc 2.7.3 .

Converting from yaml to csljson takes 50 seconds, so I can't use it in my daily work builds (where I'm frequently adding and citing new sources), but it'll be handy for the occasional use.

BTW: pandoc 2.14 is blazing fast with the json, taking less than a second to build a file using a 8MB 11K entry csljson file! Unfortunately, it dies on the 6MB 11MB yaml version.

On 21-06-16 15:33, John MacFarlane wrote:
> See https://github.com/jgm/pandoc/issues/6084
> 
> You could try grabbing an old pandoc-citeproc binary (from before
> we started using HsYAML) and use it to convert your YAML to
> bibtex or csljson.

WIP: better citation processing - nightlies now available for testing!

[John MacFarlane]

I haven't yet released the new citeproc library, but I've
merged the pandoc changes to master.  (The code currently
uses an unreleased dev version of the citeproc library.)
The plan is to include this in the next release of pandoc.

Nightlies are available for testing at

https://github.com/jgm/pandoc/actions/runs/270103289

To test the new citation support, just use the '--citeproc' option
instead of '--filter pandoc-citeproc'.

This is still a work in progress, but it would benefit
from testing, so if you could give it a try and report
anything that seems wrong, that would be great!

OT: Re: WIP: better citation processing

[Anton Shepelev]

(Some Monday humour):
I misread the thread title as:

          RIP: better citation processing

and  started  reading the article with an eye to sad
tidings, but none came.