Citeproc

Citeproc

[Denis Maier]

Hi,

John MacFarlane, the developper of Pandoc, has released a new Citeproc 
that generates citations and bibliographies using CSL style files (CSL= 
citation style language). While it is written in Haskell and while it's 
primarily intended for use with Pandoc, it can also be used in other 
contexts. Provided with a JSON encoded list of references via stdin, it 
can produce formatted output.

I know that ConTeXt has its own infrastructure to format bibliographies 
and citations, but, given the enormous amount of available styles in 
CSL, I nevertheless think that this could be a worthwile addition. What 
would be necessary to make such a toll usable with ConTeXt? How 
complicated would that be?

Best,
Denis
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Citeproc

[Aditya Mahajan]

> John MacFarlane, the developper of Pandoc, has released a new Citeproc 
> that generates citations and bibliographies using CSL style files (CSL= 
> citation style language). While it is written in Haskell and while it's 
> primarily intended for use with Pandoc, it can also be used in other 
> contexts. Provided with a JSON encoded list of references via stdin, it 
> can produce formatted output.

The man page of the new citeproc executable, for those who are interested:

https://github.com/jgm/citeproc/blob/master/man/citeproc.1.md

> I know that ConTeXt has its own infrastructure to format bibliographies 
> and citations, but, given the enormous amount of available styles in 
> CSL, I nevertheless think that this could be a worthwile addition. What 
> would be necessary to make such a toll usable with ConTeXt? How 
> complicated would that be?

In principle, this should be trivial. Define a new command, say \citeproc[ref], which saves the values of `ref` to a lua table, and at the end of the run, write that lua table to an aux .json file, call citeproc and store output to a new file, and on the second run read from that file. This is essentially how the old bibtex used to work.

Note that this scheme has a few drawbacks: First, it needs to call an external executable, which can be slower than directly reading the bib file via lua. Second, it creates a bunch of extra auxiliary files, which is always annoying. But I do agree that it will provide us with the ability to use the large number CSL styles.

Of course, a better option will be write a CSL processor in Lua, but that is a lot of tedious (but relatively simple) task. I wonder if there is already a CSL processor written in Lua.

Aditya
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Citeproc

[Hans Hagen]

On 10/5/2020 2:47 AM, Aditya Mahajan wrote:
>> John MacFarlane, the developper of Pandoc, has released a new Citeproc
>> that generates citations and bibliographies using CSL style files (CSL=
>> citation style language). While it is written in Haskell and while it's
>> primarily intended for use with Pandoc, it can also be used in other
>> contexts. Provided with a JSON encoded list of references via stdin, it
>> can produce formatted output.
> 
> The man page of the new citeproc executable, for those who are interested:
> 
> https://github.com/jgm/citeproc/blob/master/man/citeproc.1.md
> 
>> I know that ConTeXt has its own infrastructure to format bibliographies
>> and citations, but, given the enormous amount of available styles in
>> CSL, I nevertheless think that this could be a worthwile addition. What
>> would be necessary to make such a toll usable with ConTeXt? How
>> complicated would that be?
> 
> In principle, this should be trivial. Define a new command, say \citeproc[ref], which saves the values of `ref` to a lua table, and at the end of the run, write that lua table to an aux .json file, call citeproc and store output to a new file, and on the second run read from that file. This is essentially how the old bibtex used to work.
> 
> Note that this scheme has a few drawbacks: First, it needs to call an external executable, which can be slower than directly reading the bib file via lua. Second, it creates a bunch of extra auxiliary files, which is always annoying. But I do agree that it will provide us with the ability to use the large number CSL styles.
> 
> Of course, a better option will be write a CSL processor in Lua, but that is a lot of tedious (but relatively simple) task. I wonder if there is already a CSL processor written in Lua.
the bib module can read lua files (or whatever)

i have no clue what csl is but I assume it's just some key / value thing 
as the bib module itself should to the logic

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Citeproc

[Denis Maier]

Am 05.10.2020 um 14:02 schrieb Hans Hagen:
> i have no clue what csl is but I assume it's just some key / value 
> thing as the bib module itself should to the logic 

Sorry, I should have provided more background information.

CSL = Citation Style Language; see https://citationstyles.org/

In the words of the specification: "The Citation Style Language (CSL) is 
an XML-based format to describe the formatting of citations, notes and 
bibliographies". I.e., it's quite similar to what bibtex (the language, 
not the format) is, or what context's own bibliographic language does. 
The language is used by a bunch of tools, including Zotero, Pandoc, etc. 
There is also a metadata format called CSL JSON, but that's of minor 
interest here.

The language itself is defined in a specification, the citations are 
rendered by a so called citeproc. Citeproc-js, an implementation in 
Javascript, is used by Zotero and Mendeley to render citations in Word 
and LibreOffice, citeproc-php is used on some webplatforms, 
pandoc-citeproc (now being replaced by a new Haskell library) does the 
same thing for pandoc, and there are also implementations in python, c#, 
a new one being developped in Rust will eventually replace citeproc-js 
at some point.

Now, the new Haskell citeproc can be used as a standalone program as 
well, and that's why I was thinking it could be used together with 
ConTeXt in some way; essentially as Aditya described it in his response.

Best,
Denis
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Citeproc

[Denis Maier]

[-- Attachment #1.1: Type: text/plain, Size: 1818 bytes --]

Am 05.10.2020 um 02:47 schrieb Aditya Mahajan:
>> John MacFarlane, the developper of Pandoc, has released a new Citeproc
>> that generates citations and bibliographies using CSL style files (CSL=
>> citation style language). While it is written in Haskell and while it's
>> primarily intended for use with Pandoc, it can also be used in other
>> contexts. Provided with a JSON encoded list of references via stdin, it
>> can produce formatted output.
> The man page of the new citeproc executable, for those who are 
> interested:
>
> https://github.com/jgm/citeproc/blob/master/man/citeproc.1.md
Thanks for adding this.
>
>> I know that ConTeXt has its own infrastructure to format bibliographies
>> and citations, but, given the enormous amount of available styles in
>> CSL, I nevertheless think that this could be a worthwile addition. What
>> would be necessary to make such a toll usable with ConTeXt? How
>> complicated would that be?
> In principle, this should be trivial. [...]
>
> Note that this scheme has a few drawbacks: [...] But I do agree that 
> it will provide us with the ability to use the large number CSL styles.
Thanks for outlining what would be needed.
> Of course, a better option will be write a CSL processor in Lua, but 
> that is a lot of tedious (but relatively simple) task. I wonder if 
> there is already a CSL processor written in Lua.
Yeah, a Lua citeproc would be the best way to go. I contemplated some 
time ago doing this as a learning project, but I have serious doubts I'd 
be able to actually produce something usable by others. There was one 
being developped, but a computer hazard destroyed it somewhere along the 
way Besides, the person who did it told me Lua has somd shortcomings 
that make it a suboptimal tool for this task. But I can't really tell...

Best,
Denis

[-- Attachment #1.2: Type: text/html, Size: 2931 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Citeproc

[Aditya Mahajan]

On Mon, 5 Oct 2020, Hans Hagen wrote:

> On 10/5/2020 2:47 AM, Aditya Mahajan wrote:
> > > John MacFarlane, the developper of Pandoc, has released a new Citeproc
> > > that generates citations and bibliographies using CSL style files (CSL=
> > > citation style language). While it is written in Haskell and while it's
> > > primarily intended for use with Pandoc, it can also be used in other
> > > contexts. Provided with a JSON encoded list of references via stdin, it
> > > can produce formatted output.
> > 
> > The man page of the new citeproc executable, for those who are interested:
> > 
> > https://github.com/jgm/citeproc/blob/master/man/citeproc.1.md
> > 
> > > I know that ConTeXt has its own infrastructure to format bibliographies
> > > and citations, but, given the enormous amount of available styles in
> > > CSL, I nevertheless think that this could be a worthwile addition. What
> > > would be necessary to make such a toll usable with ConTeXt? How
> > > complicated would that be?
> > 
> > In principle, this should be trivial. Define a new command, say
> > \citeproc[ref], which saves the values of `ref` to a lua table, and at the
> > end of the run, write that lua table to an aux .json file, call citeproc and
> > store output to a new file, and on the second run read from that file. This
> > is essentially how the old bibtex used to work.
> > 
> > Note that this scheme has a few drawbacks: First, it needs to call an
> > external executable, which can be slower than directly reading the bib file
> > via lua. Second, it creates a bunch of extra auxiliary files, which is
> > always annoying. But I do agree that it will provide us with the ability to
> > use the large number CSL styles.
> > 
> > Of course, a better option will be write a CSL processor in Lua, but that is
> > a lot of tedious (but relatively simple) task. I wonder if there is already
> > a CSL processor written in Lua.
> the bib module can read lua files (or whatever)
> 
> i have no clue what csl is but I assume it's just some key / value thing as
> the bib module itself should to the logic

It is similar to a bst file and specifies how the bibliography should be formatted. The specification is written as an XML file, for example, this is the specification for APA style:

https://www.zotero.org/styles/apa-5th-edition

The detailed schema is here:

https://github.com/citation-style-language/schema/tree/v1.0.1

There are a large number of such specifications available. For example: 

  https://www.zotero.org/styles

So, the user can easily search for a style that matches his/her needs and simply use it. The task of finding a reference from a reference database is left to a CSL processor and there a few of them available:

https://citationstyles.org/developers/

In principle, it should be relatively straight forward (but tedious) to write a processor in Lua. But I am not volunteering for that :-)


Aditya
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Citeproc

[Hans Hagen]

On 10/5/2020 2:55 PM, Denis Maier wrote:
> Am 05.10.2020 um 02:47 schrieb Aditya Mahajan:
>>> John MacFarlane, the developper of Pandoc, has released a new Citeproc
>>> that generates citations and bibliographies using CSL style files (CSL=
>>> citation style language). While it is written in Haskell and while it's
>>> primarily intended for use with Pandoc, it can also be used in other
>>> contexts. Provided with a JSON encoded list of references via stdin, it
>>> can produce formatted output.
>> The man page of the new citeproc executable, for those who are 
>> interested:
>>
>> https://github.com/jgm/citeproc/blob/master/man/citeproc.1.md
> Thanks for adding this.
>>
>>> I know that ConTeXt has its own infrastructure to format bibliographies
>>> and citations, but, given the enormous amount of available styles in
>>> CSL, I nevertheless think that this could be a worthwile addition. What
>>> would be necessary to make such a toll usable with ConTeXt? How
>>> complicated would that be?
>> In principle, this should be trivial. [...]
>>
>> Note that this scheme has a few drawbacks: [...] But I do agree that 
>> it will provide us with the ability to use the large number CSL styles.
> Thanks for outlining what would be needed.
>> Of course, a better option will be write a CSL processor in Lua, but 
>> that is a lot of tedious (but relatively simple) task. I wonder if 
>> there is already a CSL processor written in Lua.
> Yeah, a Lua citeproc would be the best way to go. I contemplated some 
> time ago doing this as a learning project, but I have serious doubts I'd 
> be able to actually produce something usable by others. There was one 
> being developped, but a computer hazard destroyed it somewhere along the 
> way Besides, the person who did it told me Lua has somd shortcomings 
> that make it a suboptimal tool for this task. But I can't really tell...
if it's an xml spec then using lua makes not that much sense (no gain) 
.. just process the xml

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Citeproc

[Denis Maier]

Am 05.10.2020 um 22:04 schrieb Hans Hagen:
> if it's an xml spec then using lua makes not that much sense (no gain) 
> .. just process the xml 
Hmm, but the XML is just a descriptive grammar of how bibliographies and 
citations should be rendered. As Aditya has pointed out it's quite 
similar to the role of bst style files for bibtex. So, a citeproc will 
take a style file (in xml) and apply it to the metadata (based on the 
items actually cited in a document).

Denis
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Citeproc

[Taco Hoekwater]

> On 5 Oct 2020, at 22:26, Denis Maier <denis.maier.lists@mailbox.org> wrote:
> 
> Am 05.10.2020 um 22:04 schrieb Hans Hagen:
>> if it's an xml spec then using lua makes not that much sense (no gain) .. just process the xml 
> Hmm, but the XML is just a descriptive grammar of how bibliographies and citations should be rendered. As Aditya has pointed out it's quite similar to the role of bst style files for bibtex. So, a citeproc will take a style file (in xml) and apply it to the metadata (based on the items actually cited in a document).

I recall discussing this idea with Bruce D’Arcus a long time ago (somewhere in the early 00-es, maybe).

The CSL files look like XML, but really they are more like a macro processing language. Parsing and interpreting that should not be all that hard, I expect. If I had to do this, I would convert the XML <macro> tags into actual lua functions while parsing the XML. The implied processing language is quite straightforward (unlike the stack-based postfix language in bibtex bst files). I don’t have time right now, but if someone reminds me in two months or so, I might have a go at it.

Best wishes,
Taco

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Citeproc

[denis.maier.lists]

[-- Attachment #1: Type: text/html, Size: 1449 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Citeproc

[Hans Hagen]

On 10/6/2020 10:50 AM, denis.maier.lists@mailbox.org wrote:
> Taco Hoekwater <taco@elvenkind.com <mailto:taco@elvenkind.com>> hat am 
> 06.10.2020 09:25 geschrieben:
> 
>> I recall discussing this idea with Bruce D’Arcus a long time ago 
>> (somewhere in the early 00-es, maybe).
>>
>> The CSL files look like XML, but really they are more like a macro 
>> processing language. Parsing and interpreting that should not be all 
>> that hard, I expect. If I had to do this, I would convert the XML 
>> <macro> tags into actual lua functions while parsing the XML. The 
>> implied processing language is quite straightforward (unlike the 
>> stack-based postfix language in bibtex bst files). I don’t have time 
>> right now, but if someone reminds me in two months or so, I might have 
>> a go at it.

that was indeed what stroke me when i saw the code: it's just some 
simple language wrapped in angle bracked ... straightforward conversion 
to lua seems not that hard (if statements mixed with some funcition calls)

> That would be great. I'll remind you, be sure.
> 
> I agree the whole endeavour should be not to hard although there are 
> some pitfalls, especially regarding disambiguation and so. Anyway, let's 
> discuss details later, and let me know if I can be of any help, even if 
> it's just cheerleading.

i was thinking of a mid winter cold evenings project -)

(so we can pick up this thread later)

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Citeproc

[Denis Maier]

Am 07.10.2020 um 12:01 schrieb Hans Hagen:
> On 10/6/2020 10:50 AM, denis.maier.lists@mailbox.org wrote:
>> Taco Hoekwater <taco@elvenkind.com <mailto:taco@elvenkind.com>> hat 
>> am 06.10.2020 09:25 geschrieben:
>>
>>> I recall discussing this idea with Bruce D’Arcus a long time ago 
>>> (somewhere in the early 00-es, maybe).
>>>
>>> The CSL files look like XML, but really they are more like a macro 
>>> processing language. Parsing and interpreting that should not be all 
>>> that hard, I expect. If I had to do this, I would convert the XML 
>>> <macro> tags into actual lua functions while parsing the XML. The 
>>> implied processing language is quite straightforward (unlike the 
>>> stack-based postfix language in bibtex bst files). I don’t have time 
>>> right now, but if someone reminds me in two months or so, I might 
>>> have a go at it.
>
>> That would be great. I'll remind you, be sure.
>>
>> I agree the whole endeavour should be not to hard although there are 
>> some pitfalls, especially regarding disambiguation and so. Anyway, 
>> let's discuss details later, and let me know if I can be of any help, 
>> even if it's just cheerleading.
>
> i was thinking of a mid winter cold evenings project -)
>
> (so we can pick up this thread later)

Cool, so let's talk again in a couple of months then.

Best,
Denis
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________