Manipulate CSL bibliography before processing

Manipulate CSL bibliography before processing

[Denis Maier]

Hi,

is it possible to manipulate a CSL bibliography before Pandoc processes 
the citations?
Let's say I have one huge database, and I want to do ad hoc 
modifications, for example add a certain field to a couple of entries.

The specific use case would be this:
You write an essay about Kant, and you want to use the usual 
abbreviations (KrV, KpV, etc.), but you don't want to have those 
abbreviations in your main database. In  your metadata you could have 
something like this:

```
bibliography: bib.json
bibliography-modifications:
   - id: kant-id1
     add:
       - field: citation-label
         content: KrV
```

Of course, you can always implement this with some sort of 
pre-processing, e.g. with a python script. But a pandoc solution would 
be neat.

Is the content of the bibliography file exposed to the filter 
architecture before citeproc actually processes the bibliography?

Best,
Denis

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/54c0b3a9-f742-18e4-9fc2-8a3b7f2f3632%40mailbox.org.

Re: Manipulate CSL bibliography before processing

[Albert Krewinkel]

Denis Maier writes:

> is it possible to manipulate a CSL bibliography before Pandoc
> processes the citations? Let's say I have one huge database, and I
> want to do ad hoc modifications, for example add a certain field to a
> couple of entries.
>
> [...]
>
> Is the content of the bibliography file exposed to the filter architecture
> before citeproc actually processes the bibliography?

Kinda sorta. Not really. What's possible is to add the bibliography
directly into the metadata under the `references` field with a filter.
The metadata fields can then be modified by subsequent filters, and
pandoc-citeproc will use the final `references` to generate the
bibliography.

Here's a Lua filter to do the metadata inlining:

    function Meta (meta)
      local refs_yson = pandoc.pipe(
        'pandoc-citeproc',
        {'-y', pandoc.utils.stringify(meta.bibliography)},
        ''
      )
      local references = pandoc.read(refs_yson).meta.references

      -- unset bibliography to avoid pandoc-citeproc from using it again
      meta.bibliography = nil
      meta.references = references
      return meta
    end



--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Manipulate CSL bibliography before processing

[Albert Krewinkel]

Addendum: with most shells, this can also be done directly on the
command line:

    pandoc --metadata-file <(pandoc-citeproc -y biblio.bib) ...


Albert Krewinkel writes:

> Denis Maier writes:
>
>> is it possible to manipulate a CSL bibliography before Pandoc
>> processes the citations? Let's say I have one huge database, and I
>> want to do ad hoc modifications, for example add a certain field to a
>> couple of entries.
>>
>> [...]
>>
>> Is the content of the bibliography file exposed to the filter architecture
>> before citeproc actually processes the bibliography?
>
> Kinda sorta. Not really. What's possible is to add the bibliography
> directly into the metadata under the `references` field with a filter.
> The metadata fields can then be modified by subsequent filters, and
> pandoc-citeproc will use the final `references` to generate the
> bibliography.
>
> Here's a Lua filter to do the metadata inlining:
>
>     function Meta (meta)
>       local refs_yson = pandoc.pipe(
>         'pandoc-citeproc',
>         {'-y', pandoc.utils.stringify(meta.bibliography)},
>         ''
>       )
>       local references = pandoc.read(refs_yson).meta.references
>
>       -- unset bibliography to avoid pandoc-citeproc from using it again
>       meta.bibliography = nil
>       meta.references = references
>       return meta
>     end
>
>
>
> --
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124


--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Manipulate CSL bibliography before processing

[Denis Maier]

Am 17.09.2020 um 12:19 schrieb Albert Krewinkel:
> Addendum: with most shells, this can also be done directly on the
> command line:
>
>      pandoc --metadata-file <(pandoc-citeproc -y biblio.bib) ...

Thanks! That's neat.

My bibliography is already in yaml, so this here works just fine:
pandoc --metadata-file biblio.yaml

(I wasn't aware that `--bibliography` is just a shorthand for `-F 
pandoc-citeproc --metadata-file`. Thanks for pointing that out.

>
>
> Albert Krewinkel writes:
>
>> Denis Maier writes:
>>
>>> is it possible to manipulate a CSL bibliography before Pandoc
>>> processes the citations? Let's say I have one huge database, and I
>>> want to do ad hoc modifications, for example add a certain field to a
>>> couple of entries.
>>>
>>> [...]
>>>
>>> Is the content of the bibliography file exposed to the filter architecture
>>> before citeproc actually processes the bibliography?
>> Kinda sorta. Not really. What's possible is to add the bibliography
>> directly into the metadata under the `references` field with a filter.
>> The metadata fields can then be modified by subsequent filters, and
>> pandoc-citeproc will use the final `references` to generate the
>> bibliography.
>>
>> Here's a Lua filter to do the metadata inlining:
>>
>>      function Meta (meta)
>>        local refs_yson = pandoc.pipe(
>>          'pandoc-citeproc',
>>          {'-y', pandoc.utils.stringify(meta.bibliography)},
>>          ''
>>        )
>>        local references = pandoc.read(refs_yson).meta.references
>>
>>        -- unset bibliography to avoid pandoc-citeproc from using it again
>>        meta.bibliography = nil
>>        meta.references = references
>>        return meta
>>      end
>>
>>
>>
>> --
>> Albert Krewinkel
>> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
>
> --
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
>

Re: Manipulate CSL bibliography before processing

[John MacFarlane]

I wonder whether you could use --citation-abbreviations for this?
See pandoc-citeproc manual.

Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:

> Hi,
>
> is it possible to manipulate a CSL bibliography before Pandoc processes 
> the citations?
> Let's say I have one huge database, and I want to do ad hoc 
> modifications, for example add a certain field to a couple of entries.
>
> The specific use case would be this:
> You write an essay about Kant, and you want to use the usual 
> abbreviations (KrV, KpV, etc.), but you don't want to have those 
> abbreviations in your main database. In  your metadata you could have 
> something like this:
>
> ```
> bibliography: bib.json
> bibliography-modifications:
>    - id: kant-id1
>      add:
>        - field: citation-label
>          content: KrV
> ```
>
> Of course, you can always implement this with some sort of 
> pre-processing, e.g. with a python script. But a pandoc solution would 
> be neat.
>
> Is the content of the bibliography file exposed to the filter 
> architecture before citeproc actually processes the bibliography?
>
> Best,
> Denis
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/54c0b3a9-f742-18e4-9fc2-8a3b7f2f3632%40mailbox.org.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m2y2l8i8so.fsf%40MacBook-Pro.hsd1.ca.comcast.net.

Re: Manipulate CSL bibliography before processing

[Denis Maier]

That's related, but not the same. --citation-abbreviatons is currently 
used to supply short forms of certain fields. E.g container-titles, and 
so. Also, they match against strings. The citation-label works similar 
to biblatex's shorthand, but there is currently no way to define them on 
a per-document basis.
But that could of course be added to the --citation-abbreviations feature.

Am 17.09.2020 um 19:59 schrieb John MacFarlane:
> I wonder whether you could use --citation-abbreviations for this?
> See pandoc-citeproc manual.
>
> Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>
>> Hi,
>>
>> is it possible to manipulate a CSL bibliography before Pandoc processes
>> the citations?
>> Let's say I have one huge database, and I want to do ad hoc
>> modifications, for example add a certain field to a couple of entries.
>>
>> The specific use case would be this:
>> You write an essay about Kant, and you want to use the usual
>> abbreviations (KrV, KpV, etc.), but you don't want to have those
>> abbreviations in your main database. In  your metadata you could have
>> something like this:
>>
>> ```
>> bibliography: bib.json
>> bibliography-modifications:
>>     - id: kant-id1
>>       add:
>>         - field: citation-label
>>           content: KrV
>> ```
>>
>> Of course, you can always implement this with some sort of
>> pre-processing, e.g. with a python script. But a pandoc solution would
>> be neat.
>>
>> Is the content of the bibliography file exposed to the filter
>> architecture before citeproc actually processes the bibliography?
>>
>> Best,
>> Denis
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/54c0b3a9-f742-18e4-9fc2-8a3b7f2f3632%40mailbox.org.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/9af493a5-336d-a726-4266-b89104f0cebc%40mailbox.org.

Re: Manipulate CSL bibliography before processing

[John MacFarlane]

Our library does allow you to define citation-label for a
reference in your CSL JSON or pandoc YAML bibliography.
(If you don't define it, and a label is called for, then
the library will generate one in a manner similar to
citeproc.js.  But you CAN define it explicitly, and in that
case the defined version will be used.)  I think that solves
your problem all by itself.

Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:

> That's related, but not the same. --citation-abbreviatons is currently 
> used to supply short forms of certain fields. E.g container-titles, and 
> so. Also, they match against strings. The citation-label works similar 
> to biblatex's shorthand, but there is currently no way to define them on 
> a per-document basis.
> But that could of course be added to the --citation-abbreviations feature.
>
> Am 17.09.2020 um 19:59 schrieb John MacFarlane:
>> I wonder whether you could use --citation-abbreviations for this?
>> See pandoc-citeproc manual.
>>
>> Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>>
>>> Hi,
>>>
>>> is it possible to manipulate a CSL bibliography before Pandoc processes
>>> the citations?
>>> Let's say I have one huge database, and I want to do ad hoc
>>> modifications, for example add a certain field to a couple of entries.
>>>
>>> The specific use case would be this:
>>> You write an essay about Kant, and you want to use the usual
>>> abbreviations (KrV, KpV, etc.), but you don't want to have those
>>> abbreviations in your main database. In  your metadata you could have
>>> something like this:
>>>
>>> ```
>>> bibliography: bib.json
>>> bibliography-modifications:
>>>     - id: kant-id1
>>>       add:
>>>         - field: citation-label
>>>           content: KrV
>>> ```
>>>
>>> Of course, you can always implement this with some sort of
>>> pre-processing, e.g. with a python script. But a pandoc solution would
>>> be neat.
>>>
>>> Is the content of the bibliography file exposed to the filter
>>> architecture before citeproc actually processes the bibliography?
>>>
>>> Best,
>>> Denis
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/54c0b3a9-f742-18e4-9fc2-8a3b7f2f3632%40mailbox.org.
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/9af493a5-336d-a726-4266-b89104f0cebc%40mailbox.org.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m2363gw0cv.fsf%40MacBook-Pro.hsd1.ca.comcast.net.

Re: Manipulate CSL bibliography before processing

[Denis Maier]

Not directly.

The question is whether it is possible to have this field in a 
per-document bibliography while still using a global bibliography.
But I guess that should be possible with a filter.

Or, maybe an idea: currently two if we have metadata in two metadata 
files like so:

```
data:
   fieldA: X
```

```
data:
   fieldB: Y
```

--metadata-file first --metadata-file second will result in:

```
data:
   fieldB: Y
```

That is, the later metadata in the second file just overwrites the 
first. I don't know if there's an easy way to enable merging of metadata 
blocks?


By the way, the behavior for automatic label creation will change in CSL 
1.1. (The citation-label field will serve as a shorthand, but it should 
also be possible to use this only for selected items. So automatic 
generation of citation-label only happens when explicitly enabled in a 
style.)

Am 17.09.2020 um 23:37 schrieb John MacFarlane:
> Our library does allow you to define citation-label for a
> reference in your CSL JSON or pandoc YAML bibliography.
> (If you don't define it, and a label is called for, then
> the library will generate one in a manner similar to
> citeproc.js.  But you CAN define it explicitly, and in that
> case the defined version will be used.)  I think that solves
> your problem all by itself.
>
> Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>
>> That's related, but not the same. --citation-abbreviatons is currently
>> used to supply short forms of certain fields. E.g container-titles, and
>> so. Also, they match against strings. The citation-label works similar
>> to biblatex's shorthand, but there is currently no way to define them on
>> a per-document basis.
>> But that could of course be added to the --citation-abbreviations feature.
>>
>> Am 17.09.2020 um 19:59 schrieb John MacFarlane:
>>> I wonder whether you could use --citation-abbreviations for this?
>>> See pandoc-citeproc manual.
>>>
>>> Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>>>
>>>> Hi,
>>>>
>>>> is it possible to manipulate a CSL bibliography before Pandoc processes
>>>> the citations?
>>>> Let's say I have one huge database, and I want to do ad hoc
>>>> modifications, for example add a certain field to a couple of entries.
>>>>
>>>> The specific use case would be this:
>>>> You write an essay about Kant, and you want to use the usual
>>>> abbreviations (KrV, KpV, etc.), but you don't want to have those
>>>> abbreviations in your main database. In  your metadata you could have
>>>> something like this:
>>>>
>>>> ```
>>>> bibliography: bib.json
>>>> bibliography-modifications:
>>>>      - id: kant-id1
>>>>        add:
>>>>          - field: citation-label
>>>>            content: KrV
>>>> ```
>>>>
>>>> Of course, you can always implement this with some sort of
>>>> pre-processing, e.g. with a python script. But a pandoc solution would
>>>> be neat.
>>>>
>>>> Is the content of the bibliography file exposed to the filter
>>>> architecture before citeproc actually processes the bibliography?
>>>>
>>>> Best,
>>>> Denis
>>>>
>>>> -- 
>>>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/54c0b3a9-f742-18e4-9fc2-8a3b7f2f3632%40mailbox.org.
>> -- 
>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/9af493a5-336d-a726-4266-b89104f0cebc%40mailbox.org.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/a5b830a3-fed0-1024-cf80-d878b1514bb9%40mailbox.org.

Re: Manipulate CSL bibliography before processing

[Denis Maier]

Am 18.09.2020 um 08:32 schrieb Denis Maier:
> [...] I don't know if there's an easy way to enable merging of 
> metadata blocks?
As often, googling first is a good idea: There seems to be a tool that 
is designed for exactly that purpose: https://mikefarah.gitbook.io/yq/

So, that would mean I can use Albert's suggestion. Something like this 
should work:
pandoc input.md -o output.pdf --metadata-file <(yd options file1 file2) 
-citeproc ...

> Am 17.09.2020 um 23:37 schrieb John MacFarlane:
>> Our library does allow you to define citation-label for a
>> reference in your CSL JSON or pandoc YAML bibliography.
>> (If you don't define it, and a label is called for, then
>> the library will generate one in a manner similar to
>> citeproc.js.  But you CAN define it explicitly, and in that
>> case the defined version will be used.)  I think that solves
>> your problem all by itself.
>>
>> Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>>
>>> That's related, but not the same. --citation-abbreviatons is currently
>>> used to supply short forms of certain fields. E.g container-titles, and
>>> so. Also, they match against strings. The citation-label works similar
>>> to biblatex's shorthand, but there is currently no way to define 
>>> them on
>>> a per-document basis.
>>> But that could of course be added to the --citation-abbreviations 
>>> feature.
>>>
>>> Am 17.09.2020 um 19:59 schrieb John MacFarlane:
>>>> I wonder whether you could use --citation-abbreviations for this?
>>>> See pandoc-citeproc manual.
>>>>
>>>> Denis Maier <denis.maier.lists-cl+VPiYnx/1AfugRpC6u6w@public.gmane.org> writes:
>>>>
>>>>> Hi,
>>>>>
>>>>> is it possible to manipulate a CSL bibliography before Pandoc 
>>>>> processes
>>>>> the citations?
>>>>> Let's say I have one huge database, and I want to do ad hoc
>>>>> modifications, for example add a certain field to a couple of 
>>>>> entries.
>>>>>
>>>>> The specific use case would be this:
>>>>> You write an essay about Kant, and you want to use the usual
>>>>> abbreviations (KrV, KpV, etc.), but you don't want to have those
>>>>> abbreviations in your main database. In  your metadata you could have
>>>>> something like this:
>>>>>
>>>>> ```
>>>>> bibliography: bib.json
>>>>> bibliography-modifications:
>>>>>      - id: kant-id1
>>>>>        add:
>>>>>          - field: citation-label
>>>>>            content: KrV
>>>>> ```
>>>>>
>>>>> Of course, you can always implement this with some sort of
>>>>> pre-processing, e.g. with a python script. But a pandoc solution 
>>>>> would
>>>>> be neat.
>>>>>
>>>>> Is the content of the bibliography file exposed to the filter
>>>>> architecture before citeproc actually processes the bibliography?
>>>>>
>>>>> Best,
>>>>> Denis
>>>>>
>>>>> -- 
>>>>> You received this message because you are subscribed to the Google 
>>>>> Groups "pandoc-discuss" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, 
>>>>> send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>>>> To view this discussion on the web visit 
>>>>> https://groups.google.com/d/msgid/pandoc-discuss/54c0b3a9-f742-18e4-9fc2-8a3b7f2f3632%40mailbox.org.
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "pandoc-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, 
>>> send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/pandoc-discuss/9af493a5-336d-a726-4266-b89104f0cebc%40mailbox.org.
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/3d4526b7-d83f-41ba-5b77-a9969261632c%40mailbox.org.