Moving footnotes to just after paragraph with the citation

Moving footnotes to just after paragraph with the citation

[Lyndon Drake]

[-- Attachment #1.1: Type: text/plain, Size: 1317 bytes --]

Hi,

I'm working with a long pandoc-flavoured Markdown file generated by 
Scrivener. All the footnotes appear at the end of the document. I'd like to 
be able to process the Markdown file to shift each footnote to immediately 
follow the paragraph which contains the footnote reference. I don't want to 
inline the footnotes, because some of the footnotes have multiple 
paragraphs.

So I would like to transform something like this:

```
This is the first paragraph.[^1]

This is the second paragraph.[^2]

[^1]: First footnote.

     The first footnote has two paragraphs.

[^2]: Second footnote.
````

To something like this:

```
This is the first paragraph.[^1]

[^1]: First footnote.

     The first footnote has two paragraphs.

This is the second paragraph.[^2]

[^2]: Second footnote.
````

Is there a straightforward way to do this? E.g. as a pandoc lua filter or 
custom writer?

Thanks,
Lyndon

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/868187f2-1fd4-4009-b48e-2a948749b518n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2037 bytes --]

Re: Moving footnotes to just after paragraph with the citation

[Gwern Branwen]

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

This seems possibly tricky? I don't know what the Lua filters can or cannot
do, but I'm not sure how one could solve this using just the Haskell API...
Pandoc was not designed as a source-2-source processor so it doesn't make
much effort to support Markdown-to-Markdown passes - like the obvious
roundtrip will potentially change your Markdown formatting to a
semantically-equivalent but entirely different-looking Markdown doc.
(Macfarlane has said repeatedly that this will not be changed.)

In your case, footnotes throw away the positioning information.
Consider the 3 possibilities:

    Foo[^foo].

    Foo bar^[Bar.]

    Foo baz[^baz]

    [^baz]: Baz.

    [^foo]: foo.

This parses into:

[ Para
    [ Str "Foo" , Note [ Para [ Str "foo." ] ] , Str "." ]
, Para
    [ Str "Foo"
    , Space
    , Str "bar"
    , Note [ Para [ Str "Bar." ] ]
    ]
, Para
    [ Str "Foo"
    , Space
    , Str "baz"
    , Note [ Para [ Str "Baz." ] ]
    ]
]

So you can't know 'where' a footnote body was put, whether at the end of
the doc or elsewhere.

Further, Pandoc will write out the Markdown according to its own
conventions, which go opposite of what you want, by putting *all* footnote
bodies at the end, regardless of where they were or whether they could be
inline footnotes, and the Markdown writer probably takes no options which
would change that:

Foo[^1].

Foo bar[^2]

Foo baz[^3]

[^1]: foo.

[^2]: Bar.

[^3]: Baz.

-- 
gwern
https://gwern.net <https://www.gwern.net>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAMwO0gw1W6_CHiCr8RGfDkLjt-dTg4y9dPkbU2UouZt%3D1Lj3Cg%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 2836 bytes --]

Re: Moving footnotes to just after paragraph with the citation

[EBkysko]

[-- Attachment #1.1: Type: text/plain, Size: 2301 bytes --]

try (with markdown output 
<https://pandoc.org/MANUAL.html#options-affecting-specific-writers>)

```
--reference-location=block
``` 

On Tuesday, November 21, 2023 at 4:23:57 PM UTC-5 Gwern Branwen wrote:

> This seems possibly tricky? I don't know what the Lua filters can or 
> cannot do, but I'm not sure how one could solve this using just the Haskell 
> API... Pandoc was not designed as a source-2-source processor so it doesn't 
> make much effort to support Markdown-to-Markdown passes - like the obvious 
> roundtrip will potentially change your Markdown formatting to a 
> semantically-equivalent but entirely different-looking Markdown doc. 
> (Macfarlane has said repeatedly that this will not be changed.)
>
> In your case, footnotes throw away the positioning information.
> Consider the 3 possibilities:
>
>     Foo[^foo].
>
>     Foo bar^[Bar.]
>
>     Foo baz[^baz]
>
>     [^baz]: Baz.
>  
>     [^foo]: foo.
>
> This parses into:
>
> [ Para
>     [ Str "Foo" , Note [ Para [ Str "foo." ] ] , Str "." ]
> , Para
>     [ Str "Foo"
>     , Space
>     , Str "bar"
>     , Note [ Para [ Str "Bar." ] ]
>     ]
> , Para
>     [ Str "Foo"
>     , Space
>     , Str "baz"
>     , Note [ Para [ Str "Baz." ] ]
>     ]
> ]
>
> So you can't know 'where' a footnote body was put, whether at the end of 
> the doc or elsewhere.
>
> Further, Pandoc will write out the Markdown according to its own 
> conventions, which go opposite of what you want, by putting *all* footnote 
> bodies at the end, regardless of where they were or whether they could be 
> inline footnotes, and the Markdown writer probably takes no options which 
> would change that:
>
> Foo[^1].
>
> Foo bar[^2]
>
> Foo baz[^3]
>
> [^1]: foo.
>
> [^2]: Bar.
>
> [^3]: Baz.
>
> -- 
> gwern
> https://gwern.net <https://www.gwern.net>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/aa407627-870e-474e-856e-49559c74d46cn%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3541 bytes --]

Re: Moving footnotes to just after paragraph with the citation

[Lyndon Drake]

[-- Attachment #1.1: Type: text/plain, Size: 2518 bytes --]

Brilliant @EBkysko — that works exactly as I'd hoped. Thank you!

On Tuesday, 21 November 2023 at 19:11:24 UTC-6 EBkysko wrote:

> try (with markdown output 
> <https://pandoc.org/MANUAL.html#options-affecting-specific-writers>)
>
> ```
> --reference-location=block
> ``` 
>
> On Tuesday, November 21, 2023 at 4:23:57 PM UTC-5 Gwern Branwen wrote:
>
>> This seems possibly tricky? I don't know what the Lua filters can or 
>> cannot do, but I'm not sure how one could solve this using just the Haskell 
>> API... Pandoc was not designed as a source-2-source processor so it doesn't 
>> make much effort to support Markdown-to-Markdown passes - like the obvious 
>> roundtrip will potentially change your Markdown formatting to a 
>> semantically-equivalent but entirely different-looking Markdown doc. 
>> (Macfarlane has said repeatedly that this will not be changed.)
>>
>> In your case, footnotes throw away the positioning information.
>> Consider the 3 possibilities:
>>
>>     Foo[^foo].
>>
>>     Foo bar^[Bar.]
>>
>>     Foo baz[^baz]
>>
>>     [^baz]: Baz.
>>  
>>     [^foo]: foo.
>>
>> This parses into:
>>
>> [ Para
>>     [ Str "Foo" , Note [ Para [ Str "foo." ] ] , Str "." ]
>> , Para
>>     [ Str "Foo"
>>     , Space
>>     , Str "bar"
>>     , Note [ Para [ Str "Bar." ] ]
>>     ]
>> , Para
>>     [ Str "Foo"
>>     , Space
>>     , Str "baz"
>>     , Note [ Para [ Str "Baz." ] ]
>>     ]
>> ]
>>
>> So you can't know 'where' a footnote body was put, whether at the end of 
>> the doc or elsewhere.
>>
>> Further, Pandoc will write out the Markdown according to its own 
>> conventions, which go opposite of what you want, by putting *all* footnote 
>> bodies at the end, regardless of where they were or whether they could be 
>> inline footnotes, and the Markdown writer probably takes no options which 
>> would change that:
>>
>> Foo[^1].
>>
>> Foo bar[^2]
>>
>> Foo baz[^3]
>>
>> [^1]: foo.
>>
>> [^2]: Bar.
>>
>> [^3]: Baz.
>>
>> -- 
>> gwern
>> https://gwern.net <https://www.gwern.net>
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/3b95f72e-34ba-486b-9b5d-2b0d01a49088n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 4122 bytes --]

Re: Moving footnotes to just after paragraph with the citation

[EBkysko]

[-- Attachment #1.1: Type: text/plain, Size: 3237 bytes --]

Note there's a caveat to this solution: as the guide mentions, with 'block' 
the notes are placed at the end of the current **top-level** block, which 
might not be the paragraph itself.

Examples:

   - If the paragraph is an item in a list, the note is placed below the 
   list, not the item (and below main list if a nested list).
   - If the paragraph is within a fenced div, the note is placed after the 
   fenced div.
   - A corollary to the above: if a filter uses `make_sections`, the notes 
   might end up at end of document after all.


On Tuesday, November 21, 2023 at 9:55:26 PM UTC-5 Lyndon Drake wrote:

> Brilliant @EBkysko — that works exactly as I'd hoped. Thank you!
>
> On Tuesday, 21 November 2023 at 19:11:24 UTC-6 EBkysko wrote:
>
>> try (with markdown output 
>> <https://pandoc.org/MANUAL.html#options-affecting-specific-writers>)
>>
>> ```
>> --reference-location=block
>> ``` 
>>
>> On Tuesday, November 21, 2023 at 4:23:57 PM UTC-5 Gwern Branwen wrote:
>>
>>> This seems possibly tricky? I don't know what the Lua filters can or 
>>> cannot do, but I'm not sure how one could solve this using just the Haskell 
>>> API... Pandoc was not designed as a source-2-source processor so it doesn't 
>>> make much effort to support Markdown-to-Markdown passes - like the obvious 
>>> roundtrip will potentially change your Markdown formatting to a 
>>> semantically-equivalent but entirely different-looking Markdown doc. 
>>> (Macfarlane has said repeatedly that this will not be changed.)
>>>
>>> In your case, footnotes throw away the positioning information.
>>> Consider the 3 possibilities:
>>>
>>>     Foo[^foo].
>>>
>>>     Foo bar^[Bar.]
>>>
>>>     Foo baz[^baz]
>>>
>>>     [^baz]: Baz.
>>>  
>>>     [^foo]: foo.
>>>
>>> This parses into:
>>>
>>> [ Para
>>>     [ Str "Foo" , Note [ Para [ Str "foo." ] ] , Str "." ]
>>> , Para
>>>     [ Str "Foo"
>>>     , Space
>>>     , Str "bar"
>>>     , Note [ Para [ Str "Bar." ] ]
>>>     ]
>>> , Para
>>>     [ Str "Foo"
>>>     , Space
>>>     , Str "baz"
>>>     , Note [ Para [ Str "Baz." ] ]
>>>     ]
>>> ]
>>>
>>> So you can't know 'where' a footnote body was put, whether at the end of 
>>> the doc or elsewhere.
>>>
>>> Further, Pandoc will write out the Markdown according to its own 
>>> conventions, which go opposite of what you want, by putting *all* footnote 
>>> bodies at the end, regardless of where they were or whether they could be 
>>> inline footnotes, and the Markdown writer probably takes no options which 
>>> would change that:
>>>
>>> Foo[^1].
>>>
>>> Foo bar[^2]
>>>
>>> Foo baz[^3]
>>>
>>> [^1]: foo.
>>>
>>> [^2]: Bar.
>>>
>>> [^3]: Baz.
>>>
>>> -- 
>>> gwern
>>> https://gwern.net <https://www.gwern.net>
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ff6a6898-e1c0-413f-b6c2-d81ea2d949d4n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 4963 bytes --]

Re: Moving footnotes to just after paragraph with the citation

['William Lupton' via pandoc-discuss]

[-- Attachment #1: Type: text/plain, Size: 4196 bytes --]

If --reference-location=block turns out not to be sufficient, I don't think
a lua filter will help, but a custom writer could (the writer is
responsible for handling footnotes).

On Wed, 22 Nov 2023 at 12:23, EBkysko <ebkysko-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Note there's a caveat to this solution: as the guide mentions, with
> 'block' the notes are placed at the end of the current **top-level** block,
> which might not be the paragraph itself.
>
> Examples:
>
>    - If the paragraph is an item in a list, the note is placed below the
>    list, not the item (and below main list if a nested list).
>    - If the paragraph is within a fenced div, the note is placed after
>    the fenced div.
>    - A corollary to the above: if a filter uses `make_sections`, the
>    notes might end up at end of document after all.
>
>
> On Tuesday, November 21, 2023 at 9:55:26 PM UTC-5 Lyndon Drake wrote:
>
>> Brilliant @EBkysko — that works exactly as I'd hoped. Thank you!
>>
>> On Tuesday, 21 November 2023 at 19:11:24 UTC-6 EBkysko wrote:
>>
>>> try (with markdown output
>>> <https://pandoc.org/MANUAL.html#options-affecting-specific-writers>)
>>>
>>> ```
>>> --reference-location=block
>>> ```
>>>
>>> On Tuesday, November 21, 2023 at 4:23:57 PM UTC-5 Gwern Branwen wrote:
>>>
>>>> This seems possibly tricky? I don't know what the Lua filters can or
>>>> cannot do, but I'm not sure how one could solve this using just the Haskell
>>>> API... Pandoc was not designed as a source-2-source processor so it doesn't
>>>> make much effort to support Markdown-to-Markdown passes - like the obvious
>>>> roundtrip will potentially change your Markdown formatting to a
>>>> semantically-equivalent but entirely different-looking Markdown doc.
>>>> (Macfarlane has said repeatedly that this will not be changed.)
>>>>
>>>> In your case, footnotes throw away the positioning information.
>>>> Consider the 3 possibilities:
>>>>
>>>>     Foo[^foo].
>>>>
>>>>     Foo bar^[Bar.]
>>>>
>>>>     Foo baz[^baz]
>>>>
>>>>     [^baz]: Baz.
>>>>
>>>>     [^foo]: foo.
>>>>
>>>> This parses into:
>>>>
>>>> [ Para
>>>>     [ Str "Foo" , Note [ Para [ Str "foo." ] ] , Str "." ]
>>>> , Para
>>>>     [ Str "Foo"
>>>>     , Space
>>>>     , Str "bar"
>>>>     , Note [ Para [ Str "Bar." ] ]
>>>>     ]
>>>> , Para
>>>>     [ Str "Foo"
>>>>     , Space
>>>>     , Str "baz"
>>>>     , Note [ Para [ Str "Baz." ] ]
>>>>     ]
>>>> ]
>>>>
>>>> So you can't know 'where' a footnote body was put, whether at the end
>>>> of the doc or elsewhere.
>>>>
>>>> Further, Pandoc will write out the Markdown according to its own
>>>> conventions, which go opposite of what you want, by putting *all* footnote
>>>> bodies at the end, regardless of where they were or whether they could be
>>>> inline footnotes, and the Markdown writer probably takes no options which
>>>> would change that:
>>>>
>>>> Foo[^1].
>>>>
>>>> Foo bar[^2]
>>>>
>>>> Foo baz[^3]
>>>>
>>>> [^1]: foo.
>>>>
>>>> [^2]: Bar.
>>>>
>>>> [^3]: Baz.
>>>>
>>>> --
>>>> gwern
>>>> https://gwern.net <https://www.gwern.net>
>>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/ff6a6898-e1c0-413f-b6c2-d81ea2d949d4n%40googlegroups.com
> <https://groups.google.com/d/msgid/pandoc-discuss/ff6a6898-e1c0-413f-b6c2-d81ea2d949d4n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAEe_xxh74qC5fLmzhoMjS1gyWwyo%3DrSKVmbXTs--fH8me-EAeg%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 5932 bytes --]

Re: Moving footnotes to just after paragraph with the citation

[Lyndon Drake]

[-- Attachment #1.1: Type: text/plain, Size: 4538 bytes --]

I think a custom writer is a bit beyond my capability at the moment. I can 
see that you can do some very cool stuff with custom writers though!

On Wednesday, 22 November 2023 at 04:54:46 UTC-8 William Lupton wrote:

> If --reference-location=block turns out not to be sufficient, I don't 
> think a lua filter will help, but a custom writer could (the writer is 
> responsible for handling footnotes).
>
> On Wed, 22 Nov 2023 at 12:23, EBkysko <ebk...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>> Note there's a caveat to this solution: as the guide mentions, with 
>> 'block' the notes are placed at the end of the current **top-level** block, 
>> which might not be the paragraph itself.
>>
>> Examples:
>>
>>    - If the paragraph is an item in a list, the note is placed below the 
>>    list, not the item (and below main list if a nested list).
>>    - If the paragraph is within a fenced div, the note is placed after 
>>    the fenced div.
>>    - A corollary to the above: if a filter uses `make_sections`, the 
>>    notes might end up at end of document after all.
>>
>>
>> On Tuesday, November 21, 2023 at 9:55:26 PM UTC-5 Lyndon Drake wrote:
>>
>>> Brilliant @EBkysko — that works exactly as I'd hoped. Thank you!
>>>
>>> On Tuesday, 21 November 2023 at 19:11:24 UTC-6 EBkysko wrote:
>>>
>>>> try (with markdown output 
>>>> <https://pandoc.org/MANUAL.html#options-affecting-specific-writers>)
>>>>
>>>> ```
>>>> --reference-location=block
>>>> ``` 
>>>>
>>>> On Tuesday, November 21, 2023 at 4:23:57 PM UTC-5 Gwern Branwen wrote:
>>>>
>>>>> This seems possibly tricky? I don't know what the Lua filters can or 
>>>>> cannot do, but I'm not sure how one could solve this using just the Haskell 
>>>>> API... Pandoc was not designed as a source-2-source processor so it doesn't 
>>>>> make much effort to support Markdown-to-Markdown passes - like the obvious 
>>>>> roundtrip will potentially change your Markdown formatting to a 
>>>>> semantically-equivalent but entirely different-looking Markdown doc. 
>>>>> (Macfarlane has said repeatedly that this will not be changed.)
>>>>>
>>>>> In your case, footnotes throw away the positioning information.
>>>>> Consider the 3 possibilities:
>>>>>
>>>>>     Foo[^foo].
>>>>>
>>>>>     Foo bar^[Bar.]
>>>>>
>>>>>     Foo baz[^baz]
>>>>>
>>>>>     [^baz]: Baz.
>>>>>  
>>>>>     [^foo]: foo.
>>>>>
>>>>> This parses into:
>>>>>
>>>>> [ Para
>>>>>     [ Str "Foo" , Note [ Para [ Str "foo." ] ] , Str "." ]
>>>>> , Para
>>>>>     [ Str "Foo"
>>>>>     , Space
>>>>>     , Str "bar"
>>>>>     , Note [ Para [ Str "Bar." ] ]
>>>>>     ]
>>>>> , Para
>>>>>     [ Str "Foo"
>>>>>     , Space
>>>>>     , Str "baz"
>>>>>     , Note [ Para [ Str "Baz." ] ]
>>>>>     ]
>>>>> ]
>>>>>
>>>>> So you can't know 'where' a footnote body was put, whether at the end 
>>>>> of the doc or elsewhere.
>>>>>
>>>>> Further, Pandoc will write out the Markdown according to its own 
>>>>> conventions, which go opposite of what you want, by putting *all* footnote 
>>>>> bodies at the end, regardless of where they were or whether they could be 
>>>>> inline footnotes, and the Markdown writer probably takes no options which 
>>>>> would change that:
>>>>>
>>>>> Foo[^1].
>>>>>
>>>>> Foo bar[^2]
>>>>>
>>>>> Foo baz[^3]
>>>>>
>>>>> [^1]: foo.
>>>>>
>>>>> [^2]: Bar.
>>>>>
>>>>> [^3]: Baz.
>>>>>
>>>>> -- 
>>>>> gwern
>>>>> https://gwern.net <https://www.gwern.net>
>>>>>
>>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pandoc-discuss/ff6a6898-e1c0-413f-b6c2-d81ea2d949d4n%40googlegroups.com 
>> <https://groups.google.com/d/msgid/pandoc-discuss/ff6a6898-e1c0-413f-b6c2-d81ea2d949d4n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/94b41d5e-7edd-4176-ba72-1f7f5ab697fcn%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 7150 bytes --]