User story: typst multiline math in pandoc markdown and pandoc AST

User story: typst multiline math in pandoc markdown and pandoc AST

[Guillaume Dehaene]

[-- Attachment #1.1: Type: text/plain, Size: 3177 bytes --]

Hello everyone,

Here is a "user story" in which I would like to discuss how math is handled 
by pandoc AST and pandoc markdown.
I haven't been able to find existing discussion on this topic. I hope I'm 
not retreading something that has been discussed before.
I'm sorry for any issues with clarity: this is my first time submitting 
something like this to a floss project.

I write this as a mathematician, somewhat familiar with latex but 
frustrated by its opacity and how ugly .tex files look.
I thus want to edit documents in a lighter markup while still having the 
following latex features:
- math shortcodes: to simplify source code and for faster typing
- multi-line equations
- cross-references for a given equation line
- cross-references for text blocks, for theorems
I'd like to author documents in HTML, since it is the standard output 
format for the web, and PDF, if possible via latex, for back-compatibility 
with editors.

Pandoc provides a solution for almost all of these. The only issue is 
multi-line equations.
A work-around I am aware of is to just use raw latex code like:
```latex
\begin{align}
1 + 1 &= 2 \label{first_line} \\
1 + 2 &= 3  \label{second_line}
\end{align}
```
which works fine when exporting to both HTML and latex: it is added as is 
to the HTML where it is parsed by both mathjax and katex.

However, I propose that formatting equations like typst might be a good way 
to make multiline math native to Pandoc, instead of piggy-backing on raw 
latex.

Multiline equations could be written as in latex, wrapped in `$$ $$`:
```
$$
1 + 1 &= 2 \\
1 + 2 &= 3 
$$
```

which could be parsed as a new DisplayMathMultiline type in the AST.
(This is a mockup: I really don't understand how the AST is written)
```
[ Math DisplayMathMultiline
    [
        Math DisplayMathLine ["1+1", "=2"]
        Math DisplayMathLine ["1+2", "=3"]
    ]
]
```

This type can be written to:
- HTML and latex: as an `align` block
- typst: as a multiline math block
- pandoc markdown: as a multiline math block
- other: as multiple successive displayMath blocks

I do not see how modifying the AST in this way would produce an issue: it's 
an additive feature as far as I can tell. It can always default down to 
multiple displayMath blocks
For pandoc markdown, this makes the characters `&` and `\\` take on special 
meaning in math mode. I'm not familiar enough with math in markdown to know 
whether that could cause issues. If the behavior is implemented as an 
extension that the user can enable/disable, then this remains backwards 
compatible. It's also possible to use other characters, but this would mean 
that the syntax differs from latex.

Do you believe that this is an interesting feature to have for pandoc?

Best regards
Guillaume

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/672a5917-b04d-4eb9-966d-75b54df97837n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3780 bytes --]

Re: User story: typst multiline math in pandoc markdown and pandoc AST

[John MacFarlane]

> Pandoc provides a solution for almost all of these. The only issue is multi-line equations.
> A work-around I am aware of is to just use raw latex code like:
> ```latex
> \begin{align}
> 1 + 1 &= 2 \label{first_line} \\
> 1 + 2 &= 3  \label{second_line}
> \end{align}
> ```
> which works fine when exporting to both HTML and latex: it is added as is to the HTML where it is parsed by both mathjax and katex.

Note that there's another way to do this that will create a native Math element in pandoc:

$$
\begin{aligned}
1 + 1 &= 2 \\
1 + 2 &= 3
\end{aligned}
$$


> However, I propose that formatting equations like typst might be a good way to make multiline math native to Pandoc, instead of piggy-backing on raw latex.
> 
> Multiline equations could be written as in latex, wrapped in `$$ $$`:
> ```
> $$
> 1 + 1 &= 2 \\
> 1 + 2 &= 3 
> $$
> ```
> 
> which could be parsed as a new DisplayMathMultiline type in the AST.
> (This is a mockup: I really don't understand how the AST is written)
> ```
> [ Math DisplayMathMultiline
>     [
>         Math DisplayMathLine ["1+1", "=2"]
>         Math DisplayMathLine ["1+2", "=3"]
>     ]
> ]
> ```
> 
> This type can be written to:
> - HTML and latex: as an `align` block
> - typst: as a multiline math block
> - pandoc markdown: as a multiline math block
> - other: as multiple successive displayMath blocks
> 
> I do not see how modifying the AST in this way would produce an issue: it's an additive feature as far as I can tell.

Any change to our algebraic data types is a breaking change that would require modifications throughout the ecosystem (all writers and readers, filters, etc.).  That's why we're very conservative about such changes and make them infrequently and only with very strong reasons.

I suppose one possibility would be to parse this syntax into

\begin{aligned}
1 + 1 &= 2 \\
1 + 2 &= 3
\end{aligned}

which should work with all the formats you mention...

You could  even create a filter that does this, without changes to pandoc itself.


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/E89C2FED-79B7-4150-B015-D11C177D3F9B%40gmail.com.

Re: User story: typst multiline math in pandoc markdown and pandoc AST

[Guillaume Dehaene]

[-- Attachment #1.1: Type: text/plain, Size: 3637 bytes --]


Thank you for your insight.

1. I'm not 100% satisfied with your solution since it does not work with 
latex environments which open math-mode (such as a align). These produce 
errors because of the double-opening of math-mode. It's also pretty 
cumbersome compared to usual md.

I see that this is how pandoc parses typst equations (I should have looked!)
[ Math
DisplayMath "\\begin{aligned}\n1 + 1 & = 2 \\\\\n1 + 1 & = 
2\n\\end{aligned}" ]
Is that now going to be the accepted format for multiline equations?

2. I had overlooked the possibility of writing a filter.
The principle of the filter would then be:

parse DisplayMath
if there are & and/or \\ present, modify the block
depending on writer:
- latex, HTML: transform to align
- everything else: transform to accepted format(??) \begin{aligned}

Is that correct? If so, I'll start hacking at it in the near future.

Best regards,
Guillaume
On Thursday, October 19, 2023 at 6:11:41 PM UTC+2 John MacFarlane wrote:

> > Pandoc provides a solution for almost all of these. The only issue is 
> multi-line equations.
> > A work-around I am aware of is to just use raw latex code like:
> > ```latex
> > \begin{align}
> > 1 + 1 &= 2 \label{first_line} \\
> > 1 + 2 &= 3 \label{second_line}
> > \end{align}
> > ```
> > which works fine when exporting to both HTML and latex: it is added as 
> is to the HTML where it is parsed by both mathjax and katex.
>
> Note that there's another way to do this that will create a native Math 
> element in pandoc:
>
> $$
> \begin{aligned}
> 1 + 1 &= 2 \\
> 1 + 2 &= 3
> \end{aligned}
> $$
>
>
> > However, I propose that formatting equations like typst might be a good 
> way to make multiline math native to Pandoc, instead of piggy-backing on 
> raw latex.
> > 
> > Multiline equations could be written as in latex, wrapped in `$$ $$`:
> > ```
> > $$
> > 1 + 1 &= 2 \\
> > 1 + 2 &= 3 
> > $$
> > ```
> > 
> > which could be parsed as a new DisplayMathMultiline type in the AST.
> > (This is a mockup: I really don't understand how the AST is written)
> > ```
> > [ Math DisplayMathMultiline
> > [
> > Math DisplayMathLine ["1+1", "=2"]
> > Math DisplayMathLine ["1+2", "=3"]
> > ]
> > ]
> > ```
> > 
> > This type can be written to:
> > - HTML and latex: as an `align` block
> > - typst: as a multiline math block
> > - pandoc markdown: as a multiline math block
> > - other: as multiple successive displayMath blocks
> > 
> > I do not see how modifying the AST in this way would produce an issue: 
> it's an additive feature as far as I can tell.
>
> Any change to our algebraic data types is a breaking change that would 
> require modifications throughout the ecosystem (all writers and readers, 
> filters, etc.). That's why we're very conservative about such changes and 
> make them infrequently and only with very strong reasons.
>
> I suppose one possibility would be to parse this syntax into
>
> \begin{aligned}
> 1 + 1 &= 2 \\
> 1 + 2 &= 3
> \end{aligned}
>
> which should work with all the formats you mention...
>
> You could even create a filter that does this, without changes to pandoc 
> itself.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/daaf510b-50a3-4f9d-b7d5-58609d3821c6n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 4938 bytes --]

Re: User story: typst multiline math in pandoc markdown and pandoc AST

[John MacFarlane]

> On Oct 20, 2023, at 12:40 AM, Guillaume Dehaene <guillaume.dehaene-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> 
> Thank you for your insight.
> 
> 1. I'm not 100% satisfied with your solution since it does not work with latex environments which open math-mode (such as a align). These produce errors because of the double-opening of math-mode. It's also pretty cumbersome compared to usual md.
> 
> I see that this is how pandoc parses typst equations (I should have looked!)
>     [ Math
>         DisplayMath
>         "\\begin{aligned}\n1 + 1 & = 2 \\\\\n1 + 1 & = 2\n\\end{aligned}"
>     ]
> 
> Is that now going to be the accepted format for multiline equations?

I'm not sure I understand.  This has always been possible in pandoc (and LaTeX); we use it here because it's the way to get equivalent LaTeX math to the typst formula.


Side note:

in pandoc you can also do

$$
\begin{align}
x & = 5\\
y &= 3
\end{align}
$$

even though this isn't possible in LaTeX.  Pandoc will do the right thing the output format.  Try it.

> 2. I had overlooked the possibility of writing a filter.
> The principle of the filter would then be:
> 
> parse DisplayMath
> if there are & and/or \\ present, modify the block
> depending on writer:
> - latex, HTML: transform to align
> - everything else: transform to accepted format(??) \begin{aligned}
> 
> Is that correct? If so, I'll start hacking at it in the near future.

Yes, something like that.

Re: User story: typst multiline math in pandoc markdown and pandoc AST

[Gwern Branwen]

[-- Attachment #1: Type: text/plain, Size: 2281 bytes --]

On Thu, Oct 19, 2023 at 12:11 PM John MacFarlane <fiddlosopher@gmail.com>
wrote:

> Any change to our algebraic data types is a breaking change that would
> require modifications throughout the ecosystem (all writers and readers,
> filters, etc.).  That's why we're very conservative about such changes and
> make them infrequently and only with very strong reasons.
>

This expression-problem topic merits an FAQ entry at this point.

Dehaene's puzzlement here is a common & understandable one, because in many
languages there would be little problem, and it's intuitive: how could
merely tucking in an additional optional completely-backwards-compatible
field be such a big deal? Only to a hardcore static-typing Haskeller, I
think, is it immediately obvious that adding to the AST is a huge deal
which will break *everything*, almost, because the problem is not
backwards-compatibility, but forward-compatibility. It's an instance of the
'expression problem': the cost of static type safety of code operating on
the AST is that they are going to break that static type safety should the
type ever change.

You can tell it's common & understandable because it comes up several times
a year on this mailing list or the bugtracker, and shows no sign of going
away.

I think at this point it'd be good to have a short section somewhere on the
expression problem and why changes to the AST are not going to happen...
Not in https://pandoc.org/MANUAL.html , of course, as only extension
writers will run into this, but perhaps in
https://pandoc.org/using-the-pandoc-api.html ? (To explain why 'Yes, your
extension just ran into a problem where enriching the AST would help, but
that's not going to happen no matter how good it'd be, so don't bother
asking'.)

-- 
gwern
https://gwern.net <https://www.gwern.net>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAMwO0gyPC%3DeknmXkrAzGf%2Byfef1miXF4aFmJn4xG0atzkKJCzA%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 3254 bytes --]

Re: User story: typst multiline math in pandoc markdown and pandoc AST

[Guillaume Dehaene]

[-- Attachment #1.1: Type: text/plain, Size: 3630 bytes --]

Sorry for my lack of clarity. I'm asking too many questions at the same 
time and it must be hard to follow. I should have rewritten that last email 
before sending it.

1. My question about aligned.
I see in the code that the typst reader parses typst multilines into this 
DisplayMath begin{aligned} structure.
Is that how you / the pandoc team has decided to handle multiline math in 
the future?

2. You propose to wrap align.
This is not working on my side, but I might be doing something wrong?
Here are some minimal examples:
```
$$
\begin{align}
x & = 5\\
y &= 3
\end{align}
$$
```
parses into latex as (with pandoc test.md --to latex; on pandoc 3.1.8)
```
\[
\begin{align}
x & = 5\\
y &= 3
\end{align}
\]
```
which fails to compile to pdf (with pandoc test.md --to pdf -o test.pdf
```
Error producing PDF.
! Package amsmath Error: Erroneous nesting of equation structures;
(amsmath)                trying to recover with `aligned'.

See the amsmath package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.62 \end{align}
```

removing the $$
```
\begin{align}
x & = 5\\
y &= 3
\end{align}
```
parses as is into latex and works

Am I missing something?

3. I'm surprised at how easy it actually is to write down a filter.
This is a really great feature of pandoc

Thank you for all the great work on this project, and thank you for your 
help
Best
Guillaume

On Friday, October 20, 2023 at 7:27:05 PM UTC+2 John MacFarlane wrote:

>
>
> > On Oct 20, 2023, at 12:40 AM, Guillaume Dehaene <guillaum...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 
> wrote:
> > 
> > 
> > Thank you for your insight.
> > 
> > 1. I'm not 100% satisfied with your solution since it does not work with 
> latex environments which open math-mode (such as a align). These produce 
> errors because of the double-opening of math-mode. It's also pretty 
> cumbersome compared to usual md.
> > 
> > I see that this is how pandoc parses typst equations (I should have 
> looked!)
> > [ Math
> > DisplayMath
> > "\\begin{aligned}\n1 + 1 & = 2 \\\\\n1 + 1 & = 2\n\\end{aligned}"
> > ]
> > 
> > Is that now going to be the accepted format for multiline equations?
>
> I'm not sure I understand. This has always been possible in pandoc (and 
> LaTeX); we use it here because it's the way to get equivalent LaTeX math to 
> the typst formula.
>
>
> Side note:
>
> in pandoc you can also do
>
> $$
> \begin{align}
> x & = 5\\
> y &= 3
> \end{align}
> $$
>
> even though this isn't possible in LaTeX. Pandoc will do the right thing 
> the output format. Try it.
>
> > 2. I had overlooked the possibility of writing a filter.
> > The principle of the filter would then be:
> > 
> > parse DisplayMath
> > if there are & and/or \\ present, modify the block
> > depending on writer:
> > - latex, HTML: transform to align
> > - everything else: transform to accepted format(??) \begin{aligned}
> > 
> > Is that correct? If so, I'll start hacking at it in the near future.
>
> Yes, something like that.
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/e9222236-9514-4cdc-b986-ae8e7e198576n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 4946 bytes --]

Re: User story: typst multiline math in pandoc markdown and pandoc AST

[John MacFarlane]

[-- Attachment #1: Type: text/plain, Size: 1499 bytes --]



> On Oct 20, 2023, at 12:25 PM, Guillaume Dehaene <guillaume.dehaene-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> Sorry for my lack of clarity. I'm asking too many questions at the same time and it must be hard to follow. I should have rewritten that last email before sending it.
> 
> 1. My question about aligned.
> I see in the code that the typst reader parses typst multilines into this DisplayMath begin{aligned} structure.
> Is that how you / the pandoc team has decided to handle multiline math in the future?

What would be the alternative?  (Short of changing the AST, which is probably out of the question.)

> 2. You propose to wrap align.
> This is not working on my side, but I might be doing something wrong?
> Here are some minimal examples:
> ```
> $$
> \begin{align}
> x & = 5\\
> y &= 3
> \end{align}
> $$
> ```
> parses into latex as (with pandoc test.md --to latex; on pandoc 3.1.8)
> ```
> \[
> \begin{align}
> x & = 5\\
> y &= 3
> \end{align}
> \]
> ```
> which fails to compile to pdf (with pandoc test.md --to pdf -o test.pdf

You're right. My mistake on this.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/73813E01-A490-4C3E-8B23-8952A80487DD%40gmail.com.

[-- Attachment #2: Type: text/html, Size: 7215 bytes --]

Re: User story: typst multiline math in pandoc markdown and pandoc AST

[Guillaume Dehaene]

[-- Attachment #1.1: Type: text/plain, Size: 2123 bytes --]

oh I'm not sure there is an alternative.
I'm just curious whether the situation is: "we're using aligned to 
represent multiline-math always" or "we're using aligned to represent typst 
multilines".

I've almost finished my extension: it works perfectly.
I still need to finish the readme and stuff like that before I publish.
I'll update here once it's done: I'm sure I've missed some low hanging 
fruit to make it better.

Best regards,
Guillaume

On Saturday, October 21, 2023 at 12:00:58 AM UTC+2 John MacFarlane wrote:

>
> On Oct 20, 2023, at 12:25 PM, Guillaume Dehaene <guillaum...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 
> wrote:
>
> Sorry for my lack of clarity. I'm asking too many questions at the same 
> time and it must be hard to follow. I should have rewritten that last email 
> before sending it.
>
> 1. My question about aligned.
> I see in the code that the typst reader parses typst multilines into this 
> DisplayMath begin{aligned} structure.
> Is that how you / the pandoc team has decided to handle multiline math in 
> the future?
>
>
> What would be the alternative?  (Short of changing the AST, which is 
> probably out of the question.)
>
> 2. You propose to wrap align.
> This is not working on my side, but I might be doing something wrong?
> Here are some minimal examples:
> ```
> $$
> \begin{align}
> x & = 5\\
> y &= 3
> \end{align}
> $$
> ```
> parses into latex as (with pandoc test.md --to latex; on pandoc 3.1.8)
> ```
> \[
> \begin{align}
> x & = 5\\
> y &= 3
> \end{align}
> \]
> ```
> which fails to compile to pdf (with pandoc test.md --to pdf -o test.pdf
>
>
> You're right. My mistake on this.
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/9a81a7ad-5779-4166-805f-88c9400b8b96n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 6788 bytes --]