Possible bugs in the markdown to PDF via lualatex conversion process for bilingual documents

Possible bugs in the markdown to PDF via lualatex conversion process for bilingual documents

[R (Chandra) Chandrasekhar]

[-- Attachment #1: Type: text/plain, Size: 3098 bytes --]

I am writing a bilingual test file for English and Tamil in the same 
markdown document to yield a PDF via lualatex and babel. The file, which 
correctly gives HTML and epub outputs,  is:

```
---
title: "Bilingual Test for LuaLaTeX and Babel: English and Tamil"
lang: en-GB
---

This is English.

{\sffamily This is also English.}

இது தமிழ்.

{\sffamily இதுவும் தமிழ்.}

:::{lang=ta}
இது கூட தமிழ் தான்.
:::

Back to English and EOF.
```

The included `babel-in-header.tex` file is:

```
% babel-in-header.textsf
\babelfont{rm}[Scale=1.0,Language=Default,BoldFont={* SemiBold}]{Noto Serif}
\babelfont{sf}[Language=Default,BoldFont={* SemiBold}]{Noto Sans}
\babelfont[tamil]{rm}[Language=Default]{Noto Serif Tamil}
\babelfont[tamil]{sf}[Language=Default]{Noto Sans Tamil}
\babelprovide[import,onchar=ids fonts]{tamil}
\defaultfontfeatures{Scale=MatchLowercase,Ligatures=TeX,Renderer=HarfBuzz}
\thispagestyle{empty}
```

The compilation command is:

pandoc --from markdown --to latex --include-in-header 
babel-in-header.tex --standalone --output en-ta-test.tex en-ta-test.md

I find that the generated .tex file cannot be compiled to PDF because 
there appear to be two errors, which might be bugs:

1. The \babelfont commands appear _before_ the \usepackage{babel} 
invocation in the .tex file because of a commented requirement for 
polyglossia, which I am not using. As a result, I get an

```
! Undefined control sequence.
l.59 \babelfont
              {rm}[Scale=1.0,Language=Default,BoldFont={* 
SemiBold}]{Noto Serif}
```

error with lualatex.

2. If I manually transpose the \babelfont commands inserted by the 
`babel-in-header.tex` file to appear _after_ the babel invocation but 
before \title in the .tex file, I get this error:

```
! Package babel Error: Unknown option 'tamil'. Either you misspelled it
(babel)                or the language definition file tamil.ldf was not 
found.


See the babel package documentation for explanation.
```

3. If I remove `tamil,` from the babel invocation,

```
\usepackage[tamil,main=british]{babel}
```

I am able to compile with lualatex, using the same command as before, 
and get the expected PDF, as shown in the attached screenshot.

I realize that both pandoc and the babel package are under development, 
and are therefore moving targets.

Nevertheless, I believe these are bugs in the markdown to latex 
conversion process and point them out so that they may be corrected or 
circumvented, automatically rather than manually, from the pandoc end.

Thanks.

Chandra

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/b865ce6f-aa23-a1f0-b5f7-35601791d215%40gmail.com.

[-- Attachment #2: bilingual.png --]
[-- Type: image/png, Size: 35255 bytes --]

Re: Possible bugs in the markdown to PDF via lualatex conversion process for bilingual documents

[John MacFarlane]

Let us know what version of pandoc you're using.
There have been extensive changes recently that might be
relevant (we no longer use polyglossia, but of course
it's possible that something relating to our old polyglossia
support was left in the template...).

"R (Chandra) Chandrasekhar" <chyavana-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> I am writing a bilingual test file for English and Tamil in the same 
> markdown document to yield a PDF via lualatex and babel. The file, which 
> correctly gives HTML and epub outputs,  is:
>
> ```
> ---
> title: "Bilingual Test for LuaLaTeX and Babel: English and Tamil"
> lang: en-GB
> ---
>
> This is English.
>
> {\sffamily This is also English.}
>
> இது தமிழ்.
>
> {\sffamily இதுவும் தமிழ்.}
>
> :::{lang=ta}
> இது கூட தமிழ் தான்.
> :::
>
> Back to English and EOF.
> ```
>
> The included `babel-in-header.tex` file is:
>
> ```
> % babel-in-header.textsf
> \babelfont{rm}[Scale=1.0,Language=Default,BoldFont={* SemiBold}]{Noto Serif}
> \babelfont{sf}[Language=Default,BoldFont={* SemiBold}]{Noto Sans}
> \babelfont[tamil]{rm}[Language=Default]{Noto Serif Tamil}
> \babelfont[tamil]{sf}[Language=Default]{Noto Sans Tamil}
> \babelprovide[import,onchar=ids fonts]{tamil}
> \defaultfontfeatures{Scale=MatchLowercase,Ligatures=TeX,Renderer=HarfBuzz}
> \thispagestyle{empty}
> ```
>
> The compilation command is:
>
> pandoc --from markdown --to latex --include-in-header 
> babel-in-header.tex --standalone --output en-ta-test.tex en-ta-test.md
>
> I find that the generated .tex file cannot be compiled to PDF because 
> there appear to be two errors, which might be bugs:
>
> 1. The \babelfont commands appear _before_ the \usepackage{babel} 
> invocation in the .tex file because of a commented requirement for 
> polyglossia, which I am not using. As a result, I get an
>
> ```
> ! Undefined control sequence.
> l.59 \babelfont
>               {rm}[Scale=1.0,Language=Default,BoldFont={* 
> SemiBold}]{Noto Serif}
> ```
>
> error with lualatex.
>
> 2. If I manually transpose the \babelfont commands inserted by the 
> `babel-in-header.tex` file to appear _after_ the babel invocation but 
> before \title in the .tex file, I get this error:
>
> ```
> ! Package babel Error: Unknown option 'tamil'. Either you misspelled it
> (babel)                or the language definition file tamil.ldf was not 
> found.
>
>
> See the babel package documentation for explanation.
> ```
>
> 3. If I remove `tamil,` from the babel invocation,
>
> ```
> \usepackage[tamil,main=british]{babel}
> ```
>
> I am able to compile with lualatex, using the same command as before, 
> and get the expected PDF, as shown in the attached screenshot.
>
> I realize that both pandoc and the babel package are under development, 
> and are therefore moving targets.
>
> Nevertheless, I believe these are bugs in the markdown to latex 
> conversion process and point them out so that they may be corrected or 
> circumvented, automatically rather than manually, from the pandoc end.
>
> Thanks.
>
> Chandra
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/b865ce6f-aa23-a1f0-b5f7-35601791d215%40gmail.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/yh480kbkzhm4a2.fsf%40johnmacfarlane.net.

Re: Possible bugs in the markdown to PDF via lualatex conversion process for bilingual documents

[Clément Aubert]

[-- Attachment #1.1: Type: text/plain, Size: 5861 bytes --]

Hi,

I may have found a similar bug, but find it *very hard* to understand its 
exact causes.

In a rather long document, I have the following options in my YAML:

---
documentclass: scrbook
mainfont: texgyretermes
mainfontoptions:
- Path=fonts/
- Extension=.otf
- UprightFont=*-regular
- BoldFont=*-bold
- ItalicFont=*-italic
- BoldItalicFont=*-BoldItalic
fontsize: 12pt
---

Now, some combination of LaTeX packages (or something else?) that I load 
makes pandoc output, in the .tex document

\ifLuaTeX
\usepackage[bidi=basic]{babel}
\else
\usepackage[bidi=default]{babel}
\fi
\babelprovide[main,import]{english}
\ifPDFTeX
\else
\babelfont[english]{rm}{texgyretermes}
\fi

When I compile said document with XeLaTeX, I get

kpathsea: Running mktextfm TeXGyreTermes/OT


but that font is not found, and hence I get the error


Package fontspec Error: The font "TeXGyreTermes" cannot be found. 
\babel@aux{english}{}


If I replace 


\babelfont[english]{rm}{texgyretermes}


with


\babelfont[english]{rm}{fonts/texgyretermes-regular.otf}

then the document can be compiled.

Is this a bug?

I am using the latest version,
pandoc 3.0.1
Features: +server +lua
Scripting engine: Lua 5.4
User data directory: /home/caubert/.local/share/pandoc
Copyright (C) 2006-2023 John MacFarlane. Web:  https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.

Best,
And thanks,
Clément.
Le mardi 8 février 2022 à 16:35:00 UTC-5, John MacFarlane a écrit :

>
> Let us know what version of pandoc you're using.
> There have been extensive changes recently that might be
> relevant (we no longer use polyglossia, but of course
> it's possible that something relating to our old polyglossia
> support was left in the template...).
>
> "R (Chandra) Chandrasekhar" <chya...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
> > I am writing a bilingual test file for English and Tamil in the same 
> > markdown document to yield a PDF via lualatex and babel. The file, which 
> > correctly gives HTML and epub outputs, is:
> >
> > ```
> > ---
> > title: "Bilingual Test for LuaLaTeX and Babel: English and Tamil"
> > lang: en-GB
> > ---
> >
> > This is English.
> >
> > {\sffamily This is also English.}
> >
> > இது தமிழ்.
> >
> > {\sffamily இதுவும் தமிழ்.}
> >
> > :::{lang=ta}
> > இது கூட தமிழ் தான்.
> > :::
> >
> > Back to English and EOF.
> > ```
> >
> > The included `babel-in-header.tex` file is:
> >
> > ```
> > % babel-in-header.textsf
> > \babelfont{rm}[Scale=1.0,Language=Default,BoldFont={* SemiBold}]{Noto 
> Serif}
> > \babelfont{sf}[Language=Default,BoldFont={* SemiBold}]{Noto Sans}
> > \babelfont[tamil]{rm}[Language=Default]{Noto Serif Tamil}
> > \babelfont[tamil]{sf}[Language=Default]{Noto Sans Tamil}
> > \babelprovide[import,onchar=ids fonts]{tamil}
> > 
> \defaultfontfeatures{Scale=MatchLowercase,Ligatures=TeX,Renderer=HarfBuzz}
> > \thispagestyle{empty}
> > ```
> >
> > The compilation command is:
> >
> > pandoc --from markdown --to latex --include-in-header 
> > babel-in-header.tex --standalone --output en-ta-test.tex en-ta-test.md
> >
> > I find that the generated .tex file cannot be compiled to PDF because 
> > there appear to be two errors, which might be bugs:
> >
> > 1. The \babelfont commands appear _before_ the \usepackage{babel} 
> > invocation in the .tex file because of a commented requirement for 
> > polyglossia, which I am not using. As a result, I get an
> >
> > ```
> > ! Undefined control sequence.
> > l.59 \babelfont
> > {rm}[Scale=1.0,Language=Default,BoldFont={* 
> > SemiBold}]{Noto Serif}
> > ```
> >
> > error with lualatex.
> >
> > 2. If I manually transpose the \babelfont commands inserted by the 
> > `babel-in-header.tex` file to appear _after_ the babel invocation but 
> > before \title in the .tex file, I get this error:
> >
> > ```
> > ! Package babel Error: Unknown option 'tamil'. Either you misspelled it
> > (babel) or the language definition file tamil.ldf was not 
> > found.
> >
> >
> > See the babel package documentation for explanation.
> > ```
> >
> > 3. If I remove `tamil,` from the babel invocation,
> >
> > ```
> > \usepackage[tamil,main=british]{babel}
> > ```
> >
> > I am able to compile with lualatex, using the same command as before, 
> > and get the expected PDF, as shown in the attached screenshot.
> >
> > I realize that both pandoc and the babel package are under development, 
> > and are therefore moving targets.
> >
> > Nevertheless, I believe these are bugs in the markdown to latex 
> > conversion process and point them out so that they may be corrected or 
> > circumvented, automatically rather than manually, from the pandoc end.
> >
> > Thanks.
> >
> > Chandra
> >
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "pandoc-discuss" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pandoc-discuss/b865ce6f-aa23-a1f0-b5f7-35601791d215%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/017f0e7b-dd57-452b-95b4-f89c655d39ben%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 7869 bytes --]

Re: Possible bugs in the markdown to PDF via lualatex conversion process for bilingual documents

[Clément Aubert]

[-- Attachment #1.1: Type: text/plain, Size: 6562 bytes --]

Ok, we could also use 

\babelfont[english]{rm}[Path=fonts/,Extension=.otf,UprightFont=*-regular,BoldFont=*-bold,ItalicFont=*-italic,BoldItalicFont=*-BoldItalic]{texgyretermes}

instead of 

\babelfont[english]{rm}{fonts/texgyretermes-regular.otf}

This could make everything easier…
Le jeudi 9 février 2023 à 12:27:53 UTC-5, Clément Aubert a écrit :

> Hi,
>
> I may have found a similar bug, but find it *very hard* to understand its 
> exact causes.
>
> In a rather long document, I have the following options in my YAML:
>
> ---
> documentclass: scrbook
> mainfont: texgyretermes
> mainfontoptions:
> - Path=fonts/
> - Extension=.otf
> - UprightFont=*-regular
> - BoldFont=*-bold
> - ItalicFont=*-italic
> - BoldItalicFont=*-BoldItalic
> fontsize: 12pt
> ---
>
> Now, some combination of LaTeX packages (or something else?) that I load 
> makes pandoc output, in the .tex document
>
> \ifLuaTeX
> \usepackage[bidi=basic]{babel}
> \else
> \usepackage[bidi=default]{babel}
> \fi
> \babelprovide[main,import]{english}
> \ifPDFTeX
> \else
> \babelfont[english]{rm}{texgyretermes}
> \fi
>
> When I compile said document with XeLaTeX, I get
>
> kpathsea: Running mktextfm TeXGyreTermes/OT
>
>
> but that font is not found, and hence I get the error
>
>
> Package fontspec Error: The font "TeXGyreTermes" cannot be found. 
> \babel@aux{english}{}
>
>
> If I replace 
>
>
> \babelfont[english]{rm}{texgyretermes}
>
>
> with
>
>
> \babelfont[english]{rm}{fonts/texgyretermes-regular.otf}
>
> then the document can be compiled.
>
> Is this a bug?
>
> I am using the latest version,
> pandoc 3.0.1
> Features: +server +lua
> Scripting engine: Lua 5.4
> User data directory: /home/caubert/.local/share/pandoc
> Copyright (C) 2006-2023 John MacFarlane. Web:  https://pandoc.org
> This is free software; see the source for copying conditions. There is no
> warranty, not even for merchantability or fitness for a particular purpose.
>
> Best,
> And thanks,
> Clément.
> Le mardi 8 février 2022 à 16:35:00 UTC-5, John MacFarlane a écrit :
>
>>
>> Let us know what version of pandoc you're using. 
>> There have been extensive changes recently that might be 
>> relevant (we no longer use polyglossia, but of course 
>> it's possible that something relating to our old polyglossia 
>> support was left in the template...). 
>>
>> "R (Chandra) Chandrasekhar" <chya...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes: 
>>
>> > I am writing a bilingual test file for English and Tamil in the same 
>> > markdown document to yield a PDF via lualatex and babel. The file, 
>> which 
>> > correctly gives HTML and epub outputs, is: 
>> > 
>> > ``` 
>> > --- 
>> > title: "Bilingual Test for LuaLaTeX and Babel: English and Tamil" 
>> > lang: en-GB 
>> > --- 
>> > 
>> > This is English. 
>> > 
>> > {\sffamily This is also English.} 
>> > 
>> > இது தமிழ். 
>> > 
>> > {\sffamily இதுவும் தமிழ்.} 
>> > 
>> > :::{lang=ta} 
>> > இது கூட தமிழ் தான். 
>> > ::: 
>> > 
>> > Back to English and EOF. 
>> > ``` 
>> > 
>> > The included `babel-in-header.tex` file is: 
>> > 
>> > ``` 
>> > % babel-in-header.textsf 
>> > \babelfont{rm}[Scale=1.0,Language=Default,BoldFont={* SemiBold}]{Noto 
>> Serif} 
>> > \babelfont{sf}[Language=Default,BoldFont={* SemiBold}]{Noto Sans} 
>> > \babelfont[tamil]{rm}[Language=Default]{Noto Serif Tamil} 
>> > \babelfont[tamil]{sf}[Language=Default]{Noto Sans Tamil} 
>> > \babelprovide[import,onchar=ids fonts]{tamil} 
>> > 
>> \defaultfontfeatures{Scale=MatchLowercase,Ligatures=TeX,Renderer=HarfBuzz} 
>> > \thispagestyle{empty} 
>> > ``` 
>> > 
>> > The compilation command is: 
>> > 
>> > pandoc --from markdown --to latex --include-in-header 
>> > babel-in-header.tex --standalone --output en-ta-test.tex en-ta-test.md 
>> > 
>> > I find that the generated .tex file cannot be compiled to PDF because 
>> > there appear to be two errors, which might be bugs: 
>> > 
>> > 1. The \babelfont commands appear _before_ the \usepackage{babel} 
>> > invocation in the .tex file because of a commented requirement for 
>> > polyglossia, which I am not using. As a result, I get an 
>> > 
>> > ``` 
>> > ! Undefined control sequence. 
>> > l.59 \babelfont 
>> > {rm}[Scale=1.0,Language=Default,BoldFont={* 
>> > SemiBold}]{Noto Serif} 
>> > ``` 
>> > 
>> > error with lualatex. 
>> > 
>> > 2. If I manually transpose the \babelfont commands inserted by the 
>> > `babel-in-header.tex` file to appear _after_ the babel invocation but 
>> > before \title in the .tex file, I get this error: 
>> > 
>> > ``` 
>> > ! Package babel Error: Unknown option 'tamil'. Either you misspelled it 
>> > (babel) or the language definition file tamil.ldf was not 
>> > found. 
>> > 
>> > 
>> > See the babel package documentation for explanation. 
>> > ``` 
>> > 
>> > 3. If I remove `tamil,` from the babel invocation, 
>> > 
>> > ``` 
>> > \usepackage[tamil,main=british]{babel} 
>> > ``` 
>> > 
>> > I am able to compile with lualatex, using the same command as before, 
>> > and get the expected PDF, as shown in the attached screenshot. 
>> > 
>> > I realize that both pandoc and the babel package are under development, 
>> > and are therefore moving targets. 
>> > 
>> > Nevertheless, I believe these are bugs in the markdown to latex 
>> > conversion process and point them out so that they may be corrected or 
>> > circumvented, automatically rather than manually, from the pandoc end. 
>> > 
>> > Thanks. 
>> > 
>> > Chandra 
>> > 
>> > -- 
>> > You received this message because you are subscribed to the Google 
>> Groups "pandoc-discuss" group. 
>> > To unsubscribe from this group and stop receiving emails from it, send 
>> an email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org 
>> > To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pandoc-discuss/b865ce6f-aa23-a1f0-b5f7-35601791d215%40gmail.com. 
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/d2d6d890-37d5-4309-b792-a833de39cc5fn%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 8647 bytes --]