Lua filters and possible error

Lua filters and possible error

[Apprentice Ship]

[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]

I am trying to understand a bit more about Lua filters, in particular
"spellcheck" in the repository.

   https://github.com/pandoc/lua-filters/tree/master/spellcheck

and my understanding is that the filters will act on the AST, but a
sequence of:

$ pandoc sample.md -s -o sample.tex

$ pandoc --lua-filter spellcheck.lua sample.md
summer
missspeling [en]

$ pandoc --lua-filter spellcheck.lua sample.tex
des
français
missspeling
english
sont
Ces
Mais

seems to show that the Pandoc (ver 3.1.9) is not understanding the
language metadata, as used by the LaTeX-writer. In fact, I do not see the
the main-language metadata entry under either of the commands:

     pandoc -t native sample.md
     pandoc -t native sample.tex

Is that what is going on here?

Joshua

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAH114nCcs0R1-ayRGf5W%2BR7VAAMLAYY-L62MvSK4AtWoqON%3Dtw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 1901 bytes --]

Re: Lua filters and possible error

[John MacFarlane]

Do you have something like this in sample.md?

---
lang: fr-FR
...


> On Nov 18, 2023, at 3:39 PM, Apprentice Ship <tex.apprentice-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> I am trying to understand a bit more about Lua filters, in particular "spellcheck" in the repository.
> 
>    https://github.com/pandoc/lua-filters/tree/master/spellcheck
> 
> and my understanding is that the filters will act on the AST, but a sequence of:
> 
> $ pandoc sample.md -s -o sample.tex
> 
> $ pandoc --lua-filter spellcheck.lua sample.md
> summer
> missspeling [en]
> 
> $ pandoc --lua-filter spellcheck.lua sample.tex
> des
> français
> missspeling
> english
> sont
> Ces
> Mais
> 
> seems to show that the Pandoc (ver 3.1.9) is not understanding the language metadata, as used by the LaTeX-writer. In fact, I do not see the the main-language metadata entry under either of the commands:
> 
>      pandoc -t native sample.md
>      pandoc -t native sample.tex
> 
> Is that what is going on here? 
> 
> Joshua
> 
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAH114nCcs0R1-ayRGf5W%2BR7VAAMLAYY-L62MvSK4AtWoqON%3Dtw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/69D4EAA3-A22A-47BA-ACD5-3F694B47CEAB%40gmail.com.

Re: Lua filters and possible error

[Bastien DUMONT]

If I understand the question correctly, it has to do with the fact that the LaTeX reader doesn't read the line \babelprovide[main,import]{french} from the file produced by `pandoc sample.md -s -o sample.tex`, as the following command evidences:

$ pandoc -t latex -s <<< '
---
lang: fr-FR
date: 01/01/2001
---

Mot
' | pandoc -f latex -t native -s

Le Sunday 19 November 2023 à 11:02:45AM, John MacFarlane a écrit :
> Do you have something like this in sample.md?
> 
> ---
> lang: fr-FR
> ...
> 
> 
> > On Nov 18, 2023, at 3:39 PM, Apprentice Ship <tex.apprentice-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > 
> > I am trying to understand a bit more about Lua filters, in particular "spellcheck" in the repository.
> > 
> >    https://github.com/pandoc/lua-filters/tree/master/spellcheck
> > 
> > and my understanding is that the filters will act on the AST, but a sequence of:
> > 
> > $ pandoc sample.md -s -o sample.tex
> > 
> > $ pandoc --lua-filter spellcheck.lua sample.md
> > summer
> > missspeling [en]
> > 
> > $ pandoc --lua-filter spellcheck.lua sample.tex
> > des
> > français
> > missspeling
> > english
> > sont
> > Ces
> > Mais
> > 
> > seems to show that the Pandoc (ver 3.1.9) is not understanding the language metadata, as used by the LaTeX-writer. In fact, I do not see the the main-language metadata entry under either of the commands:
> > 
> >      pandoc -t native sample.md
> >      pandoc -t native sample.tex
> > 
> > Is that what is going on here? 
> > 
> > Joshua
> > 
> > 
> > 
> > 
> > 
> > -- 
> > You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> > To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAH114nCcs0R1-ayRGf5W%2BR7VAAMLAYY-L62MvSK4AtWoqON%3Dtw%40mail.gmail.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/69D4EAA3-A22A-47BA-ACD5-3F694B47CEAB%40gmail.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ZVpfFntx258OC54W%40localhost.

Re: Lua filters and possible error

[John MacFarlane]

Ah yes, I understand now.  Sorry for my irrelevant reply.

You can work around this by providing the metadata on the command line:

$ pandoc --lua-filter spellcheck.lua sample.tex --metadata lang=fr


> On Nov 19, 2023, at 11:16 AM, Bastien DUMONT <bastien.dumont-VwIFZPTo/vqsTnJN9+BGXg@public.gmane.org> wrote:
> 
> If I understand the question correctly, it has to do with the fact that the LaTeX reader doesn't read the line \babelprovide[main,import]{french} from the file produced by `pandoc sample.md -s -o sample.tex`, as the following command evidences:
> 
> $ pandoc -t latex -s <<< '
> ---
> lang: fr-FR
> date: 01/01/2001
> ---
> 
> Mot
> ' | pandoc -f latex -t native -s
> 
> Le Sunday 19 November 2023 à 11:02:45AM, John MacFarlane a écrit :
>> Do you have something like this in sample.md?
>> 
>> ---
>> lang: fr-FR
>> ...
>> 
>> 
>>> On Nov 18, 2023, at 3:39 PM, Apprentice Ship <tex.apprentice-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> 
>>> I am trying to understand a bit more about Lua filters, in particular "spellcheck" in the repository.
>>> 
>>>   https://github.com/pandoc/lua-filters/tree/master/spellcheck
>>> 
>>> and my understanding is that the filters will act on the AST, but a sequence of:
>>> 
>>> $ pandoc sample.md -s -o sample.tex
>>> 
>>> $ pandoc --lua-filter spellcheck.lua sample.md
>>> summer
>>> missspeling [en]
>>> 
>>> $ pandoc --lua-filter spellcheck.lua sample.tex
>>> des
>>> français
>>> missspeling
>>> english
>>> sont
>>> Ces
>>> Mais
>>> 
>>> seems to show that the Pandoc (ver 3.1.9) is not understanding the language metadata, as used by the LaTeX-writer. In fact, I do not see the the main-language metadata entry under either of the commands:
>>> 
>>>     pandoc -t native sample.md
>>>     pandoc -t native sample.tex
>>> 
>>> Is that what is going on here? 
>>> 
>>> Joshua
>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAH114nCcs0R1-ayRGf5W%2BR7VAAMLAYY-L62MvSK4AtWoqON%3Dtw%40mail.gmail.com.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/69D4EAA3-A22A-47BA-ACD5-3F694B47CEAB%40gmail.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ZVpfFntx258OC54W%40localhost.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/F948E13C-92F3-45B2-987E-9A5652BDBD58%40gmail.com.

Re: Lua filters and possible error

[Apprentice Ship]

[-- Attachment #1: Type: text/plain, Size: 4819 bytes --]

Thanks John, that works a bit better, it does recognize the main text as
being in French, but there are still some problems:

It did recognize

\foreignlanguage{english}{This is a sentence in English, with one
missspeling.}


as being English, but does not recognize:

\begin{otherlanguage}{english}
Here's a div in English. Code is ignored: \texttt{baoeuthasoe}. So are
\href{http://example.com/notaword}{URLs}.
\end{otherlanguage}


as being English.

Joshua





On Sun, Nov 19, 2023 at 12:14 PM John MacFarlane <fiddlosopher@gmail.com>
wrote:

> Ah yes, I understand now.  Sorry for my irrelevant reply.
>
> You can work around this by providing the metadata on the command line:
>
> $ pandoc --lua-filter spellcheck.lua sample.tex --metadata lang=fr
>
>
> > On Nov 19, 2023, at 11:16 AM, Bastien DUMONT <bastien.dumont-VwIFZPTo/vqsTnJN9+BGXg@public.gmane.org>
> wrote:
> >
> > If I understand the question correctly, it has to do with the fact that
> the LaTeX reader doesn't read the line \babelprovide[main,import]{french}
> from the file produced by `pandoc sample.md -s -o sample.tex`, as the
> following command evidences:
> >
> > $ pandoc -t latex -s <<< '
> > ---
> > lang: fr-FR
> > date: 01/01/2001
> > ---
> >
> > Mot
> > ' | pandoc -f latex -t native -s
> >
> > Le Sunday 19 November 2023 à 11:02:45AM, John MacFarlane a écrit :
> >> Do you have something like this in sample.md?
> >>
> >> ---
> >> lang: fr-FR
> >> ...
> >>
> >>
> >>> On Nov 18, 2023, at 3:39 PM, Apprentice Ship <tex.apprentice-Re5JQEeQqe8@public.gmane.orgm>
> wrote:
> >>>
> >>> I am trying to understand a bit more about Lua filters, in particular
> "spellcheck" in the repository.
> >>>
> >>>   https://github.com/pandoc/lua-filters/tree/master/spellcheck
> >>>
> >>> and my understanding is that the filters will act on the AST, but a
> sequence of:
> >>>
> >>> $ pandoc sample.md -s -o sample.tex
> >>>
> >>> $ pandoc --lua-filter spellcheck.lua sample.md
> >>> summer
> >>> missspeling [en]
> >>>
> >>> $ pandoc --lua-filter spellcheck.lua sample.tex
> >>> des
> >>> français
> >>> missspeling
> >>> english
> >>> sont
> >>> Ces
> >>> Mais
> >>>
> >>> seems to show that the Pandoc (ver 3.1.9) is not understanding the
> language metadata, as used by the LaTeX-writer. In fact, I do not see the
> the main-language metadata entry under either of the commands:
> >>>
> >>>     pandoc -t native sample.md
> >>>     pandoc -t native sample.tex
> >>>
> >>> Is that what is going on here?
> >>>
> >>> Joshua
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> Groups "pandoc-discuss" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send
> an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> >>> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/CAH114nCcs0R1-ayRGf5W%2BR7VAAMLAYY-L62MvSK4AtWoqON%3Dtw%40mail.gmail.com
> .
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups "pandoc-discuss" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> >> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/69D4EAA3-A22A-47BA-ACD5-3F694B47CEAB%40gmail.com
> .
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "pandoc-discuss" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/ZVpfFntx258OC54W%40localhost
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/F948E13C-92F3-45B2-987E-9A5652BDBD58%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAH114nD0y-fygRpqFHQsUCc3ORh5d_-G5SpJPDn-5OQFcqFYBw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 7784 bytes --]

Re: Lua filters and possible error

[John MacFarlane]

\begin{otherlanguage}{english}
Here's a div in English. Code is ignored: \texttt{baoeuthasoe}. So are
\href{http://example.com/notaword}{URLs}.
\end{otherlanguage}

is being parsed as

[ Div
    ( "" , [ "otherlanguage" ] , [] )
 [ Para
        [ Span ( "" , [] , [] ) [ Str "english" ]
        , SoftBreak
...

That could definitely be improved; we can teach pandoc that the otherlanguage
environment has an argument for the language which should be put in the lang
attribute of the div. Once that is done I think the filter will work.
 
I added an issue for this: https://github.com/jgm/pandoc/issues/9202


> On Nov 19, 2023, at 4:45 PM, Apprentice Ship <tex.apprentice-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> 
> \begin{otherlanguage}{english}
> Here's a div in English. Code is ignored: \texttt{baoeuthasoe}. So are
> \href{http://example.com/notaword}{URLs}.
> \end{otherlanguage}

Re: Lua filters and possible error

[Paulo Ney de Souza]

[-- Attachment #1: Type: text/plain, Size: 2636 bytes --]

I think we will need a bit more. The space of Language Tags in LaTeX is
all over the place, Babel and Polyglossia alike.

Things like:

\foreignlanguage{farsi}
\begin{otherlanguage}{farsi} ...\end{otherlanguage}
\textpersian{}
\begin{persian} ... \end{persian}

will all have to be translated to the "almost" BCP-47 Language Tags used by
Aspell,
which in this case is:

        fa

I say "almost" because Aspell uses an underscore for dictionaries like
"en_GB" instead
of the correct "en-GB", for British English.

The Polyglossia package already has this equivalence
(Babel--Polyglossia--BCP-47) in
a table. We should use it. It is not easy to write it down.

Paulo Ney



On Sun, Nov 19, 2023 at 6:35 PM John MacFarlane <fiddlosopher@gmail.com>
wrote:

>
> \begin{otherlanguage}{english}
> Here's a div in English. Code is ignored: \texttt{baoeuthasoe}. So are
> \href{http://example.com/notaword}{URLs}.
> \end{otherlanguage}
>
> is being parsed as
>
> [ Div
>     ( "" , [ "otherlanguage" ] , [] )
>  [ Para
>         [ Span ( "" , [] , [] ) [ Str "english" ]
>         , SoftBreak
> ...
>
> That could definitely be improved; we can teach pandoc that the
> otherlanguage
> environment has an argument for the language which should be put in the
> lang
> attribute of the div. Once that is done I think the filter will work.
>
> I added an issue for this: https://github.com/jgm/pandoc/issues/9202
>
>
> > On Nov 19, 2023, at 4:45 PM, Apprentice Ship <tex.apprentice-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> wrote:
> >
> >
> > \begin{otherlanguage}{english}
> > Here's a div in English. Code is ignored: \texttt{baoeuthasoe}. So are
> > \href{http://example.com/notaword}{URLs}.
> > \end{otherlanguage}
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/8B6C889E-AE79-464D-A28C-6AADC2D28FB0%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAFVhNZOqQoFuEXJgnHvp7HLwE3wpmAfs7S%2BPCGLwe0dG2JBEfA%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 4272 bytes --]

Re: Lua filters and possible error

[John MacFarlane]

Comment on the issue or it will be lost...

> On Nov 19, 2023, at 8:00 PM, Paulo Ney de Souza <pauloney-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> I think we will need a bit more. The space of Language Tags in LaTeX is
> all over the place, Babel and Polyglossia alike. 
> 
> Things like:
> 
> \foreignlanguage{farsi}
> \begin{otherlanguage}{farsi} ...\end{otherlanguage}
> \textpersian{}
> \begin{persian} ... \end{persian}
> 
> will all have to be translated to the "almost" BCP-47 Language Tags used by Aspell,
> which in this case is:
> 
>         fa
> 
> I say "almost" because Aspell uses an underscore for dictionaries like "en_GB" instead
> of the correct "en-GB", for British English.
> 
> The Polyglossia package already has this equivalence (Babel--Polyglossia--BCP-47) in 
> a table. We should use it. It is not easy to write it down.
> 
> Paulo Ney
> 
> 
> 
> On Sun, Nov 19, 2023 at 6:35 PM John MacFarlane <fiddlosopher@gmail.com> wrote:
> 
> \begin{otherlanguage}{english}
> Here's a div in English. Code is ignored: \texttt{baoeuthasoe}. So are
> \href{http://example.com/notaword}{URLs}.
> \end{otherlanguage}
> 
> is being parsed as
> 
> [ Div
>     ( "" , [ "otherlanguage" ] , [] )
>  [ Para
>         [ Span ( "" , [] , [] ) [ Str "english" ]
>         , SoftBreak
> ...
> 
> That could definitely be improved; we can teach pandoc that the otherlanguage
> environment has an argument for the language which should be put in the lang
> attribute of the div. Once that is done I think the filter will work.
> 
> I added an issue for this: https://github.com/jgm/pandoc/issues/9202
> 
> 
> > On Nov 19, 2023, at 4:45 PM, Apprentice Ship <tex.apprentice-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > 
> > 
> > \begin{otherlanguage}{english}
> > Here's a div in English. Code is ignored: \texttt{baoeuthasoe}. So are
> > \href{http://example.com/notaword}{URLs}.
> > \end{otherlanguage}
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/8B6C889E-AE79-464D-A28C-6AADC2D28FB0%40gmail.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAFVhNZOqQoFuEXJgnHvp7HLwE3wpmAfs7S%2BPCGLwe0dG2JBEfA%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/718912E3-D2A0-4493-BA19-C0EBF4CE1A09%40gmail.com.