latex template syntax vs Lua filters

latex template syntax vs Lua filters

[bapt a]

[-- Attachment #1.1: Type: text/plain, Size: 2699 bytes --]

Hi,

I'm curious to understand the status of / motivations for the 
"mini-language" described in

https://pandoc.org/MANUAL.html#template-syntax

It has become quite complex, with partials, pipes, etc. presumably in 
response to ever-increasing needs to fine-tune the string processing and 
formatting in LaTeX templates. A rather intricate use-case I recently came 
across is to process a list of author-affiliations from Metadata into the 
format requested by a particular journal – for instance:

https://github.com/quarto-journals/acs/blob/main/_extensions/acs/title.tex#L1

to follow instructions along those lines in the latex class:

% repeat the \author .. \affiliation etc. as needed
% \email, \thanks, \homepage, \altaffiliation all apply to the current
% author. Explanatory text should go in the []'s, actual e-mail
% address or url should go in the {}'s for \email and \homepage.
% Please use the appropriate macro foreach each type of information

% \affiliation command applies to all authors since the last
% \affiliation command. The \affiliation command should follow the
% other information
% \affiliation can be followed by \email, \homepage, \thanks as well.

I'm a bit puzzled by the design choice in pandoc templates – would it not 
be (much) easier to process variables in Lua, a fully fledged language with 
built-in support for string manipulation, loops, etc.?

I can imagine a partial "custom writer" of sorts that would be in charge of 
producing, say, the formatted author-affiliation for a given format (latex, 
html, etc.), by manipulating the AST in Lua. This function would be called 
to insert the required content into the TeX file. 

In fact, this can be done right now by adding a generic 
`$insert-author-affiliation$` placeholder in the latex template, and using 
a Lua filter to generate the corresponding data in the AST.

Clearly I'm missing something – there must be a good reason for the 
development of the mini-language for templates, beyond the simple scenarios 
(basic for/if constructs). I could see the value of performing simple tasks 
directly in the target latex template, but when it gets as complex as the 
example above I don't quite follow.

Many thanks,

baptiste



-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/d890f12c-84d1-4ab8-a836-0da8fb9c8caan%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3348 bytes --]

Re: latex template syntax vs Lua filters

[John MacFarlane]

[-- Attachment #1: Type: text/plain, Size: 4247 bytes --]

Partly we’ve been motivated by backwards compatibility — adding new features but not breaking the original template syntax, which goes back 16 years or so.

Why not Lua?  In addition to backwards compatibility, you might have concerns about safety.  Right now you can be confident that using a template won’t launch the missiles, delete a file, or cause an infinite loop.

> In fact, this can be done right now by adding a generic `$insert-author-affiliation$` placeholder in the latex template, and using a Lua filter to generate the corresponding data in the AST.


That’s a good approach!

Another approach would be to use a custom writer that calls the regular pandoc writer but uses its own templating system, written in Lua or whatever you like.



> On Apr 3, 2023, at 2:49 PM, bapt a <auguieba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> Hi,
> 
> I'm curious to understand the status of / motivations for the "mini-language" described in
> 
> https://pandoc.org/MANUAL.html#template-syntax
> 
> It has become quite complex, with partials, pipes, etc. presumably in response to ever-increasing needs to fine-tune the string processing and formatting in LaTeX templates. A rather intricate use-case I recently came across is to process a list of author-affiliations from Metadata into the format requested by a particular journal – for instance:
> 
> https://github.com/quarto-journals/acs/blob/main/_extensions/acs/title.tex#L1
> 
> to follow instructions along those lines in the latex class:
> 
> % repeat the \author .. \affiliation etc. as needed
> % \email, \thanks, \homepage, \altaffiliation all apply to the current
> % author. Explanatory text should go in the []'s, actual e-mail
> % address or url should go in the {}'s for \email and \homepage.
> % Please use the appropriate macro foreach each type of information
> 
> % \affiliation command applies to all authors since the last
> % \affiliation command. The \affiliation command should follow the
> % other information
> % \affiliation can be followed by \email, \homepage, \thanks as well.
> 
> I'm a bit puzzled by the design choice in pandoc templates – would it not be (much) easier to process variables in Lua, a fully fledged language with built-in support for string manipulation, loops, etc.?
> 
> I can imagine a partial "custom writer" of sorts that would be in charge of producing, say, the formatted author-affiliation for a given format (latex, html, etc.), by manipulating the AST in Lua. This function would be called to insert the required content into the TeX file. 
> 
> In fact, this can be done right now by adding a generic `$insert-author-affiliation$` placeholder in the latex template, and using a Lua filter to generate the corresponding data in the AST.
> 
> Clearly I'm missing something – there must be a good reason for the development of the mini-language for templates, beyond the simple scenarios (basic for/if constructs). I could see the value of performing simple tasks directly in the target latex template, but when it gets as complex as the example above I don't quite follow.
> 
> Many thanks,
> 
> baptiste
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/d890f12c-84d1-4ab8-a836-0da8fb9c8caan%40googlegroups.com <https://groups.google.com/d/msgid/pandoc-discuss/d890f12c-84d1-4ab8-a836-0da8fb9c8caan%40googlegroups.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/BCE2FA1F-296D-4C7E-9495-E4B6E1C5A885%40gmail.com.

[-- Attachment #2: Type: text/html, Size: 6435 bytes --]

Re: latex template syntax vs Lua filters

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 5668 bytes --]

Also remember that a Lua (or any other programming language) program is
processing instructions with embedded data/variables/text (strings) while a
template is a text with variables and possibly processing instructions
(loops, logic) embedded, i.e. templates are text-centric in a way programs
are not. A program doing the same job as a template will be full of string
concatenations and format strings[^1] which will look disconnected in a way
which template text will not. This was the original use case for templates
before security and separation of functions were big concerns, and it is
still valid.

[^1]: A format string can of course be thought of as a (low-power) template.

Den tis 4 apr. 2023 02:49John MacFarlane <fiddlosopher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> skrev:

> Partly we’ve been motivated by backwards compatibility — adding new
> features but not breaking the original template syntax, which goes back 16
> years or so.
>
> Why not Lua?  In addition to backwards compatibility, you might have
> concerns about safety.  Right now you can be confident that using a
> template won’t launch the missiles, delete a file, or cause an infinite
> loop.
>
> In fact, this can be done right now by adding a generic
> `$insert-author-affiliation$` placeholder in the latex template, and using
> a Lua filter to generate the corresponding data in the AST.
>
>
> That’s a good approach!
>
> Another approach would be to use a custom writer that calls the regular
> pandoc writer but uses its own templating system, written in Lua or
> whatever you like.
>
>
>
> On Apr 3, 2023, at 2:49 PM, bapt a <auguieba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> Hi,
>
> I'm curious to understand the status of / motivations for the
> "mini-language" described in
>
> https://pandoc.org/MANUAL.html#template-syntax
>
> It has become quite complex, with partials, pipes, etc. presumably in
> response to ever-increasing needs to fine-tune the string processing and
> formatting in LaTeX templates. A rather intricate use-case I recently came
> across is to process a list of author-affiliations from Metadata into the
> format requested by a particular journal – for instance:
>
>
> https://github.com/quarto-journals/acs/blob/main/_extensions/acs/title.tex#L1
>
> to follow instructions along those lines in the latex class:
>
> % repeat the \author .. \affiliation etc. as needed
> % \email, \thanks, \homepage, \altaffiliation all apply to the current
> % author. Explanatory text should go in the []'s, actual e-mail
> % address or url should go in the {}'s for \email and \homepage.
> % Please use the appropriate macro foreach each type of information
>
> % \affiliation command applies to all authors since the last
> % \affiliation command. The \affiliation command should follow the
> % other information
> % \affiliation can be followed by \email, \homepage, \thanks as well.
>
> I'm a bit puzzled by the design choice in pandoc templates – would it not
> be (much) easier to process variables in Lua, a fully fledged language with
> built-in support for string manipulation, loops, etc.?
>
> I can imagine a partial "custom writer" of sorts that would be in charge
> of producing, say, the formatted author-affiliation for a given format
> (latex, html, etc.), by manipulating the AST in Lua. This function would be
> called to insert the required content into the TeX file.
>
> In fact, this can be done right now by adding a generic
> `$insert-author-affiliation$` placeholder in the latex template, and using
> a Lua filter to generate the corresponding data in the AST.
>
> Clearly I'm missing something – there must be a good reason for the
> development of the mini-language for templates, beyond the simple scenarios
> (basic for/if constructs). I could see the value of performing simple tasks
> directly in the target latex template, but when it gets as complex as the
> example above I don't quite follow.
>
> Many thanks,
>
> baptiste
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/d890f12c-84d1-4ab8-a836-0da8fb9c8caan%40googlegroups.com
> <https://groups.google.com/d/msgid/pandoc-discuss/d890f12c-84d1-4ab8-a836-0da8fb9c8caan%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/BCE2FA1F-296D-4C7E-9495-E4B6E1C5A885%40gmail.com
> <https://groups.google.com/d/msgid/pandoc-discuss/BCE2FA1F-296D-4C7E-9495-E4B6E1C5A885%40gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhBkxMhLAeCTOymphguqG_uHwpUyAnYC4edi%3DSaBkmHS6g%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 7503 bytes --]

Re: latex template syntax vs Lua filters

[Albert Krewinkel]

BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> A program doing the same job as a template will be full of string
> concatenations and format strings which will look disconnected in
> a way which template text will not. This was the original use case for
> templates before security and separation of functions were big
> concerns, and it is still valid.

My favorite example to reinforce this point is PHP: it's already really
easy to use PHP in HTML templates, and yet, people wrote extra template
languages in PHP (e.g., https://smarty.net) because those were more
pleasant to use.


-- 
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: latex template syntax vs Lua filters

[bapt a]

[-- Attachment #1.1: Type: text/plain, Size: 2761 bytes --]

Thank you both for the additional perspective.

I can see a trade-off in the relative merits of a general purpose language 
(e..g Lua) vs mini-DSL (templates), but for me the general language would 
win as soon as there's a non-trivial amount of manipulations to perform. Of 
course this will be a matter of personal taste more than anything (until 
something turns out to be impossible, and the DSL needs to be augmented). A 
key advantage of the programming language approach, in my view, is that any 
complexity can be captured inside a function. 

Specifically, I'm looking at the heavy pre-processing done in Quarto[^1] to 
thoroughly normalise the author-affiliation metadata, including tasks such 
as identifying unique affiliations (maybeAddAffiliation() is an example of 
wrapping non-trivial operations in a convenient function), grouping authors 
by unique affiliations with unique IDs, etc. Doing all this without the 
facilities afforded by Lua would seem unsurmountable to me. And then, if 
one has already gone 99% of the way in Lua by producing the right AST 
structure, it seems easiest to me to carry on with Lua and have a custom 
partial writer or filter insert the final \author{}...\affiliation{} 
strings in the output. 
Admittedly, this is a rather extreme situation, where a lot of 
manipulations need to happen between the relatively free-form metadata and 
the output.

Best regards,

baptiste


[^1] 
https://github.com/quarto-dev/quarto-cli/blob/ad4281d71317c6a491434a080a408de2aff789a4/src/resources/filters/common/authors.lua#L346




On Wednesday, 5 April 2023 at 04:00:18 UTC+12 Albert Krewinkel wrote:

>
> BPJ <mel...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
> > A program doing the same job as a template will be full of string
> > concatenations and format strings which will look disconnected in
> > a way which template text will not. This was the original use case for
> > templates before security and separation of functions were big
> > concerns, and it is still valid.
>
> My favorite example to reinforce this point is PHP: it's already really
> easy to use PHP in HTML templates, and yet, people wrote extra template
> languages in PHP (e.g., https://smarty.net) because those were more
> pleasant to use.
>
>
> -- 
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe e836 388d c0b2 1f63 1124
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/1251527d-ba92-406f-a3e2-3009190253d3n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3838 bytes --]