Open Discussion: Features of Templating

Open Discussion: Features of Templating

[Stephan Meijer]

[-- Attachment #1.1: Type: text/plain, Size: 1085 bytes --]



I would like to create an open discussion around this.
------------------------------

Hi all,

As I understand, the templating engine Pandoc uses is built-in. This is all 
good, but I feel like there are some missing features.

One feature I personally miss:
If-statement conditions

What I miss most, is being able to create conditions in if-statments. For 
example:

%if(pagetitle == "null")%
...
%endif%

Currently this seems not possible.

Other users that have noticed this:

   - 
   https://stackoverflow.com/questions/45404607/if-conditionals-in-pandoc-templates-depending-on-value-of-a-variable

------------------------------

Feel free to add your own points.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/dbe4c60c-1178-406f-b794-ce8c9807c2c1n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3571 bytes --]

Re: Open Discussion: Features of Templating

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 7208 bytes --]

There are several things which have to be decided for something as simple
as equality comparison. I have no idea how hard or easy it would be to
implement it in any form for Pandoc templates, but will point to some
decisions which might need to be made.

Pandoc variables/metadata can be of several types: strings, booleans,
complex data structures — lists and key—value mappings which in turn can
contain any type. I’m not sure if Pandoc treats numbers as distinct from
strings; metadata seems not to. This raises several questions: What happens
when the compared objects are of different types: are they silently unequal
or is there a warning or an error? How do complex data types compare to
each other: content or object equality, same keys or also same values, or
same number of keys for mappings? Same items or same number of items for
lists? You probably end up needing a length operator!

If there are going to be size comparison (less/greater (or equal)) how is
size determined. I touched on that above for complex data types, but there
are problems for strings too: do we compare byte length or (UTF-8)
character length? (I hope the latter!) I don’t know to what extent metadata
contains AST-like objects at the template processing stage, but if they do
how do they compare: similar to complex data data structures and/or text
content equality? If both there may be need for an operator which
stringifies its operand. Shall it be possible to regex match strings?

It may be possible to mimic the way Pandoc’s Lua API does these things,
perhaps even possible to reuse code but pretty soon you will start to have
features similar to a general programming language. Where do you draw the
line? You could even allow embedding for example Lua, but that may clash
with more core templating features, there may be security concerns, and how
do you pass what to Lua-land?

I have thought a lot about this kind of thing, having put quite some work
into creating a small template-like (pre)processor — I call it an
interpolation engine because it can’t really handle formatting of
multi-line blocks. Two things which I needed to decide were whether to
support complex data structures and whether it should be possible to assign
values to variables. For now the answer is yes to the first — with the
limitation that you can only compare string and/or number xor boolean leaf
values — and no to the second. In core only comparison, pattern matching
and true/false — Lua style where only `false` and `nil` evaluate as false —
or empty/nonempty — empty string and 0 also evaluate as false — boolean
evaluation with something like a ternary operator syntax is in core but
there is extensibility by letting the program using the engine supply
functions to do arbitrary things, with a syntax for calling and passing
arguments to such functions. There is also a mechanism for inserting
characters by codepoint or entity name. The thing is that even with
comparison etc. built in from scratch and with the limitations on what can
be compared a lot of decisions and different operators had to be made. I
ended up with several things which made the language not-so-small anymore:

-   Distinct logic operators for if-true/if-false and if-non-empty/if-empty
(they are modular `??` vs. `?!` and `&?` vs. `&!` but anyway!

-   Distinct operators for numeric (e.g. `==#`) and case-folded (e.g.
`==?`) comparison in addition to straight string comparison (e.g. `==`).
Added complication: there is (of course) no numeric pattern matching, but
the parser has to take special note of attempts to use the would-be
operator!

-   Distinct “text” — parenthesis delimited, with interpolation — and
“string” — quotes delimited without interpolation.

-   Distinct string-variable/function/character-by-id/math interpolation
syntaxes (and math is basically an additional DSL which isn’t fully
implemented yet!)

-   Distinct string and math variable spaces.

-   Support methods for doing expansion-like things with function arguments.

-   There are all of for/while/until style loops (although limited to the
kinds of comparisons described above).

The outcome: I have switched to a simpler syntax which basically is sprintf
on not quite as much steroids:

(1) Arguments are specified as variable names inside the string.

(2) Custom conversions implemented as functions passed when instantiating
the formatter object.

(3) Custom conversions can have almost arbitrary names, including
multi-character names.

(4) Multiple arguments per format specifier are possible:

    (a) by default this means “use the value of the first existing variable
corresponding to a name in the argument list”, but
    (b) custom conversions can use the arguments (all strings) however they
please, including using them as embedded format strings.

Nothing more, but (2), (3) and (4)(b) make a wide variety of extensibility
possible — including everything the more complex syntax described above
does —, so inevitably I wrote a “micro” implementation which leaves any
non-standard argument parsing to custom conversion functions and in core
supports only single variable name arguments and standard sprintf
conversions.

/bpj


Den tis 9 maj 2023 16:08Stephan Meijer <me-nPKYAObcRdo6Blr+0TYHagC/G2K4zDHf@public.gmane.org> skrev:

> I would like to create an open discussion around this.
> ------------------------------
>
> Hi all,
>
> As I understand, the templating engine Pandoc uses is built-in. This is
> all good, but I feel like there are some missing features.
>
> One feature I personally miss:
> If-statement conditions
>
> What I miss most, is being able to create conditions in if-statments. For
> example:
>
> %if(pagetitle == "null")%
> ...
> %endif%
>
> Currently this seems not possible.
>
> Other users that have noticed this:
>
>    -
>    https://stackoverflow.com/questions/45404607/if-conditionals-in-pandoc-templates-depending-on-value-of-a-variable
>
> ------------------------------
>
> Feel free to add your own points.
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/dbe4c60c-1178-406f-b794-ce8c9807c2c1n%40googlegroups.com
> <https://groups.google.com/d/msgid/pandoc-discuss/dbe4c60c-1178-406f-b794-ce8c9807c2c1n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhC%3D5P-rTeCFbuuVE%2B5YG7b2ND7r%2B0v7vQGoEzFOyAjsDw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 10816 bytes --]

Re: Open Discussion: Features of Templating

[Stephan Meijer]

[-- Attachment #1.1: Type: text/plain, Size: 7690 bytes --]

Maybe at this stage it'd even be easier to be able to decouple the 
templating engine from Pandoc, so people can use their own templating 
engine.

On Tuesday, 9 May 2023 at 19:35:18 UTC+2 BPJ wrote:

> There are several things which have to be decided for something as simple 
> as equality comparison. I have no idea how hard or easy it would be to 
> implement it in any form for Pandoc templates, but will point to some 
> decisions which might need to be made.
>
> Pandoc variables/metadata can be of several types: strings, booleans, 
> complex data structures — lists and key—value mappings which in turn can 
> contain any type. I’m not sure if Pandoc treats numbers as distinct from 
> strings; metadata seems not to. This raises several questions: What happens 
> when the compared objects are of different types: are they silently unequal 
> or is there a warning or an error? How do complex data types compare to 
> each other: content or object equality, same keys or also same values, or 
> same number of keys for mappings? Same items or same number of items for 
> lists? You probably end up needing a length operator!
>
> If there are going to be size comparison (less/greater (or equal)) how is 
> size determined. I touched on that above for complex data types, but there 
> are problems for strings too: do we compare byte length or (UTF-8) 
> character length? (I hope the latter!) I don’t know to what extent metadata 
> contains AST-like objects at the template processing stage, but if they do 
> how do they compare: similar to complex data data structures and/or text 
> content equality? If both there may be need for an operator which 
> stringifies its operand. Shall it be possible to regex match strings?
>
> It may be possible to mimic the way Pandoc’s Lua API does these things, 
> perhaps even possible to reuse code but pretty soon you will start to have 
> features similar to a general programming language. Where do you draw the 
> line? You could even allow embedding for example Lua, but that may clash 
> with more core templating features, there may be security concerns, and how 
> do you pass what to Lua-land?
>
> I have thought a lot about this kind of thing, having put quite some work 
> into creating a small template-like (pre)processor — I call it an 
> interpolation engine because it can’t really handle formatting of 
> multi-line blocks. Two things which I needed to decide were whether to 
> support complex data structures and whether it should be possible to assign 
> values to variables. For now the answer is yes to the first — with the 
> limitation that you can only compare string and/or number xor boolean leaf 
> values — and no to the second. In core only comparison, pattern matching 
> and true/false — Lua style where only `false` and `nil` evaluate as false — 
> or empty/nonempty — empty string and 0 also evaluate as false — boolean 
> evaluation with something like a ternary operator syntax is in core but 
> there is extensibility by letting the program using the engine supply 
> functions to do arbitrary things, with a syntax for calling and passing 
> arguments to such functions. There is also a mechanism for inserting 
> characters by codepoint or entity name. The thing is that even with 
> comparison etc. built in from scratch and with the limitations on what can 
> be compared a lot of decisions and different operators had to be made. I 
> ended up with several things which made the language not-so-small anymore:
>
> -   Distinct logic operators for if-true/if-false and 
> if-non-empty/if-empty (they are modular `??` vs. `?!` and `&?` vs. `&!` but 
> anyway!
>
> -   Distinct operators for numeric (e.g. `==#`) and case-folded (e.g. 
> `==?`) comparison in addition to straight string comparison (e.g. `==`). 
> Added complication: there is (of course) no numeric pattern matching, but 
> the parser has to take special note of attempts to use the would-be 
> operator!
>
> -   Distinct “text” — parenthesis delimited, with interpolation — and 
> “string” — quotes delimited without interpolation.
>
> -   Distinct string-variable/function/character-by-id/math interpolation 
> syntaxes (and math is basically an additional DSL which isn’t fully 
> implemented yet!)
>
> -   Distinct string and math variable spaces.
>
> -   Support methods for doing expansion-like things with function 
> arguments.
>
> -   There are all of for/while/until style loops (although limited to the 
> kinds of comparisons described above).
>
> The outcome: I have switched to a simpler syntax which basically is 
> sprintf on not quite as much steroids:
>
> (1) Arguments are specified as variable names inside the string.
>
> (2) Custom conversions implemented as functions passed when instantiating 
> the formatter object.
>
> (3) Custom conversions can have almost arbitrary names, including 
> multi-character names.
>
> (4) Multiple arguments per format specifier are possible:
>
>     (a) by default this means “use the value of the first existing 
> variable corresponding to a name in the argument list”, but
>     (b) custom conversions can use the arguments (all strings) however 
> they please, including using them as embedded format strings.
>
> Nothing more, but (2), (3) and (4)(b) make a wide variety of extensibility 
> possible — including everything the more complex syntax described above 
> does —, so inevitably I wrote a “micro” implementation which leaves any 
> non-standard argument parsing to custom conversion functions and in core 
> supports only single variable name arguments and standard sprintf 
> conversions.
>
> /bpj
>
>
> Den tis 9 maj 2023 16:08Stephan Meijer <m...-nPKYAObcRdo6Blr+0TYHagC/G2K4zDHf@public.gmane.org> skrev:
>
>> I would like to create an open discussion around this.
>> ------------------------------
>>
>> Hi all,
>>
>> As I understand, the templating engine Pandoc uses is built-in. This is 
>> all good, but I feel like there are some missing features.
>>
>> One feature I personally miss:
>> If-statement conditions
>>
>> What I miss most, is being able to create conditions in if-statments. For 
>> example:
>>
>> %if(pagetitle == "null")%
>> ...
>> %endif%
>>
>> Currently this seems not possible.
>>
>> Other users that have noticed this:
>>
>>    - 
>>    https://stackoverflow.com/questions/45404607/if-conditionals-in-pandoc-templates-depending-on-value-of-a-variable
>>
>> ------------------------------
>>
>> Feel free to add your own points.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pandoc-discuss/dbe4c60c-1178-406f-b794-ce8c9807c2c1n%40googlegroups.com 
>> <https://groups.google.com/d/msgid/pandoc-discuss/dbe4c60c-1178-406f-b794-ce8c9807c2c1n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/0297ad79-799e-4ed4-aa5b-943c124f7f55n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 11861 bytes --]

Re: Open Discussion: Features of Templating

[Albert Krewinkel]

Stephan Meijer <me-nPKYAObcRdo6Blr+0TYHagC/G2K4zDHf@public.gmane.org> writes:

> Maybe at this stage it'd even be easier to be able to decouple the
> templating engine from Pandoc, so people can use their own templating
> engine.

It's almost possible to do that already, custom writers get us something
like 80 % of the way there: any Lua template engine with a compatible
interface could be used in theory. The limitations are:

- The pandoc.template.meta_to_context function is difficult to use with
  built-in writers.

- The custom writers cannot access the template variables that are
  auto-generated by a built-in writer.

Both limitations could be fixed by exporting more of the writer
internals from the respective Haskell modules, and then in turn to
expose those to Lua. Not difficult, but a good bit of typing, API
changes, and the need to break-down abstractions (making writers less
opaque). Could be nice to have, but I'm not sure if the result would
justify the effort and downsides.


-- 
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Open Discussion: Features of Templating

[John MacFarlane]

I'll note that the pandoc template engine can do things no external one
will be able to do, because it interacts with doclayout.  So, you can
get very natural wrapping and indenting that takes into account the
template context.  Maybe not a big deal for everyone.


> On May 9, 2023, at 12:37 PM, Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> wrote:
> 
> 
> Stephan Meijer <me-nPKYAObcRdo6Blr+0TYHagC/G2K4zDHf@public.gmane.org> writes:
> 
>> Maybe at this stage it'd even be easier to be able to decouple the
>> templating engine from Pandoc, so people can use their own templating
>> engine.
> 
> It's almost possible to do that already, custom writers get us something
> like 80 % of the way there: any Lua template engine with a compatible
> interface could be used in theory. The limitations are:
> 
> - The pandoc.template.meta_to_context function is difficult to use with
>  built-in writers.
> 
> - The custom writers cannot access the template variables that are
>  auto-generated by a built-in writer.
> 
> Both limitations could be fixed by exporting more of the writer
> internals from the respective Haskell modules, and then in turn to
> expose those to Lua. Not difficult, but a good bit of typing, API
> changes, and the need to break-down abstractions (making writers less
> opaque). Could be nice to have, but I'm not sure if the result would
> justify the effort and downsides.
> 
> 
> -- 
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/875y91s2ni.fsf%40zeitkraut.de.