Newbie question on lua filters

Newbie question on lua filters

[Pranesh Prakash]

[-- Attachment #1.1: Type: text/plain, Size: 1422 bytes --]

Dear all,
I'm trying to learn how to write lua filters.  I thought I'd try a simple 
substitution filter, but am a bit out of my depth already.  (Please do 
note: I am not a coder, so pardon me if the answer to my question is 
obvious.)

What I'm trying to get done: when generating a standalone markdown file, 
substitute `...` for `---` as the closing YAML delimiter.

By going through https://pandoc.org/lua-filters.html, I realized that the 
first thing I need to do is to figure out what kind of an element the `---` 
is, so that I can perform a substitution function.  I thought I would look 
through the AST to figure that out by doing `pandoc test.md -s -t native`.

However, I don't see `---` in the output.  For a test file, I see:
~~~
Pandoc (Meta {unMeta = fromList [("date",MetaInlines [Str 
"2020-10-01"]),("title",MetaInlines [Str "Draft",Space,Str "Document"])]})
[Header 1 ("preamble",[],[]) [Str "Preamble"]
~~~

So how do I replace a string that doesn't show up in the AST?

Regards,
Pranesh

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/0bd3cf07-77aa-4106-8e3c-f24a8d74def0n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1912 bytes --]

Re: Newbie question on lua filters

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 3976 bytes --]

Unfortunately the YAML delimiters aren't any elements at all. Rather all
YAML blocks are combined into one single Meta object, a glorified mapping
table which contains further elements like MetaMap, MetaList, MetaBlocks
and MetaInlines, the last two of which contain lists of regular element
object of the types found in the body of the document.

It is generally the case that you can't modify the syntax of input formats
with filters. By the time the filter sees the data the document has already
been parsed into an Abstract Syntax Tree (aka AST), where the operative
word is "abstract": the AST is by design the same regardless of the input
and output formats. That is how Pandoc manages to convert between dozens of
formats: first one module, a "reader" parses the input into the AST, then
any filters have their go on the AST and then another module, a "writer"
converts the AST into the output format.

Your best bet for "extending the syntax" is to use attributes on divs,
spans, code blocks and inline code elements; see <
https://pandoc.org/MANUAL.html#extension-attributes> and <
https://pandoc.org/lua-filters.html#type-attr>.

Basically it's similar to LaTeX, only more limited even more ugly! :-) The
good news is that you unlike LaTeX don't have to use that syntax all the
time, and that it is much more sane than LaTeX under the hood.

You may also want to check out pp <https://github.com/CDSoft/pp> which is a
preprocessor which modifies the input before pandoc sees it and also allows
you to define your own syntax of sorts, basically a more limited but
equally insane — as in everything is a single blob of data where you must
be careful what you (re)define — version of LaTeX! :-)


-- 
Better --help|less than helpless

Den mån 23 nov. 2020 21:36Pranesh Prakash <the.solipsist-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> skrev:

> Dear all,
> I'm trying to learn how to write lua filters.  I thought I'd try a simple
> substitution filter, but am a bit out of my depth already.  (Please do
> note: I am not a coder, so pardon me if the answer to my question is
> obvious.)
>
> What I'm trying to get done: when generating a standalone markdown file,
> substitute `...` for `---` as the closing YAML delimiter.
>
> By going through https://pandoc.org/lua-filters.html, I realized that the
> first thing I need to do is to figure out what kind of an element the `---`
> is, so that I can perform a substitution function.  I thought I would look
> through the AST to figure that out by doing `pandoc test.md -s -t native`.
>
> However, I don't see `---` in the output.  For a test file, I see:
> ~~~
> Pandoc (Meta {unMeta = fromList [("date",MetaInlines [Str
> "2020-10-01"]),("title",MetaInlines [Str "Draft",Space,Str "Document"])]})
> [Header 1 ("preamble",[],[]) [Str "Preamble"]
> ~~~
>
> So how do I replace a string that doesn't show up in the AST?
>
> Regards,
> Pranesh
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/0bd3cf07-77aa-4106-8e3c-f24a8d74def0n%40googlegroups.com
> <https://groups.google.com/d/msgid/pandoc-discuss/0bd3cf07-77aa-4106-8e3c-f24a8d74def0n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhDEvLr4aYw6QA5k5rF1LThNqqYWGjkRp2qUWHUyoUaeDw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 5475 bytes --]

Re: Newbie question on lua filters

[Pranesh Prakash]

[-- Attachment #1.1: Type: text/plain, Size: 4712 bytes --]

Just to be clear, I am not seeking to "modify the syntax of the input 
format".  I am trying to modify the syntax of the *output* document.  

(The input markdown document has `...`, and I am trying to preserve that in 
the output markdown document, whereas it gets converted to `---`.)

And hence, I am not clear how the option of pre-processing would help in 
this situation.  But it is useful to know generally.

So, in this case, lua filters won't work because lua filters can only 
operate on elements and YAML delimiters are not elements?
On Tuesday, 24 November, 2020 at 2:59:30 am UTC+5:30 BPJ wrote:

> Unfortunately the YAML delimiters aren't any elements at all. Rather all 
> YAML blocks are combined into one single Meta object, a glorified mapping 
> table which contains further elements like MetaMap, MetaList, MetaBlocks 
> and MetaInlines, the last two of which contain lists of regular element 
> object of the types found in the body of the document.
>
> It is generally the case that you can't modify the syntax of input formats 
> with filters. By the time the filter sees the data the document has already 
> been parsed into an Abstract Syntax Tree (aka AST), where the operative 
> word is "abstract": the AST is by design the same regardless of the input 
> and output formats. That is how Pandoc manages to convert between dozens of 
> formats: first one module, a "reader" parses the input into the AST, then 
> any filters have their go on the AST and then another module, a "writer" 
> converts the AST into the output format.
>
> Your best bet for "extending the syntax" is to use attributes on divs, 
> spans, code blocks and inline code elements; see <
> https://pandoc.org/MANUAL.html#extension-attributes> and <
> https://pandoc.org/lua-filters.html#type-attr>.
>
> Basically it's similar to LaTeX, only more limited even more ugly! :-) The 
> good news is that you unlike LaTeX don't have to use that syntax all the 
> time, and that it is much more sane than LaTeX under the hood.
>
> You may also want to check out pp <https://github.com/CDSoft/pp> which is 
> a preprocessor which modifies the input before pandoc sees it and also 
> allows you to define your own syntax of sorts, basically a more limited but 
> equally insane — as in everything is a single blob of data where you must 
> be careful what you (re)define — version of LaTeX! :-)
>
>
> -- 
> Better --help|less than helpless
>
> Den mån 23 nov. 2020 21:36Pranesh Prakash <the.so...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> skrev:
>
>> Dear all,
>> I'm trying to learn how to write lua filters.  I thought I'd try a simple 
>> substitution filter, but am a bit out of my depth already.  (Please do 
>> note: I am not a coder, so pardon me if the answer to my question is 
>> obvious.)
>>
>> What I'm trying to get done: when generating a standalone markdown file, 
>> substitute `...` for `---` as the closing YAML delimiter.
>>
>> By going through https://pandoc.org/lua-filters.html, I realized that 
>> the first thing I need to do is to figure out what kind of an element the 
>> `---` is, so that I can perform a substitution function.  I thought I would 
>> look through the AST to figure that out by doing `pandoc test.md -s -t 
>> native`.
>>
>> However, I don't see `---` in the output.  For a test file, I see:
>> ~~~
>> Pandoc (Meta {unMeta = fromList [("date",MetaInlines [Str 
>> "2020-10-01"]),("title",MetaInlines [Str "Draft",Space,Str "Document"])]})
>> [Header 1 ("preamble",[],[]) [Str "Preamble"]
>> ~~~
>>
>> So how do I replace a string that doesn't show up in the AST?
>>
>> Regards,
>> Pranesh
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pandoc-discuss/0bd3cf07-77aa-4106-8e3c-f24a8d74def0n%40googlegroups.com 
>> <https://groups.google.com/d/msgid/pandoc-discuss/0bd3cf07-77aa-4106-8e3c-f24a8d74def0n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/d0880381-941e-4673-a4ec-c10e240a4f48n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 7647 bytes --]

Re: Newbie question on lua filters

[John MacFarlane]

Pranesh Prakash <the.solipsist-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> Just to be clear, I am not seeking to "modify the syntax of the input 
> format".  I am trying to modify the syntax of the *output* document.  
>
> (The input markdown document has `...`, and I am trying to preserve that in 
> the output markdown document, whereas it gets converted to `---`.)
>
> And hence, I am not clear how the option of pre-processing would help in 
> this situation.  But it is useful to know generally.
>
> So, in this case, lua filters won't work because lua filters can only 
> operate on elements and YAML delimiters are not elements?

Lua filters modify the abstract syntax tree (AST).
The AST doesn't record what ending delimiter is used for
a YAML metadata block, where the block occurs in the document,
or any number of other "concrete" things.

So modifying it can't really with this.