* Lua filter to automatically tag keywords for TeX indexing
@ 2022-11-03 1:20 bapt a
[not found] ` <7f570676-2876-4e29-a8c0-9a765617f141n-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>
0 siblings, 1 reply; 4+ messages in thread
From: bapt a @ 2022-11-03 1:20 UTC (permalink / raw)
To: pandoc-discuss
[-- Attachment #1.1: Type: text/plain, Size: 2518 bytes --]
Hi all,
I've started writing a technical book using Quarto markdown, which uses
pandoc with Lua filters under the hood to produce a website as well as the
publisher's pdf format (via LaTeX).
I quite like to keep the source document as plain as possible, and I'm
wondering if I could avoid the use of [concept]{.index}, which gets turned
into \index{concept}, and instead write a Lua filter with my custom list of
keywords, and have pandoc automatically match them as they appear in the
text.
As a proof of principle I wrote the following code (see below), which
matches specific keywords, and reformats them as small-caps. I quickly
realised that trailing punctuation, such as "concept, ..." will fail to
match, so I'm using gsub to strip such punctuation before matching. It
works, but I'm a bit worried:
- what's the overhead of such a filter, in practice? From what I
understand, every single string element in the AST will be processed by
gsub then tested for a match. Are Lua filters walking down the AST fast
enough that I shouldn't worry about it? (as far as I can tell on small
examples, it seems fine)
- assuming this idea is reasonable, I might want to do a few similar
operations, e.g. reformatting program languages (as in this example code),
wrapping keywords in \index{}, etc., and the exact format will often depend
on the output target (html vs TeX etc.). Is there a better construct for
this than successive if/else statements to look for matches? (I don't know
much Lua)
Best regards,
baptiste
Lua filter:
-----
local text = require 'text'
local pandoc = require 'pandoc'
-- keywords to look for in the document
local langs = {"Matlab", "R", "Julia", "C++"}
function Includes(tab, val)
-- strip trailing punctuation before matching
local bare = string.gsub(val,"[%.|,|;|:]", "")
for index, value in ipairs(tab) do
if value == bare then
return true
end
end
return false
end
function Replace_langname(elem)
if Includes(langs, elem.text) then
return pandoc.SmallCaps(text.lower(elem.text))
else
return elem
end
end
return {{Str = Replace_langname}}
--
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/7f570676-2876-4e29-a8c0-9a765617f141n%40googlegroups.com.
[-- Attachment #1.2: Type: text/html, Size: 5243 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Lua filter to automatically tag keywords for TeX indexing
[not found] ` <7f570676-2876-4e29-a8c0-9a765617f141n-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>
@ 2022-11-03 19:18 ` Bastien DUMONT
2022-11-03 19:39 ` John MacFarlane
1 sibling, 0 replies; 4+ messages in thread
From: Bastien DUMONT @ 2022-11-03 19:18 UTC (permalink / raw)
To: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw
Iterating over all possible values at every string can be expensive. You can speed up the process by rewriting Includes like this:
```
local bool_tables = {}
function Includes(tab, val)
-- strip trailing punctuation before matching
local bare = string.gsub(val,"[%.|,|;|:]", "")
-- The first time a given value of tab is tested in Includes,
-- a table is added in bool_tables which contains
-- a boolean key-value entry for each element in tab.
if not bool_tables[tab] then
bool_tables[tab] = {}
for _, elem in ipairs(tab) do
bool_tables[tab][elem] = true
end
end
-- So we only have to test once for the key-value entry
-- instead of iterating over all elements in tab
-- every time Inclues is called.
return bool_tables[tab][val]
end
```
Le Wednesday 02 November 2022 à 06:20:37PM, bapt a a écrit :
> Hi all,
>
> I've started writing a technical book using Quarto markdown, which uses pandoc
> with Lua filters under the hood to produce a website as well as the publisher's
> pdf format (via LaTeX).
> I quite like to keep the source document as plain as possible, and I'm
> wondering if I could avoid the use of [concept]{.index}, which gets turned into
> \index{concept}, and instead write a Lua filter with my custom list of
> keywords, and have pandoc automatically match them as they appear in the text.
> As a proof of principle I wrote the following code (see below), which matches
> specific keywords, and reformats them as small-caps. I quickly realised that
> trailing punctuation, such as "concept, ..." will fail to match, so I'm using
> gsub to strip such punctuation before matching. It works, but I'm a bit
> worried:
>
> - what's the overhead of such a filter, in practice? From what I understand,
> every single string element in the AST will be processed by gsub then tested
> for a match. Are Lua filters walking down the AST fast enough that I shouldn't
> worry about it? (as far as I can tell on small examples, it seems fine)
>
> - assuming this idea is reasonable, I might want to do a few similar
> operations, e.g. reformatting program languages (as in this example code),
> wrapping keywords in \index{}, etc., and the exact format will often depend on
> the output target (html vs TeX etc.). Is there a better construct for this than
> successive if/else statements to look for matches? (I don't know much Lua)
>
> Best regards,
>
> baptiste
>
> Lua filter:
> -----
>
> local text = require 'text'
> local pandoc = require 'pandoc'
>
> -- keywords to look for in the document
> local langs = {"Matlab", "R", "Julia", "C++"}
>
> function Includes(tab, val)
> -- strip trailing punctuation before matching
> local bare = string.gsub(val,"[%.|,|;|:]", "")
>
> for index, value in ipairs(tab) do
> if value == bare then
> return true
> end
> end
>
> return false
> end
>
> function Replace_langname(elem)
> if Includes(langs, elem.text) then
> return pandoc.SmallCaps(text.lower(elem.text))
> else
> return elem
> end
> end
>
> return {{Str = Replace_langname}}
>
>
>
>
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to [1]pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit [2]https://groups.google.com/d/msgid/
> pandoc-discuss/7f570676-2876-4e29-a8c0-9a765617f141n%40googlegroups.com.
>
> References:
>
> [1] mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
> [2] https://groups.google.com/d/msgid/pandoc-discuss/7f570676-2876-4e29-a8c0-9a765617f141n%40googlegroups.com?utm_medium=email&utm_source=footer
--
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/Y2QT/FV43helVxE0%40localhost.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Lua filter to automatically tag keywords for TeX indexing
[not found] ` <7f570676-2876-4e29-a8c0-9a765617f141n-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>
2022-11-03 19:18 ` Bastien DUMONT
@ 2022-11-03 19:39 ` John MacFarlane
[not found] ` <ECDE1635-3DD4-4E57-8D66-E546B4742622-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
1 sibling, 1 reply; 4+ messages in thread
From: John MacFarlane @ 2022-11-03 19:39 UTC (permalink / raw)
To: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw
> On Nov 2, 2022, at 6:20 PM, bapt a <auguieba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> Hi all,
>
> I've started writing a technical book using Quarto markdown, which uses pandoc with Lua filters under the hood to produce a website as well as the publisher's pdf format (via LaTeX).
> I quite like to keep the source document as plain as possible, and I'm wondering if I could avoid the use of [concept]{.index}, which gets turned into \index{concept}, and instead write a Lua filter with my custom list of keywords, and have pandoc automatically match them as they appear in the text.
> As a proof of principle I wrote the following code (see below), which matches specific keywords, and reformats them as small-caps. I quickly realised that trailing punctuation, such as "concept, ..." will fail to match, so I'm using gsub to strip such punctuation before matching. It works, but I'm a bit worried:
>
> - what's the overhead of such a filter, in practice? From what I understand, every single string element in the AST will be processed by gsub then tested for a match. Are Lua filters walking down the AST fast enough that I shouldn't worry about it? (as far as I can tell on small examples, it seems fine)
The AST walking is very fast. See the benchmarks at the beginning of https://pandoc.org/lua-filters.html for one example.
> - assuming this idea is reasonable, I might want to do a few similar operations, e.g. reformatting program languages (as in this example code), wrapping keywords in \index{}, etc., and the exact format will often depend on the output target (html vs TeX etc.). Is there a better construct for this than successive if/else statements to look for matches? (I don't know much Lua)
In lua you can do
string.gusb(val, [“(%l*)”], function (word)
if indexable[word] then
.. whatever ..
end
end)
This will run the function on every group of letters in the matched string.
Here I’m assuming you have a lua table indexable that maps words to true, e.g.
{ cow: true, horse: true }
That will be much faster than iterating through an array as you’re doing here.
--
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ECDE1635-3DD4-4E57-8D66-E546B4742622%40gmail.com.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Lua filter to automatically tag keywords for TeX indexing
[not found] ` <ECDE1635-3DD4-4E57-8D66-E546B4742622-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
@ 2022-11-04 21:26 ` bapt a
0 siblings, 0 replies; 4+ messages in thread
From: bapt a @ 2022-11-04 21:26 UTC (permalink / raw)
To: pandoc-discuss
[-- Attachment #1.1: Type: text/plain, Size: 3022 bytes --]
Thank you both for the helpful replies; I don't fully understand how these
boolean tables are used in Lua (clearly it works, I just don't fully get
it), but it seems to be an important concept so I'll read up on it.
Thanks,
baptiste
On Friday, 4 November 2022 at 08:39:29 UTC+13 fiddlosopher wrote:
>
>
> > On Nov 2, 2022, at 6:20 PM, bapt a <augu...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >
> > Hi all,
> >
> > I've started writing a technical book using Quarto markdown, which uses
> pandoc with Lua filters under the hood to produce a website as well as the
> publisher's pdf format (via LaTeX).
> > I quite like to keep the source document as plain as possible, and I'm
> wondering if I could avoid the use of [concept]{.index}, which gets turned
> into \index{concept}, and instead write a Lua filter with my custom list of
> keywords, and have pandoc automatically match them as they appear in the
> text.
> > As a proof of principle I wrote the following code (see below), which
> matches specific keywords, and reformats them as small-caps. I quickly
> realised that trailing punctuation, such as "concept, ..." will fail to
> match, so I'm using gsub to strip such punctuation before matching. It
> works, but I'm a bit worried:
> >
> > - what's the overhead of such a filter, in practice? From what I
> understand, every single string element in the AST will be processed by
> gsub then tested for a match. Are Lua filters walking down the AST fast
> enough that I shouldn't worry about it? (as far as I can tell on small
> examples, it seems fine)
>
> The AST walking is very fast. See the benchmarks at the beginning of
> https://pandoc.org/lua-filters.html for one example.
>
> > - assuming this idea is reasonable, I might want to do a few similar
> operations, e.g. reformatting program languages (as in this example code),
> wrapping keywords in \index{}, etc., and the exact format will often depend
> on the output target (html vs TeX etc.). Is there a better construct for
> this than successive if/else statements to look for matches? (I don't know
> much Lua)
>
> In lua you can do
>
> string.gusb(val, [“(%l*)”], function (word)
> if indexable[word] then
> .. whatever ..
> end
> end)
>
> This will run the function on every group of letters in the matched string.
> Here I’m assuming you have a lua table indexable that maps words to true,
> e.g.
>
> { cow: true, horse: true }
>
> That will be much faster than iterating through an array as you’re doing
> here.
>
>
--
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/23ccfb2a-c458-49e1-a275-dad452f5d2e3n%40googlegroups.com.
[-- Attachment #1.2: Type: text/html, Size: 3953 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-11-04 21:26 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-03 1:20 Lua filter to automatically tag keywords for TeX indexing bapt a
[not found] ` <7f570676-2876-4e29-a8c0-9a765617f141n-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>
2022-11-03 19:18 ` Bastien DUMONT
2022-11-03 19:39 ` John MacFarlane
[not found] ` <ECDE1635-3DD4-4E57-8D66-E546B4742622-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2022-11-04 21:26 ` bapt a
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).