* Lua filter to automatically tag keywords for TeX indexing @ 2022-11-03 1:20 bapt a [not found] ` <7f570676-2876-4e29-a8c0-9a765617f141n-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> 0 siblings, 1 reply; 4+ messages in thread From: bapt a @ 2022-11-03 1:20 UTC (permalink / raw) To: pandoc-discuss [-- Attachment #1.1: Type: text/plain, Size: 2518 bytes --] Hi all, I've started writing a technical book using Quarto markdown, which uses pandoc with Lua filters under the hood to produce a website as well as the publisher's pdf format (via LaTeX). I quite like to keep the source document as plain as possible, and I'm wondering if I could avoid the use of [concept]{.index}, which gets turned into \index{concept}, and instead write a Lua filter with my custom list of keywords, and have pandoc automatically match them as they appear in the text. As a proof of principle I wrote the following code (see below), which matches specific keywords, and reformats them as small-caps. I quickly realised that trailing punctuation, such as "concept, ..." will fail to match, so I'm using gsub to strip such punctuation before matching. It works, but I'm a bit worried: - what's the overhead of such a filter, in practice? From what I understand, every single string element in the AST will be processed by gsub then tested for a match. Are Lua filters walking down the AST fast enough that I shouldn't worry about it? (as far as I can tell on small examples, it seems fine) - assuming this idea is reasonable, I might want to do a few similar operations, e.g. reformatting program languages (as in this example code), wrapping keywords in \index{}, etc., and the exact format will often depend on the output target (html vs TeX etc.). Is there a better construct for this than successive if/else statements to look for matches? (I don't know much Lua) Best regards, baptiste Lua filter: ----- local text = require 'text' local pandoc = require 'pandoc' -- keywords to look for in the document local langs = {"Matlab", "R", "Julia", "C++"} function Includes(tab, val) -- strip trailing punctuation before matching local bare = string.gsub(val,"[%.|,|;|:]", "") for index, value in ipairs(tab) do if value == bare then return true end end return false end function Replace_langname(elem) if Includes(langs, elem.text) then return pandoc.SmallCaps(text.lower(elem.text)) else return elem end end return {{Str = Replace_langname}} -- You received this message because you are subscribed to the Google Groups "pandoc-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/7f570676-2876-4e29-a8c0-9a765617f141n%40googlegroups.com. [-- Attachment #1.2: Type: text/html, Size: 5243 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
[parent not found: <7f570676-2876-4e29-a8c0-9a765617f141n-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>]
* Re: Lua filter to automatically tag keywords for TeX indexing [not found] ` <7f570676-2876-4e29-a8c0-9a765617f141n-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> @ 2022-11-03 19:18 ` Bastien DUMONT 2022-11-03 19:39 ` John MacFarlane 1 sibling, 0 replies; 4+ messages in thread From: Bastien DUMONT @ 2022-11-03 19:18 UTC (permalink / raw) To: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw Iterating over all possible values at every string can be expensive. You can speed up the process by rewriting Includes like this: ``` local bool_tables = {} function Includes(tab, val) -- strip trailing punctuation before matching local bare = string.gsub(val,"[%.|,|;|:]", "") -- The first time a given value of tab is tested in Includes, -- a table is added in bool_tables which contains -- a boolean key-value entry for each element in tab. if not bool_tables[tab] then bool_tables[tab] = {} for _, elem in ipairs(tab) do bool_tables[tab][elem] = true end end -- So we only have to test once for the key-value entry -- instead of iterating over all elements in tab -- every time Inclues is called. return bool_tables[tab][val] end ``` Le Wednesday 02 November 2022 à 06:20:37PM, bapt a a écrit : > Hi all, > > I've started writing a technical book using Quarto markdown, which uses pandoc > with Lua filters under the hood to produce a website as well as the publisher's > pdf format (via LaTeX). > I quite like to keep the source document as plain as possible, and I'm > wondering if I could avoid the use of [concept]{.index}, which gets turned into > \index{concept}, and instead write a Lua filter with my custom list of > keywords, and have pandoc automatically match them as they appear in the text. > As a proof of principle I wrote the following code (see below), which matches > specific keywords, and reformats them as small-caps. I quickly realised that > trailing punctuation, such as "concept, ..." will fail to match, so I'm using > gsub to strip such punctuation before matching. It works, but I'm a bit > worried: > > - what's the overhead of such a filter, in practice? From what I understand, > every single string element in the AST will be processed by gsub then tested > for a match. Are Lua filters walking down the AST fast enough that I shouldn't > worry about it? (as far as I can tell on small examples, it seems fine) > > - assuming this idea is reasonable, I might want to do a few similar > operations, e.g. reformatting program languages (as in this example code), > wrapping keywords in \index{}, etc., and the exact format will often depend on > the output target (html vs TeX etc.). Is there a better construct for this than > successive if/else statements to look for matches? (I don't know much Lua) > > Best regards, > > baptiste > > Lua filter: > ----- > > local text = require 'text' > local pandoc = require 'pandoc' > > -- keywords to look for in the document > local langs = {"Matlab", "R", "Julia", "C++"} > > function Includes(tab, val) > -- strip trailing punctuation before matching > local bare = string.gsub(val,"[%.|,|;|:]", "") > > for index, value in ipairs(tab) do > if value == bare then > return true > end > end > > return false > end > > function Replace_langname(elem) > if Includes(langs, elem.text) then > return pandoc.SmallCaps(text.lower(elem.text)) > else > return elem > end > end > > return {{Str = Replace_langname}} > > > > > > > > > > > > -- > You received this message because you are subscribed to the Google Groups > "pandoc-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an email > to [1]pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > To view this discussion on the web visit [2]https://groups.google.com/d/msgid/ > pandoc-discuss/7f570676-2876-4e29-a8c0-9a765617f141n%40googlegroups.com. > > References: > > [1] mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org > [2] https://groups.google.com/d/msgid/pandoc-discuss/7f570676-2876-4e29-a8c0-9a765617f141n%40googlegroups.com?utm_medium=email&utm_source=footer -- You received this message because you are subscribed to the Google Groups "pandoc-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/Y2QT/FV43helVxE0%40localhost. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Lua filter to automatically tag keywords for TeX indexing [not found] ` <7f570676-2876-4e29-a8c0-9a765617f141n-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> 2022-11-03 19:18 ` Bastien DUMONT @ 2022-11-03 19:39 ` John MacFarlane [not found] ` <ECDE1635-3DD4-4E57-8D66-E546B4742622-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 1 sibling, 1 reply; 4+ messages in thread From: John MacFarlane @ 2022-11-03 19:39 UTC (permalink / raw) To: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw > On Nov 2, 2022, at 6:20 PM, bapt a <auguieba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > Hi all, > > I've started writing a technical book using Quarto markdown, which uses pandoc with Lua filters under the hood to produce a website as well as the publisher's pdf format (via LaTeX). > I quite like to keep the source document as plain as possible, and I'm wondering if I could avoid the use of [concept]{.index}, which gets turned into \index{concept}, and instead write a Lua filter with my custom list of keywords, and have pandoc automatically match them as they appear in the text. > As a proof of principle I wrote the following code (see below), which matches specific keywords, and reformats them as small-caps. I quickly realised that trailing punctuation, such as "concept, ..." will fail to match, so I'm using gsub to strip such punctuation before matching. It works, but I'm a bit worried: > > - what's the overhead of such a filter, in practice? From what I understand, every single string element in the AST will be processed by gsub then tested for a match. Are Lua filters walking down the AST fast enough that I shouldn't worry about it? (as far as I can tell on small examples, it seems fine) The AST walking is very fast. See the benchmarks at the beginning of https://pandoc.org/lua-filters.html for one example. > - assuming this idea is reasonable, I might want to do a few similar operations, e.g. reformatting program languages (as in this example code), wrapping keywords in \index{}, etc., and the exact format will often depend on the output target (html vs TeX etc.). Is there a better construct for this than successive if/else statements to look for matches? (I don't know much Lua) In lua you can do string.gusb(val, [“(%l*)”], function (word) if indexable[word] then .. whatever .. end end) This will run the function on every group of letters in the matched string. Here I’m assuming you have a lua table indexable that maps words to true, e.g. { cow: true, horse: true } That will be much faster than iterating through an array as you’re doing here. -- You received this message because you are subscribed to the Google Groups "pandoc-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ECDE1635-3DD4-4E57-8D66-E546B4742622%40gmail.com. ^ permalink raw reply [flat|nested] 4+ messages in thread
[parent not found: <ECDE1635-3DD4-4E57-8D66-E546B4742622-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]
* Re: Lua filter to automatically tag keywords for TeX indexing [not found] ` <ECDE1635-3DD4-4E57-8D66-E546B4742622-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> @ 2022-11-04 21:26 ` bapt a 0 siblings, 0 replies; 4+ messages in thread From: bapt a @ 2022-11-04 21:26 UTC (permalink / raw) To: pandoc-discuss [-- Attachment #1.1: Type: text/plain, Size: 3022 bytes --] Thank you both for the helpful replies; I don't fully understand how these boolean tables are used in Lua (clearly it works, I just don't fully get it), but it seems to be an important concept so I'll read up on it. Thanks, baptiste On Friday, 4 November 2022 at 08:39:29 UTC+13 fiddlosopher wrote: > > > > On Nov 2, 2022, at 6:20 PM, bapt a <augu...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > Hi all, > > > > I've started writing a technical book using Quarto markdown, which uses > pandoc with Lua filters under the hood to produce a website as well as the > publisher's pdf format (via LaTeX). > > I quite like to keep the source document as plain as possible, and I'm > wondering if I could avoid the use of [concept]{.index}, which gets turned > into \index{concept}, and instead write a Lua filter with my custom list of > keywords, and have pandoc automatically match them as they appear in the > text. > > As a proof of principle I wrote the following code (see below), which > matches specific keywords, and reformats them as small-caps. I quickly > realised that trailing punctuation, such as "concept, ..." will fail to > match, so I'm using gsub to strip such punctuation before matching. It > works, but I'm a bit worried: > > > > - what's the overhead of such a filter, in practice? From what I > understand, every single string element in the AST will be processed by > gsub then tested for a match. Are Lua filters walking down the AST fast > enough that I shouldn't worry about it? (as far as I can tell on small > examples, it seems fine) > > The AST walking is very fast. See the benchmarks at the beginning of > https://pandoc.org/lua-filters.html for one example. > > > - assuming this idea is reasonable, I might want to do a few similar > operations, e.g. reformatting program languages (as in this example code), > wrapping keywords in \index{}, etc., and the exact format will often depend > on the output target (html vs TeX etc.). Is there a better construct for > this than successive if/else statements to look for matches? (I don't know > much Lua) > > In lua you can do > > string.gusb(val, [“(%l*)”], function (word) > if indexable[word] then > .. whatever .. > end > end) > > This will run the function on every group of letters in the matched string. > Here I’m assuming you have a lua table indexable that maps words to true, > e.g. > > { cow: true, horse: true } > > That will be much faster than iterating through an array as you’re doing > here. > > -- You received this message because you are subscribed to the Google Groups "pandoc-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/23ccfb2a-c458-49e1-a275-dad452f5d2e3n%40googlegroups.com. [-- Attachment #1.2: Type: text/html, Size: 3953 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-11-04 21:26 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-11-03 1:20 Lua filter to automatically tag keywords for TeX indexing bapt a [not found] ` <7f570676-2876-4e29-a8c0-9a765617f141n-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> 2022-11-03 19:18 ` Bastien DUMONT 2022-11-03 19:39 ` John MacFarlane [not found] ` <ECDE1635-3DD4-4E57-8D66-E546B4742622-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2022-11-04 21:26 ` bapt a
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).