Lua filter reading data from metadata or defaults file

Lua filter reading data from metadata or defaults file

[krulis....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]

[-- Attachment #1.1: Type: text/plain, Size: 1503 bytes --]

I would like to ask, whether it is possible for a lua filter to read data 
from metadata block (or metadata file) or from defaults file? I looked 
around website documentation, also into `lua filter repository`, but found 
no filter accessing metadata values in `defaults file` fields.
I am working on a custom lua filter, for which I want to specify a metadata 
field, which would point to external file with required definitions, very 
similar to python filter `pandoc-mustache` github source here :
<https://github.com/michaelstepner/pandoc-mustache/blob/master/pandoc_mustache/pandoc_mustache.py>

`defaults.yaml`:

```
...

mustache:
- ./src/varDefinitions.yaml

...
```

*Motivation:* Plans for this filter are to provide functionalities of LaTeX 
`glossaries` package to `pandoc`. There are already some filters, which 
take theyre own approach to the issue, but none of them gives full acronym 
support specific to given format. 
Long-term plans would be to also provide support for term definitions in 
`glossaries`-like style.

Thanks for any guidance in this issue.
Regards, Tomas

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/8a242488-066f-4c2a-aa0a-992788fa9e93n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2069 bytes --]

Re: Lua filter reading data from metadata or defaults file

[Albert Krewinkel]

You should be able to access `mustache` as a normal metadata field if it
is nested under the field `metadata` in defaults.yaml:

    ---
    metadata:
      mustache: ./src/varDefinitions.yaml
    ---


krulis....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org writes:

> I would like to ask, whether it is possible for a lua filter to read data
> from metadata block (or metadata file) or from defaults file? I looked
> around website documentation, also into `lua filter repository`, but found
> no filter accessing metadata values in `defaults file` fields.
> I am working on a custom lua filter, for which I want to specify a metadata
> field, which would point to external file with required definitions, very
> similar to python filter `pandoc-mustache` github source here :
> <https://github.com/michaelstepner/pandoc-mustache/blob/master/pandoc_mustache/pandoc_mustache.py>
>
> `defaults.yaml`:
>
> ```
> ...
>
> mustache:
> - ./src/varDefinitions.yaml
>
> ...
> ```
>
> *Motivation:* Plans for this filter are to provide functionalities of LaTeX
> `glossaries` package to `pandoc`. There are already some filters, which
> take theyre own approach to the issue, but none of them gives full acronym
> support specific to given format.
> Long-term plans would be to also provide support for term definitions in
> `glossaries`-like style.
>
> Thanks for any guidance in this issue.
> Regards, Tomas


--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Lua filter reading data from metadata or defaults file

[krulis....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]

[-- Attachment #1.1: Type: text/plain, Size: 2644 bytes --]

Those are good news!

But my question might have been a little confusing.
As a follow-up, how can load and access data provided in arbitraty yaml or 
json file with lua filter? Is there any helper function for that, or I 
should just use simple `open` or something ... ?
I am looking at lua filters part in pandoc docs; example: "Replacing 
placeholders with their metadata value;" which kinda similar to my case, 
but here are just flat values. I would like to specify a filepath to yaml 
file with actual definitions. 

Dne pátek 25. září 2020 v 12:06:24 UTC+2 uživatel Albert Krewinkel napsal:

> You should be able to access `mustache` as a normal metadata field if it
> is nested under the field `metadata` in defaults.yaml:
>
> ---
> metadata:
> mustache: ./src/varDefinitions.yaml
> ---
>
>
> krulis....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org writes:
>
> > I would like to ask, whether it is possible for a lua filter to read data
> > from metadata block (or metadata file) or from defaults file? I looked
> > around website documentation, also into `lua filter repository`, but 
> found
> > no filter accessing metadata values in `defaults file` fields.
> > I am working on a custom lua filter, for which I want to specify a 
> metadata
> > field, which would point to external file with required definitions, very
> > similar to python filter `pandoc-mustache` github source here :
> > <
> https://github.com/michaelstepner/pandoc-mustache/blob/master/pandoc_mustache/pandoc_mustache.py
> >
> >
> > `defaults.yaml`:
> >
> > ```
> > ...
> >
> > mustache:
> > - ./src/varDefinitions.yaml
> >
> > ...
> > ```
> >
> > *Motivation:* Plans for this filter are to provide functionalities of 
> LaTeX
> > `glossaries` package to `pandoc`. There are already some filters, which
> > take theyre own approach to the issue, but none of them gives full 
> acronym
> > support specific to given format.
> > Long-term plans would be to also provide support for term definitions in
> > `glossaries`-like style.
> >
> > Thanks for any guidance in this issue.
> > Regards, Tomas
>
>
> --
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe e836 388d c0b2 1f63 1124
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/9dbbc23d-1cde-4c0b-898a-ecbe4679d614n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3809 bytes --]

Re: Lua filter reading data from metadata or defaults file

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 5194 bytes --]

This function is a hack which takes a file name, reads the file, makes sure
as well as it can that Pandoc will see the text as a YAML block, parses the
text with pandoc and returns the metadata of the result.

``````lua
local function load_yaml (file_name)
  local file = assert(io.open(file_name))
  local text = assert(file:read('a'))
  file:close()
  -- remove leading/trailing whitespace and existing delimiters
  text = text:gsub('^%s+',"")
    :gsub('%s+$',"")
    :gsub('^%-%-%-',"")
    :gsub('%-%-%-$',"")
    :gsub('%.%.%.$',"")
  -- insert delimiters
  text = "---\n" .. text .. "\n---\n"
  -- parse the YAML
  return pandoc.read(text, 'markdown').meta
end
``````

Note that this returns a metadata object, not a plain data data structure.
You will need another function to convert the metadata into a plain data
structure:

``````lua
-- Turn Pandoc metadata into "plain" data:
-- a table for Meta, MetaList, MetaMap;
-- stringify all other elements;
-- return anything else as-is.
-- Either the whole metadata or a metadata item works.
local meta_to_data
do
  meta_to_data = function (meta)
    if 'table' == type(meta) then
      local tag = meta.tag
      if not(tag) or 'Meta' == tag or 'MetaList' == tag or 'MetaMap' == tag
then
        local data = {}
        for k,v in pairs(meta) do
          data[k] = meta_to_data(v)
        end
        return data
      else
        return pandoc.utils.stringify(meta)
      end
    else
      return meta
    end
  end
end
``````

Note:
(1) pandoc.utils.stringify doesn't handle block content very well.
(2) I just rewrote meta_to_data from memory but it should work OK.


-- 
Better --help|less than helpless

Den fre 25 sep. 2020 13:37krulis....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <krulis.tomas.tk-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
skrev:

> Those are good news!
>
> But my question might have been a little confusing.
> As a follow-up, how can load and access data provided in arbitraty yaml or
> json file with lua filter? Is there any helper function for that, or I
> should just use simple `open` or something ... ?
> I am looking at lua filters part in pandoc docs; example: "Replacing
> placeholders with their metadata value;" which kinda similar to my case,
> but here are just flat values. I would like to specify a filepath to yaml
> file with actual definitions.
>
> Dne pátek 25. září 2020 v 12:06:24 UTC+2 uživatel Albert Krewinkel napsal:
>
>> You should be able to access `mustache` as a normal metadata field if it
>> is nested under the field `metadata` in defaults.yaml:
>>
>> ---
>> metadata:
>> mustache: ./src/varDefinitions.yaml
>> ---
>>
>>
>> krulis....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org writes:
>>
>> > I would like to ask, whether it is possible for a lua filter to read
>> data
>> > from metadata block (or metadata file) or from defaults file? I looked
>> > around website documentation, also into `lua filter repository`, but
>> found
>> > no filter accessing metadata values in `defaults file` fields.
>> > I am working on a custom lua filter, for which I want to specify a
>> metadata
>> > field, which would point to external file with required definitions,
>> very
>> > similar to python filter `pandoc-mustache` github source here :
>> > <
>> https://github.com/michaelstepner/pandoc-mustache/blob/master/pandoc_mustache/pandoc_mustache.py>
>>
>> >
>> > `defaults.yaml`:
>> >
>> > ```
>> > ...
>> >
>> > mustache:
>> > - ./src/varDefinitions.yaml
>> >
>> > ...
>> > ```
>> >
>> > *Motivation:* Plans for this filter are to provide functionalities of
>> LaTeX
>> > `glossaries` package to `pandoc`. There are already some filters, which
>> > take theyre own approach to the issue, but none of them gives full
>> acronym
>> > support specific to given format.
>> > Long-term plans would be to also provide support for term definitions
>> in
>> > `glossaries`-like style.
>> >
>> > Thanks for any guidance in this issue.
>> > Regards, Tomas
>>
>>
>> --
>> Albert Krewinkel
>> GPG: 8eed e3e2 e8c5 6f18 81fe e836 388d c0b2 1f63 1124
>>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/9dbbc23d-1cde-4c0b-898a-ecbe4679d614n%40googlegroups.com
> <https://groups.google.com/d/msgid/pandoc-discuss/9dbbc23d-1cde-4c0b-898a-ecbe4679d614n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhBkrFo1pT6UWE6qu%3DaAaOoMsxvNPXwHGiGoauyF-2XPnw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 8026 bytes --]

Re: Lua filter reading data from metadata or defaults file

[krulis....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]

[-- Attachment #1.1: Type: text/plain, Size: 9021 bytes --]

Thank you for your help @BPJ , I test it as soon as possible with the other 
parts of my code and report back. But I must say it is a little hazy to me 
(I have very small knowledge of lua) - the "data" is table consisting of 
key-value pairs of string values in yaml file?

Maybe I have started with project that is beyond my skills now ...

Would it be (a lot) simpler if I would use json format for data structure? 
I am not limited in that regard ...

I have actually decided to describe my whole idea/plan here, since to 
continue from this point I will need help or guidance from someone more 
versed in lua than me.

The whole idea is to have in markdown source hooks like this:

``````markdown
Lets use a few acronyms, such as [!html] and [!xml].
``````

In the output, if the format would be .tex, acronym sequences would be 
translated to appropriate glossaries cmmands:

``````tex
Lets use a few acronyms, such as \gls{html} and \gls{xml}.
``````

For other formats it would do string replacement, say, for docx:

``````docx
Lets use a few acronyms, such as HTML and XML.
``````

Acronym definitions should be in an external file, in json or yaml format; 
example yaml:

``````yaml
xml:
- short: XML
- long: extensible markup language
``````

Now I would love to be able to reference this yaml file in defaults file, 
read data within and create some data structure out of it, ideally a 
dictionary (does lua even have that?).

I have now gotten into a point, in which I dont know how to load external 
data with pandoc lua filter, hence this question.

How far have I got (Still highly work in progress, certainly there are a 
lot of errors or unfinished bussinesses):

```````glossaries.lua
[[
Acronym sequence identifiers - beginning and end
]]
local acronymBegin = "[!"
local acronymEnd = "]"

[[
Acronym modifiers, for explicitly substituing long form, short form, or 
full - meaning long (short)
]]
local short = "<"
local long = ">"
local full = "*"

[[
Definition of `status` variables - those hold status information for 
acronym replacement (by command or plain text)
]]

local shortStatus = false
local longStatus = false
local fullStatus = false

[[
Matching function for each acronym command modifier. The idea is to store 
match results to `status` variables.
]]

local function matchShort(acronymSequence)
  if acronymSequence.match(short)
    return true
  end
end

local function matchLong(acronymSequence)
  if acronymSequence.match(long)
    return true
  end
end

local function matchFull(acronymSequence)
  if acronymSequence.match(full)
    return true
  end
end

[[
Acronym detection - matching for acronym pattern in Str element
]]

local function matchAcronym(acronymSequence)
  if string.match(acronymSequence, 
acronymBegin..'[<>*]?'..'(%a+)'..acronymEnd)
    return true, key -- "key" should hold acronym identifier, I hope ...
  else
    return false, nil
  end
end

[[
Core filter functions
]]

-- Match for every identifier and return command accordingly: not finished
local function processAcronym(acronymSequence)
  return command
end

-- Function connected to pandoc types, dealing with return values
local function Str(elem)
  matchAcronym(elem.text) = isAcronym, key
  
  if isAcronym = true then
    processAcronym(elem.text) = -- again, not finished
  else --not acronym, dont change the Str element
    return nil
  end
end
```````
Dne pátek 25. září 2020 v 15:06:34 UTC+2 uživatel BPJ napsal:

> This function is a hack which takes a file name, reads the file, makes 
> sure as well as it can that Pandoc will see the text as a YAML block, 
> parses the text with pandoc and returns the metadata of the result.
>
> ``````lua
> local function load_yaml (file_name)
>   local file = assert(io.open(file_name))
>   local text = assert(file:read('a'))
>   file:close()
>   -- remove leading/trailing whitespace and existing delimiters
>   text = text:gsub('^%s+',"")
>     :gsub('%s+$',"")
>     :gsub('^%-%-%-',"")
>     :gsub('%-%-%-$',"")
>     :gsub('%.%.%.$',"")
>   -- insert delimiters
>   text = "---\n" .. text .. "\n---\n"
>   -- parse the YAML
>   return pandoc.read(text, 'markdown').meta
> end
> ``````
>
> Note that this returns a metadata object, not a plain data data structure. 
> You will need another function to convert the metadata into a plain data 
> structure:
>
> ``````lua
> -- Turn Pandoc metadata into "plain" data:
> -- a table for Meta, MetaList, MetaMap;
> -- stringify all other elements;
> -- return anything else as-is.
> -- Either the whole metadata or a metadata item works.
> local meta_to_data
> do
>   meta_to_data = function (meta)
>     if 'table' == type(meta) then
>       local tag = meta.tag
>       if not(tag) or 'Meta' == tag or 'MetaList' == tag or 'MetaMap' == 
> tag then
>         local data = {}
>         for k,v in pairs(meta) do
>           data[k] = meta_to_data(v)
>         end
>         return data
>       else
>         return pandoc.utils.stringify(meta)
>       end
>     else
>       return meta
>     end
>   end
> end
> ``````
>
> Note:
> (1) pandoc.utils.stringify doesn't handle block content very well.
> (2) I just rewrote meta_to_data from memory but it should work OK.
>
>
> -- 
> Better --help|less than helpless
>
> Den fre 25 sep. 2020 13:37krulis....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <krulis....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 
> skrev:
>
>> Those are good news!
>>
>> But my question might have been a little confusing.
>> As a follow-up, how can load and access data provided in arbitraty yaml 
>> or json file with lua filter? Is there any helper function for that, or I 
>> should just use simple `open` or something ... ?
>> I am looking at lua filters part in pandoc docs; example: "Replacing 
>> placeholders with their metadata value;" which kinda similar to my case, 
>> but here are just flat values. I would like to specify a filepath to yaml 
>> file with actual definitions. 
>>
>> Dne pátek 25. září 2020 v 12:06:24 UTC+2 uživatel Albert Krewinkel napsal:
>>
>>> You should be able to access `mustache` as a normal metadata field if it 
>>> is nested under the field `metadata` in defaults.yaml: 
>>>
>>> --- 
>>> metadata: 
>>> mustache: ./src/varDefinitions.yaml 
>>> --- 
>>>
>>>
>>> krulis....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org writes: 
>>>
>>> > I would like to ask, whether it is possible for a lua filter to read 
>>> data 
>>> > from metadata block (or metadata file) or from defaults file? I looked 
>>> > around website documentation, also into `lua filter repository`, but 
>>> found 
>>> > no filter accessing metadata values in `defaults file` fields. 
>>> > I am working on a custom lua filter, for which I want to specify a 
>>> metadata 
>>> > field, which would point to external file with required definitions, 
>>> very 
>>> > similar to python filter `pandoc-mustache` github source here : 
>>> > <
>>> https://github.com/michaelstepner/pandoc-mustache/blob/master/pandoc_mustache/pandoc_mustache.py> 
>>>
>>> > 
>>> > `defaults.yaml`: 
>>> > 
>>> > ``` 
>>> > ... 
>>> > 
>>> > mustache: 
>>> > - ./src/varDefinitions.yaml 
>>> > 
>>> > ... 
>>> > ``` 
>>> > 
>>> > *Motivation:* Plans for this filter are to provide functionalities of 
>>> LaTeX 
>>> > `glossaries` package to `pandoc`. There are already some filters, 
>>> which 
>>> > take theyre own approach to the issue, but none of them gives full 
>>> acronym 
>>> > support specific to given format. 
>>> > Long-term plans would be to also provide support for term definitions 
>>> in 
>>> > `glossaries`-like style. 
>>> > 
>>> > Thanks for any guidance in this issue. 
>>> > Regards, Tomas 
>>>
>>>
>>> -- 
>>> Albert Krewinkel 
>>> GPG: 8eed e3e2 e8c5 6f18 81fe e836 388d c0b2 1f63 1124 
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pandoc-discus...-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pandoc-discuss/9dbbc23d-1cde-4c0b-898a-ecbe4679d614n%40googlegroups.com 
>> <https://groups.google.com/d/msgid/pandoc-discuss/9dbbc23d-1cde-4c0b-898a-ecbe4679d614n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/8cf956a4-8a5e-4b6d-a817-8d4a6bcbb272n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 13814 bytes --]