Lua filter to process chunkedhtml output

Lua filter to process chunkedhtml output

[ChrisD]

I would like to write a Lua filter to create HTMLHelp output from the new chunkedhtml output format. I'd like the filter to create the project (.hhp) and content or TOC (.hhc) files. After pandoc finishes, the make or batch file could run HtmlHelpWorkshop to create the output file.

Is the information in sitemap.json available to a Lua filter? How would I go about accessing it? This would be used to create the content file.

To create the project file, I would need a list of filenames to be included in the output file, including the images and any other resource files (which are not included in sitemap.json). At the time the filter is run, I don't believe the output folder or zip file exists yet. Is that information available some other way?

Thanks,

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/d5298c4a-9427-8f5b-c646-6370a3f998df%40intielectronics.com.

Re: Lua filter to process chunkedhtml output

[John MacFarlane]

sitemap.json is really for processing with external tools. If you're using a Lua filter, then you can use the new module
https://pandoc.org/lua-filters.html#module-pandoc.structure


> On Jan 19, 2023, at 5:44 AM, ChrisD <cd34-gg-4SSc53hpTiu9TMao6EloiEEOCMrvLtNR@public.gmane.org> wrote:
> 
> I would like to write a Lua filter to create HTMLHelp output from the new chunkedhtml output format. I'd like the filter to create the project (.hhp) and content or TOC (.hhc) files. After pandoc finishes, the make or batch file could run HtmlHelpWorkshop to create the output file.
> 
> Is the information in sitemap.json available to a Lua filter? How would I go about accessing it? This would be used to create the content file.
> 
> To create the project file, I would need a list of filenames to be included in the output file, including the images and any other resource files (which are not included in sitemap.json). At the time the filter is run, I don't believe the output folder or zip file exists yet. Is that information available some other way?
> 
> Thanks,
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/d5298c4a-9427-8f5b-c646-6370a3f998df%40intielectronics.com.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/1D22B433-211B-4033-8A63-F637F52B2008%40gmail.com.

Re: Lua filter to process chunkedhtml output

[John MacFarlane]

I should also note that sitemap.json is produced by the writer, so it isn't created when the Lua filter is running (after parsing, before rendering).


> On Jan 19, 2023, at 9:12 AM, John MacFarlane <fiddlosopher-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> sitemap.json is really for processing with external tools. If you're using a Lua filter, then you can use the new module
> https://pandoc.org/lua-filters.html#module-pandoc.structure
> 
> 
>> On Jan 19, 2023, at 5:44 AM, ChrisD <cd34-gg-4SSc53hpTiu9TMao6EloiEEOCMrvLtNR@public.gmane.org> wrote:
>> 
>> I would like to write a Lua filter to create HTMLHelp output from the new chunkedhtml output format. I'd like the filter to create the project (.hhp) and content or TOC (.hhc) files. After pandoc finishes, the make or batch file could run HtmlHelpWorkshop to create the output file.
>> 
>> Is the information in sitemap.json available to a Lua filter? How would I go about accessing it? This would be used to create the content file.
>> 
>> To create the project file, I would need a list of filenames to be included in the output file, including the images and any other resource files (which are not included in sitemap.json). At the time the filter is run, I don't believe the output folder or zip file exists yet. Is that information available some other way?
>> 
>> Thanks,
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/d5298c4a-9427-8f5b-c646-6370a3f998df%40intielectronics.com.
> 

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/57EF4CAC-77F6-4C71-B7C0-21B8F118DCBF%40gmail.com.

Re: Lua filter to process chunkedhtml output

[ChrisD]

On 1/19/2023 10:12 AM, John MacFarlane wrote:
> sitemap.json is really for processing with external tools. If you're using a Lua filter, then you can use the new module
> https://pandoc.org/lua-filters.html#module-pandoc.structure
I started playing with this, but quickly got confused.

Inside my lua filter, pandoc.structure is nil.  Is this available in a filter, or only in a custom writer? Should I be trying to write a custom writer rather than a filter?

Is there an example of getting or using the table of contents?

Also, I tried to print the value of PANDOC_WRITER_OPTIONS, but wlupton's logging.lua errors on it (bad argument #1 to 'for iterator' (table expected, got light userdata)). I the tried the debugging printer I have used in the past, kikito's inspect.lua, and it reports its value as " <userdata 1>". How do I access writer_options from a filter (for example, the toc-depth)? For what it's worth, I can print the value of PANDOC_READER_OPTIONS.

Thanks,

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/f761231d-87ea-6bfb-38c3-99eb15184263%40intielectronics.com.

Re: Lua filter to process chunkedhtml output

['William Lupton' via pandoc-discuss]

[-- Attachment #1: Type: text/plain, Size: 2563 bytes --]

Re this:

> Also, I tried to print the value of PANDOC_WRITER_OPTIONS, but wlupton's
logging.lua errors on it (bad argument #1 to 'for iterator' (table
expected, got light userdata))

I investigated, and parts of the writer options are "light userdata" (I
hadn't heard of that). I've committed and merged a fix that will report
such items as "<pointer>". You can now list most of the writer options
(just a few colors show as <pointer>).

I'm not sure whether the use of light userdata is intentional here (this
isn't a 3.0 thing; it was already the case in previous versions). Albert?

On Thu, 19 Jan 2023 at 21:32, ChrisD <cd34-gg-4SSc53hpTiu9TMao6EloiEEOCMrvLtNR@public.gmane.org> wrote:

> On 1/19/2023 10:12 AM, John MacFarlane wrote:
> > sitemap.json is really for processing with external tools. If you're
> using a Lua filter, then you can use the new module
> > https://pandoc.org/lua-filters.html#module-pandoc.structure
> I started playing with this, but quickly got confused.
>
> Inside my lua filter, pandoc.structure is nil.  Is this available in a
> filter, or only in a custom writer? Should I be trying to write a custom
> writer rather than a filter?
>
> Is there an example of getting or using the table of contents?
>
> Also, I tried to print the value of PANDOC_WRITER_OPTIONS, but wlupton's
> logging.lua errors on it (bad argument #1 to 'for iterator' (table
> expected, got light userdata)). I the tried the debugging printer I have
> used in the past, kikito's inspect.lua, and it reports its value as "
> <userdata 1>". How do I access writer_options from a filter (for example,
> the toc-depth)? For what it's worth, I can print the value of
> PANDOC_READER_OPTIONS.
>
> Thanks,
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/f761231d-87ea-6bfb-38c3-99eb15184263%40intielectronics.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAEe_xxiQAz2MVihALKGes6Ai%3DUrOPKgkvGq19ULiLsjBhTuTbQ%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 3731 bytes --]

Re: Lua filter to process chunkedhtml output

[Albert Krewinkel]

"'William Lupton' via pandoc-discuss" <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> writes:

> Re this:
>
>> Also, I tried to print the value of PANDOC_WRITER_OPTIONS, but
> wlupton's logging.lua errors on it (bad argument #1 to 'for iterator'
> (table expected, got light userdata))
>
> I investigated, and parts of the writer options are "light userdata"
> (I hadn't heard of that). I've committed and merged a fix that will
> report such items as "<pointer>". You can now list most of the writer
> options (just a few colors show as <pointer>).
>
> I'm not sure whether the use of light userdata is intentional here
> (this isn't a 3.0 thing; it was already the case in previous
> versions). Albert?

Let's call it semi-intentional: what happens is that we go via JSON to
create the Lua representation of the writer options. The problem is
that, if a JSON value is `null`, we can't use `nil` in Lua because that
would make the key "disappear", i.e., it would not show up when
iterating through the resulting table with `pairs`. So, in an effort to
gain compatibility with the very popular Lua library "cjson", we
represent `null` values as C null pointers. Hence the "light userdata".

The one solution I can think of is to set a custon `__pairs` metamethod
on PANDOC_WRITER_OPTIONS. That would allow us to set the missing values
to `nil`, while still having `pairs(PANDOC_WRITER_OPTIONS)` list all
supported keys. Please feel free to raise an issue for this on GitHub.


-- 
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Lua filter to process chunkedhtml output

[ChrisD]

On 1/20/2023 2:54 AM, 'William Lupton' via pandoc-discuss wrote:
> I investigated, and parts of the writer options are "light userdata" (I hadn't heard of that). I've committed and merged a fix that will report such items as "<pointer>". You can now list most of the writer options (just a few colors show as <pointer>).

Your update works well. (I didn't know about light userdata either.) Thanks!

Re: Lua filter to process chunkedhtml output

['William Lupton' via pandoc-discuss]

[-- Attachment #1: Type: text/plain, Size: 1847 bytes --]

Thanks Albert. I expect I'll raise an issue. I assume that this is the only
use of light userdata?

Looking more closely (why didn't I do this before?) I see that tostring()
returns 'userdata: 0x0' for these values, so I think it makes sense to
report them as one of:

   - Whatever tostring() returns
   - As above with the leading 'userdata: ' removed                   <---
   probably my favourite
   - As above with '0x0' reported as 'nil' (not sure about this)


On Fri, 20 Jan 2023 at 16:05, ChrisD <cd34-gg-4SSc53hpTiu9TMao6EloiEEOCMrvLtNR@public.gmane.org> wrote:

> On 1/20/2023 2:54 AM, 'William Lupton' via pandoc-discuss wrote:
> > I investigated, and parts of the writer options are "light userdata" (I
> hadn't heard of that). I've committed and merged a fix that will report
> such items as "<pointer>". You can now list most of the writer options
> (just a few colors show as <pointer>).
>
> Your update works well. (I didn't know about light userdata either.)
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/fbca8e05-fed7-39e1-08f0-0498c399f33f%40intielectronics.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAEe_xxiJdneKg%2Buqja3tdaWzQSWmc5NfO0aMFKaUGa_W1U5%2Bqg%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 2866 bytes --]

Re: Lua filter to process chunkedhtml output

['William Lupton' via pandoc-discuss]

[-- Attachment #1: Type: text/plain, Size: 2199 bytes --]

Update: Until today, logging.lua assumed that pairs() would work on all
userdata. Option 2 (my favourite) should work well if there are ever any
other types of userdata (not sure whether there ever will be).

On Fri, 20 Jan 2023 at 16:29, William Lupton <wlupton-QSt+ys/nuMyEUIsrzH9SikB+6BGkLq7r@public.gmane.org>
wrote:

> Thanks Albert. I expect I'll raise an issue. I assume that this is the
> only use of light userdata?
>
> Looking more closely (why didn't I do this before?) I see that tostring()
> returns 'userdata: 0x0' for these values, so I think it makes sense to
> report them as one of:
>
>    - Whatever tostring() returns
>    - As above with the leading 'userdata: ' removed
>     <--- probably my favourite
>    - As above with '0x0' reported as 'nil' (not sure about this)
>
>
> On Fri, 20 Jan 2023 at 16:05, ChrisD <cd34-gg-4SSc53hpTiu9TMao6EloiEEOCMrvLtNR@public.gmane.org> wrote:
>
>> On 1/20/2023 2:54 AM, 'William Lupton' via pandoc-discuss wrote:
>> > I investigated, and parts of the writer options are "light userdata" (I
>> hadn't heard of that). I've committed and merged a fix that will report
>> such items as "<pointer>". You can now list most of the writer options
>> (just a few colors show as <pointer>).
>>
>> Your update works well. (I didn't know about light userdata either.)
>> Thanks!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "pandoc-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/pandoc-discuss/fbca8e05-fed7-39e1-08f0-0498c399f33f%40intielectronics.com
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAEe_xxhhsG_NA%2BPL%2B5A2TyHHqNOC-%3Dsdi1jX-NYunf9fsHtH1A%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 3525 bytes --]

Re: Lua filter to process chunkedhtml output

['William Lupton' via pandoc-discuss]

[-- Attachment #1: Type: text/plain, Size: 2984 bytes --]

pandoc.utils.type() return values seem to be a little inconsistent. For
example, it can return:

   - 'List' ... for pandoc.List (note: logging.lua leaves this alone)
   - 'pandoc Xxx' ... for pandoc.Row etc. (note: logging.lua replaces the
   space with a dot)
   - 'HsLua.JSON.array' ... presumably for arrays that have come via a JSON
   library

I was toying with not indicating HsLua.JSON.array (i.e., showing it like
any other table) but perhaps it's better just to report things as they are.
Can any other HsLua.JSON.xxx values be returned, e.g., why not
HsLua.JSON.object or indeed HsLua.JSON.{boolean,number,string,null}?

On Fri, 20 Jan 2023 at 16:33, William Lupton <wlupton-QSt+ys/nuMyEUIsrzH9SikB+6BGkLq7r@public.gmane.org>
wrote:

> Update: Until today, logging.lua assumed that pairs() would work on all
> userdata. Option 2 (my favourite) should work well if there are ever any
> other types of userdata (not sure whether there ever will be).
>
> On Fri, 20 Jan 2023 at 16:29, William Lupton <wlupton-QSt+ys/nuMyEUIsrzH9SikB+6BGkLq7r@public.gmane.org>
> wrote:
>
>> Thanks Albert. I expect I'll raise an issue. I assume that this is the
>> only use of light userdata?
>>
>> Looking more closely (why didn't I do this before?) I see that tostring()
>> returns 'userdata: 0x0' for these values, so I think it makes sense to
>> report them as one of:
>>
>>    - Whatever tostring() returns
>>    - As above with the leading 'userdata: ' removed
>>     <--- probably my favourite
>>    - As above with '0x0' reported as 'nil' (not sure about this)
>>
>>
>> On Fri, 20 Jan 2023 at 16:05, ChrisD <cd34-gg-4SSc53hpTiu9TMao6EloiEEOCMrvLtNR@public.gmane.org> wrote:
>>
>>> On 1/20/2023 2:54 AM, 'William Lupton' via pandoc-discuss wrote:
>>> > I investigated, and parts of the writer options are "light userdata"
>>> (I hadn't heard of that). I've committed and merged a fix that will report
>>> such items as "<pointer>". You can now list most of the writer options
>>> (just a few colors show as <pointer>).
>>>
>>> Your update works well. (I didn't know about light userdata either.)
>>> Thanks!
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "pandoc-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/pandoc-discuss/fbca8e05-fed7-39e1-08f0-0498c399f33f%40intielectronics.com
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAEe_xxjkBZfgeh5jDbZNDNrFbG4SKTA0S9qjzaEXgxTr5BZVNw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 4624 bytes --]

Re: Lua filter to process chunkedhtml output

[Albert Krewinkel]

"'William Lupton' via pandoc-discuss" <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> writes:

> pandoc.utils.type() return values seem to be a little inconsistent.
> For example, it can return:
>
>     'List' ... for pandoc.List (note: logging.lua leaves this alone)
>     'pandoc Xxx' ... for pandoc.Row etc. (note: logging.lua replaces
>     the space with a dot)
>     'HsLua.JSON.array' ... presumably for arrays that have come via a
>     JSON library

Thank you for digging and asking! It's great to have more eyeballs in
this part of the code.

A bit of background: all types have a corresponding entry in the Lua
"registry", a global table that contains all sorts of info. The Lua
manual states "you should use as key a string containing your library
name", hence the "pandoc" and "HsLua" prefixes. Additional
inconsistencies are cause by the fact that I didn't follow the Lua
manual's advice in the beginning, as name clashes with other packages
where not really a concern of mine.

> I was toying with not indicating HsLua.JSON.array (i.e., showing it
> like any other table) but perhaps it's better just to report things
> as they are. Can any other HsLua.JSON.xxx values be returned, e.g.,
> why not HsLua.JSON.object or indeed HsLua.JSON.
> {boolean,number,string,null}?

There's a technical reason for this: we must be able to round-trip
from JSON to Lua to JSON. Both JSON arrays and objects are represented
as Lua tables, but how can we know whether an empty Lua table `{}`
represents an empty array or an empty object? We solve this by assigning
a metatable/type to array values.

I guess we could also do this for JSON objects, but the advantage is
less clear in that case: being able to add metamethods like `insert`
makes sense for lists, but not for objects: String fields can shadow any
method, which would most likely lead to subtle bugs. E.g.,
`obj:merge(obj2)` would stop working if `obj` has a key `merge`.
Adding a metatable to objects would therefore be little more than
performance degrading overhead.

The other JSON types, boolean, numbers, and strings, map directly to the
respective Lua types of the same name, and pandoc.utils.type will report
the Lua type name in those cases.

Anyway, please do open issues for any inconsistencies you find. We may
want to keep some of those old names to keep error messages more
readable, but consistency would be nice to have, too.

> On Fri, 20 Jan 2023 at 16:33, William Lupton <
> wlupton-QSt+ys/nuMyEUIsrzH9SikB+6BGkLq7r@public.gmane.org> wrote:
>
>     Update: Until today, logging.lua assumed that pairs() would work
>     on all userdata. Option 2 (my favourite) should work well if
>     there are ever any other types of userdata (not sure whether
>     there ever will be).
>    
>     On Fri, 20 Jan 2023 at 16:29, William Lupton <
>     wlupton-QSt+ys/nuMyEUIsrzH9SikB+6BGkLq7r@public.gmane.org> wrote:
>    
>         Thanks Albert. I expect I'll raise an issue. I assume
>         that this is the only use of light userdata?
>        
>         Looking more closely (why didn't I do this before?) I see
>         that tostring() returns 'userdata: 0x0' for these values, so
>         I think it makes sense to report them as one of:
>             Whatever tostring() returns
>             As above with the leading 'userdata: ' removed           
>                    <--- probably my favourite
>             As above with '0x0' reported as 'nil' (not sure about
>             this)
>        
>         On Fri, 20 Jan 2023 at 16:05, ChrisD <
>         cd34-gg-4SSc53hpTiu9TMao6EloiEEOCMrvLtNR@public.gmane.org> wrote:
>        
>             On 1/20/2023 2:54 AM, 'William Lupton' via pandoc-discuss
>             wrote:
>             > I investigated, and parts of the writer options are
>             "light userdata" (I hadn't heard of that). I've committed
>             and merged a fix that will report such items as "
>             <pointer>". You can now list most of the writer options
>             (just a few colors show as <pointer>).
>            
>             Your update works well. (I didn't know about light
>             userdata either.) Thanks!
>            
>             --
>             You received this message because you are subscribed to
>             the Google Groups "pandoc-discuss" group.
>             To unsubscribe from this group and stop receiving emails
>             from it, send an email to 
>             pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
>             To view this discussion on the web visit https://
>             groups.google.com/d/msgid/pandoc-discuss/
>             fbca8e05-fed7-39e1-08f0-0498c399f33f%40intielectronics.com
>             .


-- 
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/87fsc4koud.fsf%40zeitkraut.de.

logging.lua and 'light userdata'

['William Lupton' via pandoc-discuss]

[-- Attachment #1: Type: text/plain, Size: 8100 bytes --]

(Creating a new thread with cleaned-up history. Sorry for hijacking your
thread ChrisD!)

I've merged a small update to logging.lua (just removing the leading
'userdata: ' when reporting light userdata values).

Thanks for the explanation Albert. I don't suppose that you really want to
change the behaviour of pandoc.utils.type(), so I'm not really inclined to
create an issue suggesting that you should do so. However (if you like) I
could create an issue suggesting that it be slightly more fully documented
(e.g., giving some examples of what it would return for various pandoc.xxx
types).

As for your other earlier suggestion (creating an issue to suggest a custom
`__pairs` metamethod on PANDOC_WRITER_OPTIONS), would that fix the issue?
The 0x0 values aren't associated with top-level keys of
PANDOC_WRITER_OPTIONS, they're (for example) within highlight_style (see
below). I can't help feeling that we should either do nothing or else do
something that would work for arbitrary structures (not just for
PANDOC_WRITER_OPTIONS). Given that I've worked around the problem in
logging.lua I'd be quite happy to do nothing.

A specific point about the items that have 0x0 values: these are all items
that have no value, so is it the end of the world if they are simply
omitted? Mightn't that be more whatever-the-lua-equivalent-of-pythonic-is?
Yes pairs() won't visit them, but an attempt to access them will return nil
as expected.

WriterOptions {
  cite_method: "citeproc"
  ...
  highlight_style: {
    background-color: 0x0
    line-number-background-color: 0x0
    line-number-color: "#aaaaaa"
    text-color: 0x0
    ...

On Sat, 21 Jan 2023 at 13:18, Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org>
wrote:

>
> "'William Lupton' via pandoc-discuss" <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>
> writes:
>
> > pandoc.utils.type() return values seem to be a little inconsistent.
> > For example, it can return:
> >
> >     'List' ... for pandoc.List (note: logging.lua leaves this alone)
> >     'pandoc Xxx' ... for pandoc.Row etc. (note: logging.lua replaces
> >     the space with a dot)
> >     'HsLua.JSON.array' ... presumably for arrays that have come via a
> >     JSON library
>
> Thank you for digging and asking! It's great to have more eyeballs in
> this part of the code.
>
> A bit of background: all types have a corresponding entry in the Lua
> "registry", a global table that contains all sorts of info. The Lua
> manual states "you should use as key a string containing your library
> name", hence the "pandoc" and "HsLua" prefixes. Additional
> inconsistencies are cause by the fact that I didn't follow the Lua
> manual's advice in the beginning, as name clashes with other packages
> where not really a concern of mine.
>
> > I was toying with not indicating HsLua.JSON.array (i.e., showing it
> > like any other table) but perhaps it's better just to report things
> > as they are. Can any other HsLua.JSON.xxx values be returned, e.g.,
> > why not HsLua.JSON.object or indeed HsLua.JSON.
> > {boolean,number,string,null}?
>
> There's a technical reason for this: we must be able to round-trip
> from JSON to Lua to JSON. Both JSON arrays and objects are represented
> as Lua tables, but how can we know whether an empty Lua table `{}`
> represents an empty array or an empty object? We solve this by assigning
> a metatable/type to array values.
>
> I guess we could also do this for JSON objects, but the advantage is
> less clear in that case: being able to add metamethods like `insert`
> makes sense for lists, but not for objects: String fields can shadow any
> method, which would most likely lead to subtle bugs. E.g.,
> `obj:merge(obj2)` would stop working if `obj` has a key `merge`.
> Adding a metatable to objects would therefore be little more than
> performance degrading overhead.
>
> The other JSON types, boolean, numbers, and strings, map directly to the
> respective Lua types of the same name, and pandoc.utils.type will report
> the Lua type name in those cases.
>
> Anyway, please do open issues for any inconsistencies you find. We may
> want to keep some of those old names to keep error messages more
> readable, but consistency would be nice to have, too.
>
> > On Fri, 20 Jan 2023 at 16:33, William Lupton <
> > wlupton-QSt+ys/nuMyEUIsrzH9SikB+6BGkLq7r@public.gmane.org> wrote:
> >
> >     Update: Until today, logging.lua assumed that pairs() would work
> >     on all userdata. Option 2 (my favourite) should work well if
> >     there are ever any other types of userdata (not sure whether
> >     there ever will be).
> >
> >     On Fri, 20 Jan 2023 at 16:29, William Lupton <
> >     wlupton-QSt+ys/nuMyEUIsrzH9SikB+6BGkLq7r@public.gmane.org> wrote:
> >
> >         Thanks Albert. I expect I'll raise an issue. I assume
> >         that this is the only use of light userdata?
> >
> >         Looking more closely (why didn't I do this before?) I see
> >         that tostring() returns 'userdata: 0x0' for these values, so
> >         I think it makes sense to report them as one of:
> >             Whatever tostring() returns
> >             As above with the leading 'userdata: ' removed
> >                    <--- probably my favourite
> >             As above with '0x0' reported as 'nil' (not sure about
> >             this)
> >
> >         On Fri, 20 Jan 2023 at 16:05, ChrisD <
> >         cd34-gg-4SSc53hpTiu9TMao6EloiEEOCMrvLtNR@public.gmane.org> wrote:
> >
> >             On 1/20/2023 2:54 AM, 'William Lupton' via pandoc-discuss
> >             wrote:
> >             > I investigated, and parts of the writer options are
> >             "light userdata" (I hadn't heard of that). I've committed
> >             and merged a fix that will report such items as "
> >             <pointer>". You can now list most of the writer options
> >             (just a few colors show as <pointer>).
> >
> >             Your update works well. (I didn't know about light
> >             userdata either.) Thanks!
>

On Fri, 20 Jan 2023 at 15:59, Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org>
wrote:

>
> "'William Lupton' via pandoc-discuss" <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>
> writes:
>
> > Re this:
> >
> >> Also, I tried to print the value of PANDOC_WRITER_OPTIONS, but
> > wlupton's logging.lua errors on it (bad argument #1 to 'for iterator'
> > (table expected, got light userdata))
> >
> > I investigated, and parts of the writer options are "light userdata"
> > (I hadn't heard of that). I've committed and merged a fix that will
> > report such items as "<pointer>". You can now list most of the writer
> > options (just a few colors show as <pointer>).
> >
> > I'm not sure whether the use of light userdata is intentional here
> > (this isn't a 3.0 thing; it was already the case in previous
> > versions). Albert?
>
> Let's call it semi-intentional: what happens is that we go via JSON to
> create the Lua representation of the writer options. The problem is
> that, if a JSON value is `null`, we can't use `nil` in Lua because that
> would make the key "disappear", i.e., it would not show up when
> iterating through the resulting table with `pairs`. So, in an effort to
> gain compatibility with the very popular Lua library "cjson", we
> represent `null` values as C null pointers. Hence the "light userdata".
>
> The one solution I can think of is to set a custon `__pairs` metamethod
> on PANDOC_WRITER_OPTIONS. That would allow us to set the missing values
> to `nil`, while still having `pairs(PANDOC_WRITER_OPTIONS)` list all
> supported keys. Please feel free to raise an issue for this on GitHub.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CAEe_xxjsmxwx4B_f1j4buVT8P%2BcdVdVw4q8dbtxiuJNoXsNweQ%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 10840 bytes --]

Re: logging.lua and 'light userdata'

[Albert Krewinkel]

I've opened https://github.com/jgm/pandoc/issues/8574 for this. We can't
really change the return values of `pandoc.utils.types` for Inlines and
Blocks, as that would break a lot of filters. But we can still strive to
make the output more consistent.

"'William Lupton' via pandoc-discuss" <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> writes:

> (Creating a new thread with cleaned-up history. Sorry for hijacking
> your thread ChrisD!)
>
> I've merged a small update to logging.lua (just removing the leading
> 'userdata: ' when reporting light userdata values).
>
> Thanks for the explanation Albert. I don't suppose that you really
> want to change the behaviour of pandoc.utils.type(), so I'm not
> really inclined to create an issue suggesting that you should do so.
> However (if you like) I could create an issue suggesting that it be
> slightly more fully documented (e.g., giving some examples of what it
> would return for various pandoc.xxx types).
>
> As for your other earlier suggestion (creating an issue to suggest a
> custom `__pairs` metamethod on PANDOC_WRITER_OPTIONS), would that fix
> the issue? The 0x0 values aren't associated with top-level keys of
> PANDOC_WRITER_OPTIONS, they're (for example) within highlight_style
> (see below). I can't help feeling that we should either do nothing or
> else do something that would work for arbitrary structures (not just
> for PANDOC_WRITER_OPTIONS). Given that I've worked around the problem
> in logging.lua I'd be quite happy to do nothing.
>
> A specific point about the items that have 0x0 values: these are all
> items that have no value, so is it the end of the world if they are
> simply omitted? Mightn't that be more
> whatever-the-lua-equivalent-of-pythonic-is? Yes pairs() won't visit
> them, but an attempt to access them will return nil as expected.
>
> WriterOptions {
>   cite_method: "citeproc"
>   ...
>   highlight_style: {
>     background-color: 0x0
>     line-number-background-color: 0x0
>     line-number-color: "#aaaaaa"
>     text-color: 0x0
>     ...
>
> On Sat, 21 Jan 2023 at 13:18, Albert Krewinkel <
> albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> wrote:
>
>    
>     "'William Lupton' via pandoc-discuss" <
>     pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> writes:
>    
>     > pandoc.utils.type() return values seem to be a little
>     inconsistent.
>     > For example, it can return:
>     >
>     >     'List' ... for pandoc.List (note: logging.lua leaves this
>     alone)
>     >     'pandoc Xxx' ... for pandoc.Row etc. (note: logging.lua
>     replaces
>     >     the space with a dot)
>     >     'HsLua.JSON.array' ... presumably for arrays that have come
>     via a
>     >     JSON library
>    
>     Thank you for digging and asking! It's great to have more
>     eyeballs in
>     this part of the code.
>    
>     A bit of background: all types have a corresponding entry in the
>     Lua
>     "registry", a global table that contains all sorts of info. The
>     Lua
>     manual states "you should use as key a string containing your
>     library
>     name", hence the "pandoc" and "HsLua" prefixes. Additional
>     inconsistencies are cause by the fact that I didn't follow the
>     Lua
>     manual's advice in the beginning, as name clashes with other
>     packages
>     where not really a concern of mine.
>    
>     > I was toying with not indicating HsLua.JSON.array (i.e.,
>     showing it
>     > like any other table) but perhaps it's better just to report
>     things
>     > as they are. Can any other HsLua.JSON.xxx values be returned,
>     e.g.,
>     > why not HsLua.JSON.object or indeed HsLua.JSON.
>     > {boolean,number,string,null}?
>    
>     There's a technical reason for this: we must be able to
>     round-trip
>     from JSON to Lua to JSON. Both JSON arrays and objects are
>     represented
>     as Lua tables, but how can we know whether an empty Lua table `{}
>     `
>     represents an empty array or an empty object? We solve this by
>     assigning
>     a metatable/type to array values.
>    
>     I guess we could also do this for JSON objects, but the advantage
>     is
>     less clear in that case: being able to add metamethods like
>     `insert`
>     makes sense for lists, but not for objects: String fields can
>     shadow any
>     method, which would most likely lead to subtle bugs. E.g.,
>     `obj:merge(obj2)` would stop working if `obj` has a key `merge`.
>     Adding a metatable to objects would therefore be little more than
>     performance degrading overhead.
>    
>     The other JSON types, boolean, numbers, and strings, map directly
>     to the
>     respective Lua types of the same name, and pandoc.utils.type will
>     report
>     the Lua type name in those cases.
>    
>     Anyway, please do open issues for any inconsistencies you find.
>     We may
>     want to keep some of those old names to keep error messages more
>     readable, but consistency would be nice to have, too.
>    
>     > On Fri, 20 Jan 2023 at 16:33, William Lupton <
>     > wlupton-QSt+ys/nuMyEUIsrzH9SikB+6BGkLq7r@public.gmane.org> wrote:
>     >
>     >     Update: Until today, logging.lua assumed that pairs() would
>     work
>     >     on all userdata. Option 2 (my favourite) should work well
>     if
>     >     there are ever any other types of userdata (not sure
>     whether
>     >     there ever will be).
>     >   
>     >     On Fri, 20 Jan 2023 at 16:29, William Lupton <
>     >     wlupton-QSt+ys/nuMyEUIsrzH9SikB+6BGkLq7r@public.gmane.org> wrote:
>     >   
>     >         Thanks Albert. I expect I'll raise an issue. I assume
>     >         that this is the only use of light userdata?
>     >       
>     >         Looking more closely (why didn't I do this before?) I
>     see
>     >         that tostring() returns 'userdata: 0x0' for these
>     values, so
>     >         I think it makes sense to report them as one of:
>     >             Whatever tostring() returns
>     >             As above with the leading 'userdata: ' removed     
>          
>     >                    <--- probably my favourite
>     >             As above with '0x0' reported as 'nil' (not sure
>     about
>     >             this)
>     >       
>     >         On Fri, 20 Jan 2023 at 16:05, ChrisD <
>     >         cd34-gg-4SSc53hpTiu9TMao6EloiEEOCMrvLtNR@public.gmane.org> wrote:
>     >       
>     >             On 1/20/2023 2:54 AM, 'William Lupton' via
>     pandoc-discuss
>     >             wrote:
>     >             > I investigated, and parts of the writer options
>     are
>     >             "light userdata" (I hadn't heard of that). I've
>     committed
>     >             and merged a fix that will report such items as "
>     >             <pointer>". You can now list most of the writer
>     options
>     >             (just a few colors show as <pointer>).
>     >           
>     >             Your update works well. (I didn't know about light
>     >             userdata either.) Thanks!
>
>
> On Fri, 20 Jan 2023 at 15:59, Albert Krewinkel <
> albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> wrote:
>
>    
>     "'William Lupton' via pandoc-discuss" <
>     pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> writes:
>    
>     > Re this:
>     >
>     >> Also, I tried to print the value of PANDOC_WRITER_OPTIONS, but
>     > wlupton's logging.lua errors on it (bad argument #1 to 'for
>     iterator'
>     > (table expected, got light userdata))
>     >
>     > I investigated, and parts of the writer options are "light
>     userdata"
>     > (I hadn't heard of that). I've committed and merged a fix that
>     will
>     > report such items as "<pointer>". You can now list most of the
>     writer
>     > options (just a few colors show as <pointer>).
>     >
>     > I'm not sure whether the use of light userdata is intentional
>     here
>     > (this isn't a 3.0 thing; it was already the case in previous
>     > versions). Albert?
>    
>     Let's call it semi-intentional: what happens is that we go via
>     JSON to
>     create the Lua representation of the writer options. The problem
>     is
>     that, if a JSON value is `null`, we can't use `nil` in Lua
>     because that
>     would make the key "disappear", i.e., it would not show up when
>     iterating through the resulting table with `pairs`. So, in an
>     effort to
>     gain compatibility with the very popular Lua library "cjson", we
>     represent `null` values as C null pointers. Hence the "light
>     userdata".
>    
>     The one solution I can think of is to set a custon `__pairs`
>     metamethod
>     on PANDOC_WRITER_OPTIONS. That would allow us to set the missing
>     values
>     to `nil`, while still having `pairs(PANDOC_WRITER_OPTIONS)` list
>     all
>     supported keys. Please feel free to raise an issue for this on
>     GitHub.
>
>  


-- 
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/87bkmpldx5.fsf%40zeitkraut.de.