Re: logging.lua and 'light userdata'

[Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org>]

I've opened https://github.com/jgm/pandoc/issues/8574 for this. We can't
really change the return values of `pandoc.utils.types` for Inlines and
Blocks, as that would break a lot of filters. But we can still strive to
make the output more consistent.

"'William Lupton' via pandoc-discuss" <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> writes:

> (Creating a new thread with cleaned-up history. Sorry for hijacking
> your thread ChrisD!)
>
> I've merged a small update to logging.lua (just removing the leading
> 'userdata: ' when reporting light userdata values).
>
> Thanks for the explanation Albert. I don't suppose that you really
> want to change the behaviour of pandoc.utils.type(), so I'm not
> really inclined to create an issue suggesting that you should do so.
> However (if you like) I could create an issue suggesting that it be
> slightly more fully documented (e.g., giving some examples of what it
> would return for various pandoc.xxx types).
>
> As for your other earlier suggestion (creating an issue to suggest a
> custom `__pairs` metamethod on PANDOC_WRITER_OPTIONS), would that fix
> the issue? The 0x0 values aren't associated with top-level keys of
> PANDOC_WRITER_OPTIONS, they're (for example) within highlight_style
> (see below). I can't help feeling that we should either do nothing or
> else do something that would work for arbitrary structures (not just
> for PANDOC_WRITER_OPTIONS). Given that I've worked around the problem
> in logging.lua I'd be quite happy to do nothing.
>
> A specific point about the items that have 0x0 values: these are all
> items that have no value, so is it the end of the world if they are
> simply omitted? Mightn't that be more
> whatever-the-lua-equivalent-of-pythonic-is? Yes pairs() won't visit
> them, but an attempt to access them will return nil as expected.
>
> WriterOptions {
>   cite_method: "citeproc"
>   ...
>   highlight_style: {
>     background-color: 0x0
>     line-number-background-color: 0x0
>     line-number-color: "#aaaaaa"
>     text-color: 0x0
>     ...
>
> On Sat, 21 Jan 2023 at 13:18, Albert Krewinkel <
> albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> wrote:
>
>    
>     "'William Lupton' via pandoc-discuss" <
>     pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> writes:
>    
>     > pandoc.utils.type() return values seem to be a little
>     inconsistent.
>     > For example, it can return:
>     >
>     >     'List' ... for pandoc.List (note: logging.lua leaves this
>     alone)
>     >     'pandoc Xxx' ... for pandoc.Row etc. (note: logging.lua
>     replaces
>     >     the space with a dot)
>     >     'HsLua.JSON.array' ... presumably for arrays that have come
>     via a
>     >     JSON library
>    
>     Thank you for digging and asking! It's great to have more
>     eyeballs in
>     this part of the code.
>    
>     A bit of background: all types have a corresponding entry in the
>     Lua
>     "registry", a global table that contains all sorts of info. The
>     Lua
>     manual states "you should use as key a string containing your
>     library
>     name", hence the "pandoc" and "HsLua" prefixes. Additional
>     inconsistencies are cause by the fact that I didn't follow the
>     Lua
>     manual's advice in the beginning, as name clashes with other
>     packages
>     where not really a concern of mine.
>    
>     > I was toying with not indicating HsLua.JSON.array (i.e.,
>     showing it
>     > like any other table) but perhaps it's better just to report
>     things
>     > as they are. Can any other HsLua.JSON.xxx values be returned,
>     e.g.,
>     > why not HsLua.JSON.object or indeed HsLua.JSON.
>     > {boolean,number,string,null}?
>    
>     There's a technical reason for this: we must be able to
>     round-trip
>     from JSON to Lua to JSON. Both JSON arrays and objects are
>     represented
>     as Lua tables, but how can we know whether an empty Lua table `{}
>     `
>     represents an empty array or an empty object? We solve this by
>     assigning
>     a metatable/type to array values.
>    
>     I guess we could also do this for JSON objects, but the advantage
>     is
>     less clear in that case: being able to add metamethods like
>     `insert`
>     makes sense for lists, but not for objects: String fields can
>     shadow any
>     method, which would most likely lead to subtle bugs. E.g.,
>     `obj:merge(obj2)` would stop working if `obj` has a key `merge`.
>     Adding a metatable to objects would therefore be little more than
>     performance degrading overhead.
>    
>     The other JSON types, boolean, numbers, and strings, map directly
>     to the
>     respective Lua types of the same name, and pandoc.utils.type will
>     report
>     the Lua type name in those cases.
>    
>     Anyway, please do open issues for any inconsistencies you find.
>     We may
>     want to keep some of those old names to keep error messages more
>     readable, but consistency would be nice to have, too.
>    
>     > On Fri, 20 Jan 2023 at 16:33, William Lupton <
>     > wlupton-QSt+ys/nuMyEUIsrzH9SikB+6BGkLq7r@public.gmane.org> wrote:
>     >
>     >     Update: Until today, logging.lua assumed that pairs() would
>     work
>     >     on all userdata. Option 2 (my favourite) should work well
>     if
>     >     there are ever any other types of userdata (not sure
>     whether
>     >     there ever will be).
>     >   
>     >     On Fri, 20 Jan 2023 at 16:29, William Lupton <
>     >     wlupton-QSt+ys/nuMyEUIsrzH9SikB+6BGkLq7r@public.gmane.org> wrote:
>     >   
>     >         Thanks Albert. I expect I'll raise an issue. I assume
>     >         that this is the only use of light userdata?
>     >       
>     >         Looking more closely (why didn't I do this before?) I
>     see
>     >         that tostring() returns 'userdata: 0x0' for these
>     values, so
>     >         I think it makes sense to report them as one of:
>     >             Whatever tostring() returns
>     >             As above with the leading 'userdata: ' removed     
>          
>     >                    <--- probably my favourite
>     >             As above with '0x0' reported as 'nil' (not sure
>     about
>     >             this)
>     >       
>     >         On Fri, 20 Jan 2023 at 16:05, ChrisD <
>     >         cd34-gg-4SSc53hpTiu9TMao6EloiEEOCMrvLtNR@public.gmane.org> wrote:
>     >       
>     >             On 1/20/2023 2:54 AM, 'William Lupton' via
>     pandoc-discuss
>     >             wrote:
>     >             > I investigated, and parts of the writer options
>     are
>     >             "light userdata" (I hadn't heard of that). I've
>     committed
>     >             and merged a fix that will report such items as "
>     >             <pointer>". You can now list most of the writer
>     options
>     >             (just a few colors show as <pointer>).
>     >           
>     >             Your update works well. (I didn't know about light
>     >             userdata either.) Thanks!
>
>
> On Fri, 20 Jan 2023 at 15:59, Albert Krewinkel <
> albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> wrote:
>
>    
>     "'William Lupton' via pandoc-discuss" <
>     pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> writes:
>    
>     > Re this:
>     >
>     >> Also, I tried to print the value of PANDOC_WRITER_OPTIONS, but
>     > wlupton's logging.lua errors on it (bad argument #1 to 'for
>     iterator'
>     > (table expected, got light userdata))
>     >
>     > I investigated, and parts of the writer options are "light
>     userdata"
>     > (I hadn't heard of that). I've committed and merged a fix that
>     will
>     > report such items as "<pointer>". You can now list most of the
>     writer
>     > options (just a few colors show as <pointer>).
>     >
>     > I'm not sure whether the use of light userdata is intentional
>     here
>     > (this isn't a 3.0 thing; it was already the case in previous
>     > versions). Albert?
>    
>     Let's call it semi-intentional: what happens is that we go via
>     JSON to
>     create the Lua representation of the writer options. The problem
>     is
>     that, if a JSON value is `null`, we can't use `nil` in Lua
>     because that
>     would make the key "disappear", i.e., it would not show up when
>     iterating through the resulting table with `pairs`. So, in an
>     effort to
>     gain compatibility with the very popular Lua library "cjson", we
>     represent `null` values as C null pointers. Hence the "light
>     userdata".
>    
>     The one solution I can think of is to set a custon `__pairs`
>     metamethod
>     on PANDOC_WRITER_OPTIONS. That would allow us to set the missing
>     values
>     to `nil`, while still having `pairs(PANDOC_WRITER_OPTIONS)` list
>     all
>     supported keys. Please feel free to raise an issue for this on
>     GitHub.
>
>  


-- 
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/87bkmpldx5.fsf%40zeitkraut.de.