Is this a bug?

Is this a bug?

[Bücherregal Domi]

[-- Attachment #1.1: Type: text/plain, Size: 1289 bytes --]


Good day, I'm fairly new to Pandoc, so I'm not sure if this is a bug or 
expected behavior. 

Executing, `echo "**line1**" | pandoc -t native`

Outputs: `[Para [Strong [Str "line1"]]]`

So far so good. Now, with the following lua filter stored in a file called 
`lua1.lua`

```lua
function Strong(elem)
  return pandoc.Str(elem.content)
end
```

Executing `echo "**line1**" | pandoc -t native --lua-filter lua1.lua`

Outputs: `[Para []]`

Which means a blank document, because the content "line1" was deleted, it 
doesn't exist anymore, is this a bug? Compare with the following.

Modifying the file `lua1.lua` as,

```lua
function Strong(elem)
  return pandoc.Str("hello1")
end
```

Executing again `echo "**line1**" | pandoc -t native --lua-filter lua1.lua`

Outputs: `[Para [Str "hello1"]]`

This one works, why does the first one not work? Shouldn't both work? Thank 
you.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/e2b36f5d-7c73-4160-a2d1-20b57c49f129n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1953 bytes --]

Re: Is this a bug?

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 2483 bytes --]

Den sön 28 mars 2021 10:01Bücherregal Domi <gamezaerza-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> skrev:

>
> Good day, I'm fairly new to Pandoc, so I'm not sure if this is a bug or
> expected behavior.
>
> Executing, `echo "**line1**" | pandoc -t native`
>
> Outputs: `[Para [Strong [Str "line1"]]]`
>
> So far so good. Now, with the following lua filter stored in a file called
> `lua1.lua`
>
> ```lua
> function Strong(elem)
>   return pandoc.Str(elem.content)
> end
> ```
>
> Executing `echo "**line1**" | pandoc -t native --lua-filter lua1.lua`
>
> Outputs: `[Para []]`
>
> Which means a blank document, because the content "line1" was deleted, it
> doesn't exist anymore, is this a bug?
>

No. The Str constructor expects a string as argument/content but the
content of a Strong element is a list of objects. If there is a bug it is
that you don't get an error/warning message about illegal content for a Str
element.

If the purpose of the filter is to remove the emphasis you should just
return the content (a list of inline elements) of the Strong element.


Compare with the following.
>
> Modifying the file `lua1.lua` as,
>
> ```lua
> function Strong(elem)
>   return pandoc.Str("hello1")
> end
> ```
>
> Executing again `echo "**line1**" | pandoc -t native --lua-filter lua1.lua`
>
> Outputs: `[Para [Str "hello1"]]`
>
> This one works, why does the first one not work? Shouldn't both work?
> Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/e2b36f5d-7c73-4160-a2d1-20b57c49f129n%40googlegroups.com
> <https://groups.google.com/d/msgid/pandoc-discuss/e2b36f5d-7c73-4160-a2d1-20b57c49f129n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhC0cK8iWsrohbSFaOe5POWU1QqSwpGCTpm3VNjEpgaZ8w%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 3922 bytes --]

Re: Is this a bug?

[Albert Krewinkel]

BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> Den sön 28 mars 2021 10:01Bücherregal Domi <gamezaerza-Re5JQEeQqe8@public.gmane.orgm> skrev:
>
>> Which means a blank document, because the content "line1" was deleted, it
>> doesn't exist anymore, is this a bug?
>
> No. The Str constructor expects a string as argument/content but the
> content of a Strong element is a list of objects. If there is a bug it is
> that you don't get an error/warning message about illegal content for a Str
> element.

I agree, the lack of a warning can be considered a bug. FWIW, I'm
refactoring the Lua subsystem and rewriting the HsLua library on which
it depends. As is tradition with software projects, the whole process
takes a really long time. If someone is interested in its progress, this
is the branch that I'm working on:
<https://github.com/tarleb/pandoc/tree/hslua-2.0>
And here is the HsLua repo: <https://github.com/hslua/hslua>

Current improvements, not yet complete:

 - better error messages while retrieving values;
 - better error reporting when calling functions;
 - clean method to define functions and types in Lua;
 - linking of functions and their documentation.

Type warnings, as described in issue #4158, are part of my personal todo
list. I'm still experimenting with alternative element marshalling
methods, which I hope will make this easier.

Other issues that have been mentioned in the past and that could be
improved:

 - Make read-only objects throw errors when a filter tries to alter
   attributes.
 - Separation of Haskell and Lua code can be confusing and should be
   reduced.

Let me know about other shortcomings that you'd like to see tackled in
this refactoring.

Cheers,
Albert

--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/875z1bfv12.fsf%40zeitkraut.de.

Re: Is this a bug?

[Bücherregal Domi]

[-- Attachment #1.1: Type: text/plain, Size: 2986 bytes --]

Thank you guys for your quick answer.

@Albert Krewinkel, I can't quote you directly, but I wanted to ask you, you 
said

> Let me know about other shortcomings that you'd like to see tackled in 
this refactoring. 

How should we? I saw your hslua repo, is ok to open issues there directly? 
Or do you have a mailing list or a google group like this one? (It doesn't 
say it in your repo, as far as I saw). Also what you said about element 
marshalling methods, do you mean the conversion of Lua data into Haskell 
data in vice versa? Thank you.

Albert Krewinkel schrieb am Sonntag, 28. März 2021 um 05:10:58 UTC-5:

>
> BPJ <mel...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
> > Den sön 28 mars 2021 10:01Bücherregal Domi <gamez...-Re5JQEeQqe8@public.gmane.orgm> skrev:
> >
> >> Which means a blank document, because the content "line1" was deleted, 
> it
> >> doesn't exist anymore, is this a bug?
> >
> > No. The Str constructor expects a string as argument/content but the
> > content of a Strong element is a list of objects. If there is a bug it is
> > that you don't get an error/warning message about illegal content for a 
> Str
> > element.
>
> I agree, the lack of a warning can be considered a bug. FWIW, I'm
> refactoring the Lua subsystem and rewriting the HsLua library on which
> it depends. As is tradition with software projects, the whole process
> takes a really long time. If someone is interested in its progress, this
> is the branch that I'm working on:
> <https://github.com/tarleb/pandoc/tree/hslua-2.0>
> And here is the HsLua repo: <https://github.com/hslua/hslua>
>
> Current improvements, not yet complete:
>
> - better error messages while retrieving values;
> - better error reporting when calling functions;
> - clean method to define functions and types in Lua;
> - linking of functions and their documentation.
>
> Type warnings, as described in issue #4158, are part of my personal todo
> list. I'm still experimenting with alternative element marshalling
> methods, which I hope will make this easier.
>
> Other issues that have been mentioned in the past and that could be
> improved:
>
> - Make read-only objects throw errors when a filter tries to alter
> attributes.
> - Separation of Haskell and Lua code can be confusing and should be
> reduced.
>
> Let me know about other shortcomings that you'd like to see tackled in
> this refactoring.
>
> Cheers,
> Albert
>
> --
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe e836 388d c0b2 1f63 1124
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/33f13d23-8708-439a-a322-245821a5e900n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 4405 bytes --]

Re: Is this a bug?

[Albert Krewinkel]

Bücherregal Domi <gamezaerza-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> @Albert Krewinkel, I can't quote you directly, but I wanted to ask you, you
> said
>
>> Let me know about other shortcomings that you'd like to see tackled in
> this refactoring.
>
> How should we? I saw your hslua repo, is ok to open issues there directly?
> Or do you have a mailing list or a google group like this one? (It doesn't
> say it in your repo, as far as I saw).

The issue tracker is a good place. Or, if you are not sure if it's an
HsLua or a pandoc issue, just raise it here on this list.

> Also what you said about element
> marshalling methods, do you mean the conversion of Lua data into Haskell
> data in vice versa? Thank you.

Yes, that's what I refer to as "unmarshalling" and "marshalling",
respectively. We currently use a simple but effective method of creating
tables to hold the element data, and metatables to provide additional
behavior. This gives simple Lua objects and has decent performance, but
is somewhat convoluted in its implementation and requires additional Lua
code to work. What I'm experimenting with is to make use of Lua's
"userdata" objects, which would keep the Haskell value attached to the
Lua object. It would allow lazy loading and should speedup manipulation
of larger elements, like "Div", but slow down handling of small objects,
like "Str". Mixing two garbage collected languages, one with mutable and
the other with immutable semantics, has its challenges, and the whole
thing is tricky to get right.


> Albert Krewinkel schrieb am Sonntag, 28. März 2021 um 05:10:58 UTC-5:
>
>>
>> BPJ <mel...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>
>> > Den sön 28 mars 2021 10:01Bücherregal Domi <gamez...@gmail.com> skrev:
>> >
>> >> Which means a blank document, because the content "line1" was deleted,
>> it
>> >> doesn't exist anymore, is this a bug?
>> >
>> > No. The Str constructor expects a string as argument/content but the
>> > content of a Strong element is a list of objects. If there is a bug it is
>> > that you don't get an error/warning message about illegal content for a
>> Str
>> > element.
>>
>> I agree, the lack of a warning can be considered a bug. FWIW, I'm
>> refactoring the Lua subsystem and rewriting the HsLua library on which
>> it depends. As is tradition with software projects, the whole process
>> takes a really long time. If someone is interested in its progress, this
>> is the branch that I'm working on:
>> <https://github.com/tarleb/pandoc/tree/hslua-2.0>
>> And here is the HsLua repo: <https://github.com/hslua/hslua>
>>
>> Current improvements, not yet complete:
>>
>> - better error messages while retrieving values;
>> - better error reporting when calling functions;
>> - clean method to define functions and types in Lua;
>> - linking of functions and their documentation.
>>
>> Type warnings, as described in issue #4158, are part of my personal todo
>> list. I'm still experimenting with alternative element marshalling
>> methods, which I hope will make this easier.
>>
>> Other issues that have been mentioned in the past and that could be
>> improved:
>>
>> - Make read-only objects throw errors when a filter tries to alter
>> attributes.
>> - Separation of Haskell and Lua code can be confusing and should be
>> reduced.
>>
>> Let me know about other shortcomings that you'd like to see tackled in
>> this refactoring.
>>
>> Cheers,
>> Albert
>>
>> --
>> Albert Krewinkel
>> GPG: 8eed e3e2 e8c5 6f18 81fe e836 388d c0b2 1f63 1124
>>


--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/87zgyndtmm.fsf%40zeitkraut.de.

Re: Is this a bug?

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 5929 bytes --]

TL;DR it would be good if userdata objects know how to serialize themselves
for debugging purposes.

One downside of using userdata rather than tables is that it makes it
harder to debug data structures. Currently I can get a reasonable idea of
what the internals of an object looks like by running it through a function
similar to the one below and passing the result to Penlight's or
MoonScript's pretty-printer.

``````lua
local show_dump
show_dump = function (obj)
  if 'table' == type(obj) then
    data = {}
    for k,v in pairs(obj) do
      data[k] = show_dump(v)
    end
    return {
      tag  = obj.tag,
      data = data
    }
  else
    return obj
  end
end
``````

if the object is a userdata I will have to know which fields to copy for
each kind of object, which is non-trivial. Perhaps they could have a `json`
or `serialize` method which internally uses the same code as is used to
produce JSON or native output? (I'm guessing that native output is produced
using some function which Haskell provides out of the box?)


Den sön 28 mars 2021 20:24Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org>
skrev:

>
> Bücherregal Domi <gamezaerza-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
> > @Albert Krewinkel, I can't quote you directly, but I wanted to ask you,
> you
> > said
> >
> >> Let me know about other shortcomings that you'd like to see tackled in
> > this refactoring.
> >
> > How should we? I saw your hslua repo, is ok to open issues there
> directly?
> > Or do you have a mailing list or a google group like this one? (It
> doesn't
> > say it in your repo, as far as I saw).
>
> The issue tracker is a good place. Or, if you are not sure if it's an
> HsLua or a pandoc issue, just raise it here on this list.
>
> > Also what you said about element
> > marshalling methods, do you mean the conversion of Lua data into Haskell
> > data in vice versa? Thank you.
>
> Yes, that's what I refer to as "unmarshalling" and "marshalling",
> respectively. We currently use a simple but effective method of creating
> tables to hold the element data, and metatables to provide additional
> behavior. This gives simple Lua objects and has decent performance, but
> is somewhat convoluted in its implementation and requires additional Lua
> code to work. What I'm experimenting with is to make use of Lua's
> "userdata" objects, which would keep the Haskell value attached to the
> Lua object. It would allow lazy loading and should speedup manipulation
> of larger elements, like "Div", but slow down handling of small objects,
> like "Str". Mixing two garbage collected languages, one with mutable and
> the other with immutable semantics, has its challenges, and the whole
> thing is tricky to get right.
>
>
> > Albert Krewinkel schrieb am Sonntag, 28. März 2021 um 05:10:58 UTC-5:
> >
> >>
> >> BPJ <mel...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
> >>
> >> > Den sön 28 mars 2021 10:01Bücherregal Domi <gamez...@gmail.com>
> skrev:
> >> >
> >> >> Which means a blank document, because the content "line1" was
> deleted,
> >> it
> >> >> doesn't exist anymore, is this a bug?
> >> >
> >> > No. The Str constructor expects a string as argument/content but the
> >> > content of a Strong element is a list of objects. If there is a bug
> it is
> >> > that you don't get an error/warning message about illegal content for
> a
> >> Str
> >> > element.
> >>
> >> I agree, the lack of a warning can be considered a bug. FWIW, I'm
> >> refactoring the Lua subsystem and rewriting the HsLua library on which
> >> it depends. As is tradition with software projects, the whole process
> >> takes a really long time. If someone is interested in its progress, this
> >> is the branch that I'm working on:
> >> <https://github.com/tarleb/pandoc/tree/hslua-2.0>
> >> And here is the HsLua repo: <https://github.com/hslua/hslua>
> >>
> >> Current improvements, not yet complete:
> >>
> >> - better error messages while retrieving values;
> >> - better error reporting when calling functions;
> >> - clean method to define functions and types in Lua;
> >> - linking of functions and their documentation.
> >>
> >> Type warnings, as described in issue #4158, are part of my personal todo
> >> list. I'm still experimenting with alternative element marshalling
> >> methods, which I hope will make this easier.
> >>
> >> Other issues that have been mentioned in the past and that could be
> >> improved:
> >>
> >> - Make read-only objects throw errors when a filter tries to alter
> >> attributes.
> >> - Separation of Haskell and Lua code can be confusing and should be
> >> reduced.
> >>
> >> Let me know about other shortcomings that you'd like to see tackled in
> >> this refactoring.
> >>
> >> Cheers,
> >> Albert
> >>
> >> --
> >> Albert Krewinkel
> >> GPG: 8eed e3e2 e8c5 6f18 81fe e836 388d c0b2 1f63 1124
> >>
>
>
> --
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/87zgyndtmm.fsf%40zeitkraut.de
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhBVVX_KfL0DyuPq7%2BYUQRR1SK7MuOqi%2Bgiqp4ANbGLOkw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 8498 bytes --]

Re: Is this a bug?

[Albert Krewinkel]

BPJ <bpj-J3H7GcXPSITLoDKTGw+V6w@public.gmane.org> writes:

> TL;DR it would be good if userdata objects know how to serialize
> themselves for debugging purposes.

Thanks for raising this! I'm refactoring with discoverability in mind,
and I appreciate feedback on the new design.

Using the PANDOC_READER_OPTIONS as an example: below is the new type
definition:

```haskell
typeReaderOptions = deftype "pandoc ReaderOptions"
  [ operation Tostring $ luaShow (peekUD typeReaderOptions)
  ]
  [ readonly "extensions"     "" (pushString . show, readerExtensions)
  , readonly "standalone"     "" (pushBool,          readerStandalone)
  , readonly "columns"        "" (pushIntegral,      readerColumns)
  , readonly "tab_stop"       "" (pushIntegral,      readerTabStop)
  , readonly "abbreviations"  "" (pushSet pushText,  readerAbbreviations)
  , readonly "track_changes"  "" (pushString . show, readerTrackChanges)
  , readonly "strip_comments" "" (pushBool,          readerStripComments)
  , readonly "indented_code_classes" ""
      (push, readerIndentedCodeClasses)
  , readonly "default_image_extension" ""
      (pushText, readerDefaultImageExtension)
  ]
```

All member properties in the second list are included when calling
`pairs` on an object of that type; calling
`tostring(PANDOC_READER_OPTIONS)` returns Haskell's string
representation of that value.

Compared to the [current] version, this should also be much easier to
maintain and extend. E.g., adding JSON support could be as simple as
adding a json operation like so:

    operation (CustomOperation "__tojson") $ lambda (liftPure encode)
      <#> udparam typeReaderOptions "obj" "object to encode as JSON"
      =#> functionResult pushByteStringLazy "string" "JSON representation"

and defining `tojson` akin to

    function tojson (obj)
      return debug.getmetatable(obj).__json(obj)
    end

Making properties read and writable is as simple as changing `readonly`
to `property`; however, the resulting objects have weird semantics and
the feature requires more work.

----

I'm also toying with a new function `describe`, which will return
documentation for any object. This allows to get info by running

    print(describe(pandoc.utils.stringify))

and to automatically create module documentation with

    function Pandoc(_)
      return pandoc.read(describe(pandoc.system))
    end

On the development side, this has the advantage that documentation and
code become far less likely to drift apart.

[current]: https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Lua/Marshaling/ReaderOptions.hs


--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Is this a bug?

[Albert Krewinkel]

Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> writes:

> Using the PANDOC_READER_OPTIONS as an example

Actually, the CommonState (as in `PANDOC_STATE`) is a better example:

Current mess:
https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Lua/Marshaling/CommonState.hs

New:
https://github.com/tarleb/pandoc/blob/hslua-2.0/src/Text/Pandoc/Lua/Marshaling/CommonState.hs



--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124