Self-documenting Lua modules

Self-documenting Lua modules

[Albert Krewinkel]

Dear *,

I'm working on some code to improve Lua bindings and integration
and would appreciate feedback on the approach.

In a PR discussion on GitHub, jgm describes the problem which I'm
now trying to solve:

> I'm getting a bit worried about the ongoing maintenance cost of
> shadowing large parts of the pandoc (and related) API with Lua.
> In the present case: if I make further changes to doclayout,
> then this custom Lua code needs to be updated, and so does the
> documentation. However, I'm not sure anything would
> automatically alert us to the need to do these things, which
> raises a danger of breakage as things get out of sync
> (especially if you ever stop being such an active contributor to
> the project).

Currently, pandoc's Lua documentation and Lua bindings are
entirely separate. The missing documentation for the new table
format recently is an excellent example of the problems we are
facing and highlights the shortcomings of our current method.

The idea is thus to co-locate documentation and code. Here is a
first prototype, using a potential `paths` module for filepath
handling as an example:
<https://github.com/hslua/hslua-module-paths>
Explicit typing of function parameters should give better
compile-time guarantees, while keeping documentation close to the
code should reduce the risk of an outdated manual. The
documentation for this module can be generated by running `cabal
run tools/print-docs.hs`.

If anyone has experience with this kind of approach, can comment
on the code, or wants to share other ideas that could help us make
pandoc's Lua code more robust, then I'd love to hear about it.

Cheers,
Albert

--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Self-documenting Lua modules

[Gabriel Nützi]

[-- Attachment #1: Type: text/plain, Size: 3832 bytes --]

Gabriel Nützi to Albert Krewinkel:

This actually looks pretty fancy. I can’t contribute relevant input, but kindly have some questions:

The code 
-- | See @System.FilePath.takeExtensions@
take_extensions :: HaskellFunction
take_extensions = toHsFnPrecursor Path.takeExtensions
  <#> filepathParam
  =#> filepathResult "String of all extensions."
  #? "Get all extensions."
How can a novice (like me ;-)) understand this <#>, =#> #?... Is this take_extensions Function some kind of a special “thing” which can be called, but enables also to collect the doc by doing something
else with it? Is this going the way like D. Knuth  with its TeX Language? Self-documenting code :)
Is there no tooling for haskell which extracts the code documentation `—| …` etc…? Just curious?

Thanks for putting this path stuff in a separate library, I thought of it might be at the wrong place in th PR :-)… Cant wait till we have this in the filter, such that I can adapt the include-files filter to accept  relative includes…

BR Gabriel


> On 11 Aug 2020, at 11:19, Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> wrote:
> 
> Dear *,
> 
> I'm working on some code to improve Lua bindings and integration
> and would appreciate feedback on the approach.
> 
> In a PR discussion on GitHub, jgm describes the problem which I'm
> now trying to solve:
> 
>> I'm getting a bit worried about the ongoing maintenance cost of
>> shadowing large parts of the pandoc (and related) API with Lua.
>> In the present case: if I make further changes to doclayout,
>> then this custom Lua code needs to be updated, and so does the
>> documentation. However, I'm not sure anything would
>> automatically alert us to the need to do these things, which
>> raises a danger of breakage as things get out of sync
>> (especially if you ever stop being such an active contributor to
>> the project).
> 
> Currently, pandoc's Lua documentation and Lua bindings are
> entirely separate. The missing documentation for the new table
> format recently is an excellent example of the problems we are
> facing and highlights the shortcomings of our current method.
> 
> The idea is thus to co-locate documentation and code. Here is a
> first prototype, using a potential `paths` module for filepath
> handling as an example:
> <https://github.com/hslua/hslua-module-paths>
> Explicit typing of function parameters should give better
> compile-time guarantees, while keeping documentation close to the
> code should reduce the risk of an outdated manual. The
> documentation for this module can be generated by running `cabal
> run tools/print-docs.hs`.
> 
> If anyone has experience with this kind of approach, can comment
> on the code, or wants to share other ideas that could help us make
> pandoc's Lua code more robust, then I'd love to hear about it.
> 
> Cheers,
> Albert
> 
> --
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
> 
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/87h7t9ilp2.fsf%40zeitkraut.de.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/22D9BE00-5CC2-4AD6-B146-7F5CFA014E19%40gmail.com.

[-- Attachment #2: Type: text/html, Size: 6565 bytes --]

Re: Self-documenting Lua modules

[Albert Krewinkel]

Gabriel Nützi writes:

> Gabriel Nützi to Albert Krewinkel:
>
> This actually looks pretty fancy. I can’t contribute relevant
> input, but kindly have some questions:

Even better :D

> The code
> -- | See @System.FilePath.takeExtensions@
> take_extensions :: HaskellFunction
> take_extensions = toHsFnPrecursor Path.takeExtensions
>   <#> filepathParam
>   =#> filepathResult "String of all extensions."
>   #? "Get all extensions."
> How can a novice (like me ;-)) understand this <#>, =#> #?

The idea is that `<#>` is similar to the default `<*>` Haskell
operator in that it "applies" an argument. Let's say we want to
expose Haskell's `gcd :: Integral a => a -> a -> a` function. We'd
start building the function with `toHsFnPrecursor`[^1], then
describe the parameters individually

    lua_gcd = toHsFnPrecursor gcd
      <#> integerParam "x"
      <#> integerParam "y"

This defines how we get the arguments from Lua, and what we do
with them. The `=#>` operator then adds information of how to send
the result back to Lua, and `#?` adds a docstring.

      =#> integerResult "Greatest common divisor of x and y"
      #? ("gcd x y is the non-negative factor of both x and y of " <>
          "which every common factor of x and y is also a factor"

The previous method of exposing functions worked similarly, but
used a lot of magic. The proposed method requires more explicit
typing. Also, the additional information about parameter names
allows for more detailed and descriptive error messages.

[^1]: naming suggestions are welcome!

> Is this take_extensions Function some kind of a special “thing”
> which can be called, but enables also to collect the doc by
> doing something else with it?

Yes, we can "push" it to Lua and it like a normal Lua function.
The documentation is currently only on the Haskell side, but I'd
like to add something like Common Lisp's `DESCRIBE` or Python's
`help` function:
  - <http://clhs.lisp.se/Body/f_descri.htm>
  - <https://docs.python.org/3.8/library/functions.html#help>.

> Is this going the way like D. Knuth with its TeX Language?
> Self-documenting code :)

Something like WEB would be nice :) But no, I'm not writing a full
document system, and rather aim to keep this as a Haskell EDSL.

> Is there no tooling for haskell which
> extracts the code documentation `—| …` etc…? Just curious?

There is [Haddock], which is only for Haskell functions, and there
is [ldoc] for Lua functions. There is nothing for Lua functions
build with Haskell.

[Haddock]: https://www.haskell.org/haddock/
[ldoc]: https://stevedonovan.github.io/ldoc/

> Thanks for putting this path stuff in a separate library, I
> thought of it might be at the wrong place in th PR :-)… Cant
> wait till we have this in the filter, such that I can adapt the
> include-files filter to accept relative includes…

Thanks again for helping out!

Cheers

--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/87bljghqw8.fsf%40zeitkraut.de.

Re: Self-documenting Lua modules

[John MacFarlane]

I like this!