Custom readers and writer paths

Custom readers and writer paths

[John MacFarlane]

I believe that currently one must specify the path to a custom
Lua reader or writer: you can't put it in your data directory,
as you can with a Lua filter, and it won't be found in your
executable PATH.

(Is that correct, Albert?)

My question is whether we should change that.  I think it could
be quite convenient to allow custom readers and writers to be
distributed as Lua rocks.  You could then do `luarocks install
cool-pandoc-writer` and it would put `cool-writer.lua` in the
`bin` directory of your luarocks installation. Pandoc could then
be trained to look in the executable path for a custom writer
if it is not found locally.

Alternatively, we could search a custom-writers and custom-readers
subdirectory of the user data directory (as we do with filters).

It is a little cumbersome to have to copy the default
writer/reader somewhere and pass its path to pandoc.  This may
be good from a security point of view, however, as it makes it
less likely that people will blindly use custom readers/writers
from third parties, not realizing that a custom reader/writer
could in principle do just about anything on your file system.

A related question: would it be possible for pandoc to set up
the Lua environment in which custom readers/writers are run so
that it is "sandboxed," limiting I/O operations to (e.g.) a
local directory, or perhaps logging I/O operations as pandoc
warnings?

Thoughts welcome.

John

Re: Custom readers and writer paths

[Albert Krewinkel]

John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:

> I believe that currently one must specify the path to a custom
> Lua reader or writer: you can't put it in your data directory,
> as you can with a Lua filter, and it won't be found in your
> executable PATH.
>
> (Is that correct, Albert?)

I'm afraid it is.

> My question is whether we should change that.  I think it could
> be quite convenient to allow custom readers and writers to be
> distributed as Lua rocks.  You could then do `luarocks install
> cool-pandoc-writer` and it would put `cool-writer.lua` in the
> `bin` directory of your luarocks installation. Pandoc could then
> be trained to look in the executable path for a custom writer
> if it is not found locally.
>
> Alternatively, we could search a custom-writers and custom-readers
> subdirectory of the user data directory (as we do with filters).

I like both options, with a slight preference for the latter.

> It is a little cumbersome to have to copy the default
> writer/reader somewhere and pass its path to pandoc.  This may
> be good from a security point of view, however, as it makes it
> less likely that people will blindly use custom readers/writers
> from third parties, not realizing that a custom reader/writer
> could in principle do just about anything on your file system.

AFAIK, luarocks is rather relaxed when it comes to security. Just
installing a rock can lead to the execution of arbitrary code. I don't
feel like we should be too worried about the security implications of
running an already installed Lua library.

I'm not sure if it's better or worse, but we could also look for the
reader in LUA_PATH (via `require`). Using a bit of pseudo-code:

    if file_exists(reader_path) then
      dofile(reader_path)
      reader = Reader
    else
      reader = require (reader_path:gsub('.lua$', ''))
    end

The reader library would have to `return` the Reader function instead of
just defining it as a global.

> A related question: would it be possible for pandoc to set up
> the Lua environment in which custom readers/writers are run so
> that it is "sandboxed," limiting I/O operations to (e.g.) a
> local directory, or perhaps logging I/O operations as pandoc
> warnings?

It would be a lot of work. Julien Dutant asked a similar question
recently, and I listed my concerns there:
https://groups.google.com/g/pandoc-discuss/c/2dMimbemNpM/m/pLXPk4p0AQAJ


-- 
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Custom readers and writer paths

[John MacFarlane]

Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> writes:

> I'm not sure if it's better or worse, but we could also look for the
> reader in LUA_PATH (via `require`). Using a bit of pseudo-code:
>
>     if file_exists(reader_path) then
>       dofile(reader_path)
>       reader = Reader
>     else
>       reader = require (reader_path:gsub('.lua$', ''))
>     end
>
> The reader library would have to `return` the Reader function instead of
> just defining it as a global.

Probably better not to change the interface in this way.

Maybe treating these like filters is the most sensible/least
surprising thing to do.  Should we have readers/ and writers/
subdirectories of user data?  Or custom-readers/ custom-writers/ ?

Re: Custom readers and writer paths

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 2447 bytes --]

Den mån 6 juni 2022 20:39John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> skrev:

> Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> writes:
>
> > I'm not sure if it's better or worse, but we could also look for the
> > reader in LUA_PATH (via `require`). Using a bit of pseudo-code:
> >
> >     if file_exists(reader_path) then
> >       dofile(reader_path)
> >       reader = Reader
> >     else
> >       reader = require (reader_path:gsub('.lua$', ''))
> >     end
> >
> > The reader library would have to `return` the Reader function instead of
> > just defining it as a global.
>
> Probably better not to change the interface in this way.
>
> Maybe treating these like filters is the most sensible/least
> surprising thing to do.


IMO the most sensible. That way you can just clone readers/writers off
GitHub without authors needing to go through the hassle of setting up a Lua
rock. Having to do that will likely discourage people from publishing their
code at all. Also of course templates, filters, defaults all work that way
already. I have my data dir in my dropbox directory and symlink it in
~/.local/share, or rclone it on my Android devices (via Termux) where
dropbox isn't a regular storage directory.


  Should we have readers/ and writers/
> subdirectories of user data?  Or custom-readers/ custom-writers/ ?
>

Since they are already subdirectories of the dedicated pandoc data
directory `readers/` and `writers/` should suffice. Seven characters less
to type for each compared to `custom-*`.



> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/yh480kv8tdd4de.fsf%40johnmacfarlane.net
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhDdwHuvBjTCAAwrFY6RcaZ%2Bd_FRMnkVp3MMUtntrsBdYg%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 4009 bytes --]

AW: Custom readers and writer paths

[denis.maier-NSENcxR/0n0]

Yes please. Not Luarocks, at least unless someone can show an easy way to install it on Windows?

Denis
________________________________________
Von: pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org <pandoc-discuss-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> im Auftrag von BPJ <bpj-J3H7GcXPSITLoDKTGw+V6w@public.gmane.org>
Gesendet: Montag, 6. Juni 2022 21:53:16
An: pandoc-discuss
Cc: Albert Krewinkel
Betreff: Re: Custom readers and writer paths

Den mån 6 juni 2022 20:39John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org<mailto:jgm@berkeley.edu>> skrev:
Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org<mailto:albert%2Bpandoc@zeitkraut.de>> writes:

> I'm not sure if it's better or worse, but we could also look for the
> reader in LUA_PATH (via `require`). Using a bit of pseudo-code:
>
>     if file_exists(reader_path) then
>       dofile(reader_path)
>       reader = Reader
>     else
>       reader = require (reader_path:gsub('.lua$', ''))
>     end
>
> The reader library would have to `return` the Reader function instead of
> just defining it as a global.

Probably better not to change the interface in this way.

Maybe treating these like filters is the most sensible/least
surprising thing to do.

IMO the most sensible. That way you can just clone readers/writers off GitHub without authors needing to go through the hassle of setting up a Lua rock. Having to do that will likely discourage people from publishing their code at all. Also of course templates, filters, defaults all work that way already. I have my data dir in my dropbox directory and symlink it in ~/.local/share, or rclone it on my Android devices (via Termux) where dropbox isn't a regular storage directory.


  Should we have readers/ and writers/
subdirectories of user data?  Or custom-readers/ custom-writers/ ?

Since they are already subdirectories of the dedicated pandoc data directory `readers/` and `writers/` should suffice. Seven characters less to type for each compared to `custom-*`.



--
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org<mailto:pandoc-discuss%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/yh480kv8tdd4de.fsf%40johnmacfarlane.net.

--
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org<mailto:pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org>.
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhDdwHuvBjTCAAwrFY6RcaZ%2Bd_FRMnkVp3MMUtntrsBdYg%40mail.gmail.com<https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhDdwHuvBjTCAAwrFY6RcaZ%2Bd_FRMnkVp3MMUtntrsBdYg%40mail.gmail.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/f27c064156a4413ba048992ce0338c68%40unibe.ch.

Re: Custom readers and writer paths

[John MacFarlane]

BPJ <bpj-J3H7GcXPSITLoDKTGw+V6w@public.gmane.org> writes:

> IMO the most sensible. That way you can just clone readers/writers off
> GitHub without authors needing to go through the hassle of setting up a Lua
> rock. Having to do that will likely discourage people from publishing their
> code at all. Also of course templates, filters, defaults all work that way
> already. I have my data dir in my dropbox directory and symlink it in
> ~/.local/share, or rclone it on my Android devices (via Termux) where
> dropbox isn't a regular storage directory.

The one advantage of rocks is that your filter can specify
external Lua dependencies.

Re: Custom readers and writer paths

[Albert Krewinkel]

BPJ <bpj-J3H7GcXPSITLoDKTGw+V6w@public.gmane.org> writes:

> > Should we have readers/ and writers/
> > subdirectories of user data?  Or custom-readers/ custom-writers/ ?
>
> Since they are already subdirectories of the dedicated pandoc data
> directory `readers/` and `writers/` should suffice. Seven characters
> less to type for each compared to `custom-*`.

I took the freedom to open a quick draft PR for this:
https://github.com/jgm/pandoc/pull/8112


-- 
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124