Lua filter can not find lua package in data-dir

Lua filter can not find lua package in data-dir

[jiewuza]

[-- Attachment #1: Type: text/plain, Size: 281 bytes --]

I download a lua package, say json.lua, and put it in data-dir (~/.local/share).And then I require it in my lua filter. But it reports error that the pacakge is not found indata-dir. Did I make any mistake?
-- 




----Android NewsGroup Reader----
http://www.piaohong.tk/newsgroup

Re: Lua filter can not find lua package in data-dir

[John MacFarlane]

jiewuza <jiewuza-9Onoh4P/yGk@public.gmane.org> writes:

> I download a lua package, say json.lua, and put it in data-dir (~/.local/share).And then I require it in my lua filter. But it reports error that the pacakge is not found indata-dir. Did I make any mistake?

You need to install it in the path lua will search.
You can change this path by setting the LUA_PATH environment
variable.

Or, at the beginning of your lua filter (before require)
you could have something like

package.path = "/full/path/to/the/module/?.lua"

Albert:  I wonder if we should automatically set package.path
in the lua environment to include the directory containing the filter?

Re: Lua filter can not find lua package in data-dir

[Albert Krewinkel]

John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:

> Or, at the beginning of your lua filter (before require)
> you could have something like
>
> package.path = "/full/path/to/the/module/?.lua"
>
> Albert:  I wonder if we should automatically set package.path
> in the lua environment to include the directory containing the filter?

The thought of changing the path has me slightly uncomfortable, but
I also cannot think of any serious downsides. It would likely be a
usability improving change and allow groups of filters to be used
easily and without forcing users to combine them into a single file
first.

For now, the following Lua command can be used at the top of the Lua
file to the same means.

    package.path = PANDOC_SCRIPT_FILE:gsub('[\\/]+%.lua$', '?.lua;')
        .. package.path


--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Lua filter can not find lua package in data-dir

[John MacFarlane]

Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> writes:

> John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:
>
>> Or, at the beginning of your lua filter (before require)
>> you could have something like
>>
>> package.path = "/full/path/to/the/module/?.lua"
>>
>> Albert:  I wonder if we should automatically set package.path
>> in the lua environment to include the directory containing the filter?
>
> The thought of changing the path has me slightly uncomfortable, but
> I also cannot think of any serious downsides. It would likely be a

It makes me a bit uncomfortable too, and since there's an easy
enough workaround, maybe we should table the idea for a while.

Re: Lua filter can not find lua package in data-dir

[jiewuza]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

Nice.
But the error reported mentioned data-dir (and other paths). It was quite misleading. Would you change the error message a bit?


-- 


John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> 写入消息新闻:
> jiewuza <jiewuza-9Onoh4P/yGk@public.gmane.org> writes:
> 
>> I download a lua package, say json.lua, and put it in data-dir (~/.local/share).And then I require it in my lua filter. But it reports error that the pacakge is not found indata-dir. Did I make any mistake?

----Android NewsGroup Reader----
http://www.piaohong.tk/newsgroup

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ruibe9%24nf0%241%40ciao.gmane.io.

Re: Lua filter can not find lua package in data-dir

[John MacFarlane]

I'm not sure what the error was precisely.  Can you quote it
exactly, with more context about what you're doing?

jiewuza <jiewuza-9Onoh4P/yGk@public.gmane.org> writes:

> Nice.
> But the error reported mentioned data-dir (and other paths). It was quite misleading. Would you change the error message a bit?
>
>
> -- 
>
>
> John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> 写入消息新闻:
>> jiewuza <jiewuza-9Onoh4P/yGk@public.gmane.org> writes:
>> 
>>> I download a lua package, say json.lua, and put it in data-dir (~/.local/share).And then I require it in my lua filter. But it reports error that the pacakge is not found indata-dir. Did I make any mistake?
>
> ----Android NewsGroup Reader----
> http://www.piaohong.tk/newsgroup
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ruibe9%24nf0%241%40ciao.gmane.io.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m2im7ni4tq.fsf%40MacBook-Pro.hsd1.ca.comcast.net.

Re: Lua filter can not find lua package in data-dir

[jiewuza]

[-- Attachment #1: Type: text/plain, Size: 1240 bytes --]

This is the error message:

Error running filter ../../../script/my.lua:
../../../script/my.lua:135: module 'json' not found:
	no file 'json.lua' in pandoc's datadir
	no field package.preload['json']
	no file '/usr/local/share/lua/5.3/json.lua'
	no file '/usr/local/share/lua/5.3/json/init.lua'
	no file '/usr/local/lib/lua/5.3/json.lua'
	no file '/usr/local/lib/lua/5.3/json/init.lua'
	no file './json.lua'
	no file './json/init.lua'
	no file '/usr/local/lib/lua/5.3/json.so'
	no file '/usr/local/lib/lua/5.3/loadall.so'
	no file './json.so'


-- 


John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> 写入消息新闻:
> 
> I'm not sure what the error was precisely.  Can you quote it
> exactly, with more context about what you're doing?

----Android NewsGroup Reader----
http://www.piaohong.tk/newsgroup

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ruik5n%24144s%241%40ciao.gmane.io.

Re: Lua filter can not find lua package in data-dir

[John MacFarlane]

Oh, interesting. I looked at the code, and sure enough,
it already DOES try to load modules from datadir.

I tried it, and it works.

Note: I had to put the module at the top level in the user
data directory, not in the filters/ subdirectory.

I'm not sure why it didn't work for you; are you sure
you're using the right directory?  (For me,
~/.local/share/pandoc/, but it could be ~/.pandoc if you
don't have a ~/.local/share/pandoc directory.)

Albert, do we want it to work like this?

jiewuza <jiewuza-9Onoh4P/yGk@public.gmane.org> writes:

> This is the error message:
>
> Error running filter ../../../script/my.lua:
> ../../../script/my.lua:135: module 'json' not found:
> 	no file 'json.lua' in pandoc's datadir
> 	no field package.preload['json']
> 	no file '/usr/local/share/lua/5.3/json.lua'
> 	no file '/usr/local/share/lua/5.3/json/init.lua'
> 	no file '/usr/local/lib/lua/5.3/json.lua'
> 	no file '/usr/local/lib/lua/5.3/json/init.lua'
> 	no file './json.lua'
> 	no file './json/init.lua'
> 	no file '/usr/local/lib/lua/5.3/json.so'
> 	no file '/usr/local/lib/lua/5.3/loadall.so'
> 	no file './json.so'
>
>
> -- 
>
>
> John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> 写入消息新闻:
>> 
>> I'm not sure what the error was precisely.  Can you quote it
>> exactly, with more context about what you're doing?
>
> ----Android NewsGroup Reader----
> http://www.piaohong.tk/newsgroup
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/ruik5n%24144s%241%40ciao.gmane.io.

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/m2v9bmhmzj.fsf%40MacBook-Pro.hsd1.ca.comcast.net.

Re: Lua filter can not find lua package in data-dir

[Albert Krewinkel]

John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:

> Oh, interesting. I looked at the code, and sure enough,
> it already DOES try to load modules from datadir.
>
> I tried it, and it works.
>
> Note: I had to put the module at the top level in the user
> data directory, not in the filters/ subdirectory.
>
> I'm not sure why it didn't work for you; are you sure
> you're using the right directory?  (For me,
> ~/.local/share/pandoc/, but it could be ~/.pandoc if you
> don't have a ~/.local/share/pandoc directory.)
>
> Albert, do we want it to work like this?

To be honest, I had forgotten this functionality existed. We use it to
load `pandoc.List.lua`, with the idea that it can be swapped out for a
custom implementation if necessary.

We don't modify the load path, but instead provide a custom package
loader function in `package.searchers`. IIRC I did it this way to
provide better error messages, and to avoid fumbling with path strings.
The method is reasonably clean, so I'd feel much more confident to
modify it if the need arises. Should we include the "filters" path there
as well?

All this is currently undocumented. I'm going to add a paragraph to the
docs if we agree that it's sensible behavior.

--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Lua filter can not find lua package in data-dir

[jiewuza]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

Oops, I put it into ~/.local/share directly.


-- 


John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> 写入消息新闻:
> 
> Oh, interesting. I looked at the code, and sure enough,
> it already DOES try to load modules from datadir.

----Android NewsGroup Reader----
http://www.piaohong.tk/newsgroup

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/rujrpi%241742%241%40ciao.gmane.io.

Re: Lua filter can not find lua package in data-dir

[John MacFarlane]

If we're looking in datadir, then adding filters/ to the
search path makes sense I guess.

But still thinking through whether we want it this way
at all. It seems a slight security risk; someone could
put a modified pandoc.List.lua in the user data directory
and all lua filters would then do unexpected things...

Perhaps we should always use the default data file for pandoc.lua
and pandoc.List.lua, and not give people the ability to override
these.  Then the question would remain how to give them the
ability to use custom modules in their Lua filters.

One option would be to add USERDATADIR/filters to the load
path (or perhaps using the indirect way you described); another
would be to add both that at base USERDATADIR; a third would
be to add the directory containing the filter being run.
It seems to me that this third approach is most natural and
most flexible.

> We don't modify the load path, but instead provide a custom package
> loader function in `package.searchers`. IIRC I did it this way to
> provide better error messages, and to avoid fumbling with path strings.
> The method is reasonably clean, so I'd feel much more confident to
> modify it if the need arises.

Does this approach give users a way to lock things down, if they
want to limit the search path?  (e.g. if they want to ensure that
the standard version of a library from lua rocks will take
precedence over a version in the datadir)

Re: Lua filter can not find lua package in data-dir

[Albert Krewinkel]

John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:

> If we're looking in datadir, then adding filters/ to the
> search path makes sense I guess.
>
> But still thinking through whether we want it this way
> at all. It seems a slight security risk; someone could
> put a modified pandoc.List.lua in the user data directory
> and all lua filters would then do unexpected things...

Oh. Right. If we want to defend against that, then we should also
rethink support for loading an init.lua file in the data dir when
initializing Lua.

> Perhaps we should always use the default data file for pandoc.lua
> and pandoc.List.lua, and not give people the ability to override
> these.

Just to be sure I understand: This means that `--data-dir` would not
affect library loading, correct? Or more down to the code, that we'd
be using `readDefaultDataFile` instead of `readDataFile` when
loading pandoc.lua, pandoc.List.lua. I'm highly in favor of the
idea.

Maybe we can salvage the `init.lua` feature if we checked for the
file exclusively in T.P.Shared.defaultUserDataDirs?

> Then the question would remain how to give them the
> ability to use custom modules in their Lua filters.
>
> One option would be to add USERDATADIR/filters to the load
> path (or perhaps using the indirect way you described); another
> would be to add both that at base USERDATADIR; a third would
> be to add the directory containing the filter being run.
> It seems to me that this third approach is most natural and
> most flexible.

Agreed.

>> We don't modify the load path, but instead provide a custom package
>> loader function in `package.searchers`. IIRC I did it this way to
>> provide better error messages, and to avoid fumbling with path strings.
>> The method is reasonably clean, so I'd feel much more confident to
>> modify it if the need arises.
>
> Does this approach give users a way to lock things down, if they
> want to limit the search path?  (e.g. if they want to ensure that
> the standard version of a library from lua rocks will take
> precedence over a version in the datadir)

One way would be to add the below to the top of a filter or init.lua:

    local datadir_loader = table.remove(package.searchers, 1)
    table.insert(package.searchers, datadir_loader)

The datadir load function will then be called as a last resort if
all other loading functions have failed.


--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Lua filter can not find lua package in data-dir

[John MacFarlane]

Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> writes:

> John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:
>
>> If we're looking in datadir, then adding filters/ to the
>> search path makes sense I guess.
>>
>> But still thinking through whether we want it this way
>> at all. It seems a slight security risk; someone could
>> put a modified pandoc.List.lua in the user data directory
>> and all lua filters would then do unexpected things...
>
> Oh. Right. If we want to defend against that, then we should also
> rethink support for loading an init.lua file in the data dir when
> initializing Lua.

Right.  I guess this is a documented feature, so rolling it back
might affect some people.  I'm not sure who relies on it.  Of
course, it's always possible to manually include something from your
filter, and I'd say it's better to be explicit.

>> Perhaps we should always use the default data file for pandoc.lua
>> and pandoc.List.lua, and not give people the ability to override
>> these.
>
> Just to be sure I understand: This means that `--data-dir` would not
> affect library loading, correct? Or more down to the code, that we'd
> be using `readDefaultDataFile` instead of `readDataFile` when
> loading pandoc.lua, pandoc.List.lua. I'm highly in favor of the
> idea.

Exactly.

> Maybe we can salvage the `init.lua` feature if we checked for the
> file exclusively in T.P.Shared.defaultUserDataDirs?

So, that would allow the system (e.g. debian) to change init.lua,
but not allow users to put init.lua in their user data directory,
right?  Is there a need for that?

>> Then the question would remain how to give them the
>> ability to use custom modules in their Lua filters.
>>
>> One option would be to add USERDATADIR/filters to the load
>> path (or perhaps using the indirect way you described); another
>> would be to add both that at base USERDATADIR; a third would
>> be to add the directory containing the filter being run.
>> It seems to me that this third approach is most natural and
>> most flexible.
>
> Agreed.
>
>>> We don't modify the load path, but instead provide a custom package
>>> loader function in `package.searchers`. IIRC I did it this way to
>>> provide better error messages, and to avoid fumbling with path strings.
>>> The method is reasonably clean, so I'd feel much more confident to
>>> modify it if the need arises.
>>
>> Does this approach give users a way to lock things down, if they
>> want to limit the search path?  (e.g. if they want to ensure that
>> the standard version of a library from lua rocks will take
>> precedence over a version in the datadir)
>
> One way would be to add the below to the top of a filter or init.lua:
>
>     local datadir_loader = table.remove(package.searchers, 1)
>     table.insert(package.searchers, datadir_loader)
>
> The datadir load function will then be called as a last resort if
> all other loading functions have failed.

If we do change to defaulting to the path containing the lua
filter, I'm not sure we need to worry about this, since the load
path will be more obvious. Still, we could document it in the
place where we document the load path.

Re: Lua filter can not find lua package in data-dir

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 5080 bytes --]

If you remove the automatic loading of $DATADIR/init.lua then please
provide some not-too-complicated way to load it or an equivalent file at
some other location explicitly. Perhaps add a function to pandoc.system as
syntactic sugar for prepending a directory to package.path, something like

``````lua
function pandoc.system:package_path (dir)
  dir = tostring(dir)
  package.path = dir:gsub('/?$', '/?.lua;') .. package.path
end
``````

It seems to me that this would be a reasonable compromise: the filter
author must actively specify a directory to search rather than just loading
stuff behind the scene, but it can be done relatively easily. Also perhaps
set PANDOC_DATA_DIR and PANDOC_RESOURCE_PATH and why not also
PANDOC_DEFAULTS_FILE and perhaps also a function to remove the file part of
a path, something like

``````lua
pandoc.system:path_to_dir (path)
  return tostring(path):gsub('%/[^/.]+%.[^.]+$', '')
end
``````

Probably these should be adjusted for the possibility of a Windows path.
The way to get at the OS path separator in Lua is rather complicated, so
should perhaps also be provided, namely

``````lua
pandoc.system.path_sep = package.config:match"^[^\n]+"
``````
<https://www.lua.org/manual/5.3/manual.html#pdf-package.config>

although in practice it is '\' on Windows and '/' everywhere else, so
inspecting the OS name may be enough.

Just my two worthless coins!

/bpj


-- 
Better --help|less than helpless

Den sön 24 jan. 2021 22:29Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org>
skrev:

>
> John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:
>
> > If we're looking in datadir, then adding filters/ to the
> > search path makes sense I guess.
> >
> > But still thinking through whether we want it this way
> > at all. It seems a slight security risk; someone could
> > put a modified pandoc.List.lua in the user data directory
> > and all lua filters would then do unexpected things...
>
> Oh. Right. If we want to defend against that, then we should also
> rethink support for loading an init.lua file in the data dir when
> initializing Lua.
>
> > Perhaps we should always use the default data file for pandoc.lua
> > and pandoc.List.lua, and not give people the ability to override
> > these.
>
> Just to be sure I understand: This means that `--data-dir` would not
> affect library loading, correct? Or more down to the code, that we'd
> be using `readDefaultDataFile` instead of `readDataFile` when
> loading pandoc.lua, pandoc.List.lua. I'm highly in favor of the
> idea.
>
> Maybe we can salvage the `init.lua` feature if we checked for the
> file exclusively in T.P.Shared.defaultUserDataDirs?
>
> > Then the question would remain how to give them the
> > ability to use custom modules in their Lua filters.
> >
> > One option would be to add USERDATADIR/filters to the load
> > path (or perhaps using the indirect way you described); another
> > would be to add both that at base USERDATADIR; a third would
> > be to add the directory containing the filter being run.
> > It seems to me that this third approach is most natural and
> > most flexible.
>
> Agreed.
>
> >> We don't modify the load path, but instead provide a custom package
> >> loader function in `package.searchers`. IIRC I did it this way to
> >> provide better error messages, and to avoid fumbling with path strings.
> >> The method is reasonably clean, so I'd feel much more confident to
> >> modify it if the need arises.
> >
> > Does this approach give users a way to lock things down, if they
> > want to limit the search path?  (e.g. if they want to ensure that
> > the standard version of a library from lua rocks will take
> > precedence over a version in the datadir)
>
> One way would be to add the below to the top of a filter or init.lua:
>
>     local datadir_loader = table.remove(package.searchers, 1)
>     table.insert(package.searchers, datadir_loader)
>
> The datadir load function will then be called as a last resort if
> all other loading functions have failed.
>
>
> --
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/87o8hevxb8.fsf%40zeitkraut.de
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhD1ajpyoe5DvzxUdW-c-_RA_n%3Dc_sBYAByZ4tzC2x%3DG_w%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 7100 bytes --]

Re: Lua filter can not find lua package in data-dir

[John MacFarlane]

Should we infer that you're currently using this automatic
init.lua functionality -- but just to set a default package path?

Re: Lua filter can not find lua package in data-dir

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 1105 bytes --]

I use it to load some modules and helper functions which I don't want to
have in my regular Lua path (the helper functions), but if there was a less
verbose way to modify the package path, and the path to the user datq dir
was readily available I could easily insert the user data directory into
the package path at the top of my filters and then load the helper
functions from there.

-- 
Better --help|less than helpless

Den mån 25 jan. 2021 17:52John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> skrev:

>
> Should we infer that you're currently using this automatic
> init.lua functionality -- but just to set a default package path?
>
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhDUH8pnw%2B6oQzFBXZP63i8oT%3DT-yyjyH6C4KjbVukZZ5w%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 1714 bytes --]

Re: Lua filter can not find lua package in data-dir

[Albert Krewinkel]

BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> If you remove the automatic loading of $DATADIR/init.lua

I created a PR which would still load `init.lua` from
$HOME/.local/shared/pandoc/init.lua, but not from a different data
directory that has been selected with `--data-dir`.
https://github.com/jgm/pandoc/pull/7048

About path handling: I have an unfinished library lying around which
would expose some of Haskell's path handling functions to Lua. It
ensures platform-appropriate behavior and would allow us to avoid
any direct string-mangling.

--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Lua filter can not find lua package in data-dir

[John MacFarlane]

Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> writes:

> I created a PR which would still load `init.lua` from
> $HOME/.local/shared/pandoc/init.lua, but not from a different data
> directory that has been selected with `--data-dir`.
> https://github.com/jgm/pandoc/pull/7048

Actually the security worry I had concerns this -- the default
user data directory -- and is not specifically directed at the
case where someone uses --data-dir.

The former case is actually more troublesome, because files from
the default user data directory are being used without any
command line option to trigger that.  Someone could insert an
init.lua there that does virtually anything every time pandoc
is run with a lua filter, and the user wouldn't know it was
happening.

On the other hand, you might argue that once someone
gets access to your home directory, the game's already up as
far as security goes.  After all, you could just as easily
do something like put in your own bash script wrapping pandoc
in a local bin directory.

So maybe we shouldn't worry about this.  I'm just not sure.

> About path handling: I have an unfinished library lying around which
> would expose some of Haskell's path handling functions to Lua. It
> ensures platform-appropriate behavior and would allow us to avoid
> any direct string-mangling.

Seems helpful.

Re: Lua filter can not find lua package in data-dir

[Albert Krewinkel]

John MacFarlane <jgm-TVLZxgkOlNX2fBVCVOL8/A@public.gmane.org> writes:

> Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> writes:
>
>> I created a PR which would still load `init.lua` from
>> $HOME/.local/shared/pandoc/init.lua, but not from a different data
>> directory that has been selected with `--data-dir`.
>> https://github.com/jgm/pandoc/pull/7048
>
> Actually the security worry I had concerns this -- the default
> user data directory -- and is not specifically directed at the
> case where someone uses --data-dir.
>
> The former case is actually more troublesome, because files from
> the default user data directory are being used without any
> command line option to trigger that.  Someone could insert an
> init.lua there that does virtually anything every time pandoc
> is run with a lua filter, and the user wouldn't know it was
> happening.
>
> On the other hand, you might argue that once someone
> gets access to your home directory, the game's already up as
> far as security goes.  After all, you could just as easily
> do something like put in your own bash script wrapping pandoc
> in a local bin directory.
>
> So maybe we shouldn't worry about this.  I'm just not sure.

That makes sense to me. I tend to think that the init.lua is an
acceptable risk, but I'm not sure either.

The changes to loading `pandoc.lua` and `pandoc.List.lua` on the other
hand do seem like a necessary improvement, so I'll move those to a
separate PR.

>> About path handling: I have an unfinished library lying around which
>> would expose some of Haskell's path handling functions to Lua. It
>> ensures platform-appropriate behavior and would allow us to avoid
>> any direct string-mangling.
>
> Seems helpful.

Then I'm going to polish it up and create a PR to include it in pandoc.


--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Lua filter can not find lua package in data-dir

[Albert Krewinkel]

BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> If you remove the automatic loading of $DATADIR/init.lua then please
> provide some not-too-complicated way to load it or an equivalent file at
> some other location explicitly.

I've created a draft PR to add a new `pandoc.path` module:
<https://github.com/jgm/pandoc/pull/7053>

For the available functions see
<https://github.com/hslua/hslua-module-path#path>.

Example usage:

    local path = require 'pandoc.path'
    loadfile(path.join{
      path.directory(PANDOC_SCRIPT_FILE),
      'lua-packages',
      'helpers.lua'
    })

I included an experimental feature which allows to treat all strings as
file path objects, usable with overloaded operators. E.g.:

    (require 'pandoc.path').treat_strings_as_paths()
    loadfile(PANDOC_SCRIPT_PATH:directory() / 'lua-packages' / 'helpers.lua')

We could add an additional function `pandoc.system.xdg_directory`, which
should make it easy to find the correct path. E.g.,

    loadfile(path.join {pandoc.system.xdg_directory 'pandoc', 'init.lua'})

Could that work as an acceptable solution?

--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: Lua filter can not find lua package in data-dir

[John MacFarlane]

Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org> writes:


> We could add an additional function `pandoc.system.xdg_directory`, which
> should make it easy to find the correct path. E.g.,


Would it make more sense to expose the user data directory,
which might be the xdg_directory, might be legacy ~/.pandoc,
and might be something else (if they use --datadir on command
line)?

Or perhaps both?

Re: Lua filter can not find lua package in data-dir

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 2938 bytes --]

Albert, do you know that the default Lua package path separator may be
different from the system path separator? It is semicolon by default. See
`package.config` in the Lua reference manual. It probably makes sense to
have a field which holds the package path separator or even all of the
package.config strings, plus a function which prepends a package path to
package.path, probably including appending `/?.lua` or its equivalent to
the prepended path automagically unless it already end in a something like
`/?.ext`. Perhaps even a subclass of the path class which represents a
package. path value?

BTW great minds think alike. I started writing a path class in Lua the
other day prompted by this thread!


-- 
Better --help|less than helpless

Den ons 27 jan. 2021 18:22Albert Krewinkel <albert+pandoc-9EawChwDxG8hFhg+JK9F0w@public.gmane.org>
skrev:

> BPJ <melroch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
> > If you remove the automatic loading of $DATADIR/init.lua then please
> > provide some not-too-complicated way to load it or an equivalent file at
> > some other location explicitly.
>
> I've created a draft PR to add a new `pandoc.path` module:
> <https://github.com/jgm/pandoc/pull/7053>
>
> For the available functions see
> <https://github.com/hslua/hslua-module-path#path>.
>
> Example usage:
>
>     local path = require 'pandoc.path'
>     loadfile(path.join{
>       path.directory(PANDOC_SCRIPT_FILE),
>       'lua-packages',
>       'helpers.lua'
>     })
>
> I included an experimental feature which allows to treat all strings as
> file path objects, usable with overloaded operators. E.g.:
>
>     (require 'pandoc.path').treat_strings_as_paths()
>     loadfile(PANDOC_SCRIPT_PATH:directory() / 'lua-packages' /
> 'helpers.lua')
>
> We could add an additional function `pandoc.system.xdg_directory`, which
> should make it easy to find the correct path. E.g.,
>
>     loadfile(path.join {pandoc.system.xdg_directory 'pandoc', 'init.lua'})
>
> Could that work as an acceptable solution?
>
> --
> Albert Krewinkel
> GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/87ft2mwb1n.fsf%40zeitkraut.de
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhC1Ho7Z5xMiwzLBmoFk5%3DFN29HgDtCr0o2EmZ%2Bz8wLK8w%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 4443 bytes --]