[NTG-context] Calling C code from ConTeXt LMTX

[NTG-context] Calling C code from ConTeXt LMTX

[Florent Michel]

[-- Attachment #1.1: Type: text/plain, Size: 2457 bytes --]

Hello,

TL;DR: Could building LuaMetaTeX with the `-fPIC` flag for the Lua library
have negative consequences? Is there a better way to be able to call
external C functions from ConTeXt?

I am working on a small project where it would be convenient to call
functions written in C (possibly pre-compiled in a shared library) from
ConTeXt. After looking at the Lua FFI, I found a way that seems to work
(with ConTeXt LMTX 2024-08-16) but requires a small tweak to the cmake file
used for building the Lua library. I would be grateful to have the opinion
of people more knowledgeable than I am about whether this change could have
negative consequences and whether there is a more straightforward way to
call external C code.

What I am trying to achieve is:
* Compile some C code in a shared library (for instance, it may contain a
numerical solver for some partial differential equation).
* Have this code run when compiling a document with ConTeXt. (For instance,
solving the differential equation with parameters defined in the .tex file
and retrieving the results to plot a figure with MetaPost.)

One easy way to do that is:
* Write wrappers for the functions that will need to be called from ConTeXt
using the Lua C API.
* Compile the code and wrappers in a shared library and linking to the
static lua library used by LuaMetaTeX (liblua.a on Linux).
* In the .tex file, import the new library using, e.g.,
```
\startluacode
    mylib = require("mylib")
    -- additional code using the C functions
\stopluacode
```
* Run ConTeXt with the `--permitloadlib` flag.

However, the second step requires (at least on Linux) liblua.a to be
compiled with the `-fPIC` flag. I could do that by adding
```
set_property(TARGET lua PROPERTY POSITION_INDEPENDENT_CODE ON)
```
at line 47 in lua.cmake. Is there any known negative effect of doing that
(apart from a possible small performance drop)?
(Just to be clear, this is not a feature request; I'm just wondering
whether doing it on my side is fine.)

On a related note, is there a standard way to call external C functions
(either in a shared library or in a .c file) from ConTeXt LMTX? The SwigLib
library described in
https://www.pragma-ade.nl/general/manuals/swiglib-mkiv.pdf looks very
promising (and maybe could be a way to achieve what I'm trying to do in a
much cleaner way), but I (maybe for lack of trying hard enough) have not
been able to make it work with ConTeXt LMTX so far.

Best regards,
Florent

[-- Attachment #1.2: Type: text/html, Size: 2708 bytes --]

[-- Attachment #2: Type: text/plain, Size: 511 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Calling C code from ConTeXt LMTX

[Bruce Horrocks]

> On 14 Sep 2024, at 21:46, Florent Michel <florent.m42@gmail.com> wrote:
> 
> TL;DR: Could building LuaMetaTeX with the `-fPIC` flag for the Lua library have negative consequences? Is there a better way to be able to call external C functions from ConTeXt?

This seems a bit drastic. Why can’t you achieve what you want to achieve by using other methods?

For example you could setup a workflow whereby you pre-calculate your DE’s and format the results to include as input files. Or you could pre-calculate your DEs and plot them using one of any number of plotting programs and include the plots as TIFFs or PNGs.

If you’re familiar with compiling C code using tools like `make` then why not use a makefile to control the production of your DE plots and your ConTeXt document to ensure that everything is consistent?

If you absolutely must run the DE calculations as the ConTeXt document is processed then there is \executesystemcommand.

—
Bruce Horrocks
Hampshire, UK

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Calling C code from ConTeXt LMTX

[Florent Michel]

[-- Attachment #1.1: Type: text/plain, Size: 2247 bytes --]

Thanks Bruce for your very helpful reply!

The reason for my question was indeed consistency, e.g. ensuring figure
captions stay up to date with what the figures show when changing
parameters. Thinking more about it, the second solution you mention seems
to be a better option, though - I can simply define the parameters in an
external file and read it from both the PDF solver and ConTeXt to ensure
consistency. Thanks for mentioning it!

Thank you also for mentioning \executesystemcommand, which I was not aware
of!

Best regards,
Florent

Le dim. 15 sept. 2024 à 21:31, Bruce Horrocks <ntg@scorecrow.com> a écrit :

>
>
> > On 14 Sep 2024, at 21:46, Florent Michel <florent.m42@gmail.com> wrote:
> >
> > TL;DR: Could building LuaMetaTeX with the `-fPIC` flag for the Lua
> library have negative consequences? Is there a better way to be able to
> call external C functions from ConTeXt?
>
> This seems a bit drastic. Why can’t you achieve what you want to achieve
> by using other methods?
>
> For example you could setup a workflow whereby you pre-calculate your DE’s
> and format the results to include as input files. Or you could
> pre-calculate your DEs and plot them using one of any number of plotting
> programs and include the plots as TIFFs or PNGs.
>
> If you’re familiar with compiling C code using tools like `make` then why
> not use a makefile to control the production of your DE plots and your
> ConTeXt document to ensure that everything is consistent?
>
> If you absolutely must run the DE calculations as the ConTeXt document is
> processed then there is \executesystemcommand.
>
> —
> Bruce Horrocks
> Hampshire, UK
>
>
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to
> the Wiki!
>
> maillist : ntg-context@ntg.nl /
> https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
> webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
> archive  : https://github.com/contextgarden/context
> wiki     : https://wiki.contextgarden.net
>
> ___________________________________________________________________________________
>

[-- Attachment #1.2: Type: text/html, Size: 3112 bytes --]

[-- Attachment #2: Type: text/plain, Size: 511 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Calling C code from ConTeXt LMTX

[Bruce Horrocks]

> On 15 Sep 2024, at 22:11, Florent Michel <florent.m42@gmail.com> wrote:
> 
> Thanks Bruce for your very helpful reply! 
> 
> The reason for my question was indeed consistency, e.g. ensuring figure captions stay up to date with what the figures show when changing parameters. Thinking more about it, the second solution you mention seems to be a better option, though - I can simply define the parameters in an external file and read it from both the PDF solver and ConTeXt to ensure consistency. Thanks for mentioning it!  
> 
> Thank you also for mentioning \executesystemcommand, which I was not aware of! 

Another option might be to use a marker such as

% DE_figure_here param1 param2 param3

in your ConTeXt source and then use 'awk' or another Unix text pre-processor to scan through for these, run the appropriate DE calculation, plot the graph and generate the appropriate \placefigure command to go in its place.

Bonus marks if it can automatically generate the caption based on the parameters! But if not, the marker can always follow the \placefigure macro e.g.

\placefigure
  {This is a DE showing something and something else}
  % DE_figure_here param1 param2 param3

so the awk code only needs to insert the image file name.

Note I’ve deliberately omitted the braces from the bit to be substituted so that you need to generate them in the awk. That way, if you forget to process, you’ll get an error not a missing image.

HTH
—
Bruce Horrocks
Hampshire, UK

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Calling C code from ConTeXt LMTX

[Henning Hraban Ramm]

Am 16.09.24 um 12:59 schrieb Bruce Horrocks:
> 
> 
>> On 15 Sep 2024, at 22:11, Florent Michel <florent.m42@gmail.com> wrote:
>>
>> Thanks Bruce for your very helpful reply!
>>
>> The reason for my question was indeed consistency, e.g. ensuring figure captions stay up to date with what the figures show when changing parameters. Thinking more about it, the second solution you mention seems to be a better option, though - I can simply define the parameters in an external file and read it from both the PDF solver and ConTeXt to ensure consistency. Thanks for mentioning it!
>>
>> Thank you also for mentioning \executesystemcommand, which I was not aware of!
> 
> Another option might be to use a marker such as
> 
> % DE_figure_here param1 param2 param3
> 
> in your ConTeXt source and then use 'awk' or another Unix text pre-processor to scan through for these, run the appropriate DE calculation, plot the graph and generate the appropriate \placefigure command to go in its place.
> 
> Bonus marks if it can automatically generate the caption based on the parameters! But if not, the marker can always follow the \placefigure macro e.g.
> 
> \placefigure
>    {This is a DE showing something and something else}
>    % DE_figure_here param1 param2 param3
> 
> so the awk code only needs to insert the image file name.
> 
> Note I’ve deliberately omitted the braces from the bit to be substituted so that you need to generate them in the awk. That way, if you forget to process, you’ll get an error not a missing image.

That sounds needlessly convoluted to me.
I’d use Aditya’s filter module to call external programs, it also does 
caching, i.e. if the parameters don’t change, it won’t waste processing 
time.

https://github.com/adityam/filter

Hraban
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Calling C code from ConTeXt LMTX

[Bruce Horrocks]

On 16 Sep 2024, at 12:15, Henning Hraban Ramm <texml@fiee.net> wrote:
> 
> Am 16.09.24 um 12:59 schrieb Bruce Horrocks:
>>> On 15 Sep 2024, at 22:11, Florent Michel <florent.m42@gmail.com> wrote:
>>> 
>>> Thanks Bruce for your very helpful reply!
>>> 
>>> The reason for my question was indeed consistency, e.g. ensuring figure captions stay up to date with what the figures show when changing parameters. Thinking more about it, the second solution you mention seems to be a better option, though - I can simply define the parameters in an external file and read it from both the PDF solver and ConTeXt to ensure consistency. Thanks for mentioning it!
>>> 
>>> Thank you also for mentioning \executesystemcommand, which I was not aware of!
>> Another option might be to use a marker such as
>> % DE_figure_here param1 param2 param3
>> in your ConTeXt source and then use 'awk' or another Unix text pre-processor to scan through for these, run the appropriate DE calculation, plot the graph and generate the appropriate \placefigure command to go in its place.
>> Bonus marks if it can automatically generate the caption based on the parameters! But if not, the marker can always follow the \placefigure macro e.g.
>> \placefigure
>>   {This is a DE showing something and something else}
>>   % DE_figure_here param1 param2 param3
>> so the awk code only needs to insert the image file name.
>> Note I’ve deliberately omitted the braces from the bit to be substituted so that you need to generate them in the awk. That way, if you forget to process, you’ll get an error not a missing image.
> 
> That sounds needlessly convoluted to me.
> I’d use Aditya’s filter module to call external programs, it also does caching, i.e. if the parameters don’t change, it won’t waste processing time.
> 
> https://github.com/adityam/filter

I think Florent already has a situation that can be described as convoluted. Whether this is more so only he can say.

I didn’t suggest the filter module to start with because it filters a buffer through an external command which isn’t Florent’s use-case (which is just the ability to call an external command). However I see, buried deeply in the docs, under the heading "Special use case:  \write18 with caching” there is the ability to call a single command which would work.

Someone with better knowledge of the macro would need to explain how to adapt the ’size’ key to Florent’s usage to enable the caching.

Regards,
—
Bruce Horrocks
Hampshire, UK

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________

[NTG-context] Re: Calling C code from ConTeXt LMTX

[Florent Michel]

[-- Attachment #1.1: Type: text/plain, Size: 3960 bytes --]

Thanks again Bruce! I had not thought about the possibility of parsing
comments in the tex file from an external tool, but that sounds like a very
good option. I like the fact that it would allow the tex file to contain
all the information needed to keep track of what is represented without
having to duplicate the parameter values. And since my specific workflow is
indeed already convoluted it will not really add extra complexity.

Thanks also Henning for mentioning the filter module! I'll have a closer
look into how I could use it - and it will certainly come useful, either
for this project or a future one!

Best regards,
Florent

Le lun. 16 sept. 2024 à 13:08, Bruce Horrocks <ntg@scorecrow.com> a écrit :

> On 16 Sep 2024, at 12:15, Henning Hraban Ramm <texml@fiee.net> wrote:
> >
> > Am 16.09.24 um 12:59 schrieb Bruce Horrocks:
> >>> On 15 Sep 2024, at 22:11, Florent Michel <florent.m42@gmail.com>
> wrote:
> >>>
> >>> Thanks Bruce for your very helpful reply!
> >>>
> >>> The reason for my question was indeed consistency, e.g. ensuring
> figure captions stay up to date with what the figures show when changing
> parameters. Thinking more about it, the second solution you mention seems
> to be a better option, though - I can simply define the parameters in an
> external file and read it from both the PDF solver and ConTeXt to ensure
> consistency. Thanks for mentioning it!
> >>>
> >>> Thank you also for mentioning \executesystemcommand, which I was not
> aware of!
> >> Another option might be to use a marker such as
> >> % DE_figure_here param1 param2 param3
> >> in your ConTeXt source and then use 'awk' or another Unix text
> pre-processor to scan through for these, run the appropriate DE
> calculation, plot the graph and generate the appropriate \placefigure
> command to go in its place.
> >> Bonus marks if it can automatically generate the caption based on the
> parameters! But if not, the marker can always follow the \placefigure macro
> e.g.
> >> \placefigure
> >>   {This is a DE showing something and something else}
> >>   % DE_figure_here param1 param2 param3
> >> so the awk code only needs to insert the image file name.
> >> Note I’ve deliberately omitted the braces from the bit to be
> substituted so that you need to generate them in the awk. That way, if you
> forget to process, you’ll get an error not a missing image.
> >
> > That sounds needlessly convoluted to me.
> > I’d use Aditya’s filter module to call external programs, it also does
> caching, i.e. if the parameters don’t change, it won’t waste processing
> time.
> >
> > https://github.com/adityam/filter
>
> I think Florent already has a situation that can be described as
> convoluted. Whether this is more so only he can say.
>
> I didn’t suggest the filter module to start with because it filters a
> buffer through an external command which isn’t Florent’s use-case (which is
> just the ability to call an external command). However I see, buried deeply
> in the docs, under the heading "Special use case:  \write18 with caching”
> there is the ability to call a single command which would work.
>
> Someone with better knowledge of the macro would need to explain how to
> adapt the ’size’ key to Florent’s usage to enable the caching.
>
> Regards,
> —
> Bruce Horrocks
> Hampshire, UK
>
>
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to
> the Wiki!
>
> maillist : ntg-context@ntg.nl /
> https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
> webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
> archive  : https://github.com/contextgarden/context
> wiki     : https://wiki.contextgarden.net
>
> ___________________________________________________________________________________
>

[-- Attachment #1.2: Type: text/html, Size: 5123 bytes --]

[-- Attachment #2: Type: text/plain, Size: 511 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl
webpage  : https://www.pragma-ade.nl / https://context.aanhet.net (mirror)
archive  : https://github.com/contextgarden/context
wiki     : https://wiki.contextgarden.net
___________________________________________________________________________________