cgit-lua: to jit or not to jit

cgit-lua: to jit or not to jit

[Jason]

Hi folks,

Over at the cgit project [1], we're in the process of adding Lua
support. Things are going swimmingly, and I've just merged to the
master tree some commits adding full lua support [2] to our filter API
[3] and an example script [4]. At the moment we've got some moderately
messy logic in our makefile [6] that autodetects if LuaJIT is
installed, and if it isn't, falls back to mainline Lua.

What I'm wondering is -- what is the purpose of keeping around support
for mainline Lua? Why shouldn't we just go with LuaJIT and be done
with it? I'm unable to find any advantages mainline Lua has over
LuaJIT, which is why I'm emailing these lists. What features am I
forgetting about that would make some people prefer mainline Lua over
LuaJIT?

An additional motivation for wanting to go to LuaJIT is the FFI
library that comes with it [5]. It seems like this would make
packaging and swapping scripts a lot easier. For example, in the
gravatar script [4], I depend on luacrypto, which some users might not
want to install or have easily available. With FFI, I could instead
just wrap OpenSSL (which cgit already depends on). But perhaps there
are some obvious downsides to this approach that I also am missing.

My experience with embedding Lua inside cgit so far has been very
nice. What a clean simple API. I look forward to hearing your
responses and any feedback and suggestions you might have.

Thank you,
Jason Donenfeld


[1] http://git.zx2c4.com/cgit/about/
[2] http://git.zx2c4.com/cgit/tree/filter.c#n176
[3] http://git.zx2c4.com/cgit/tree/cgitrc.5.txt#n569
[4] http://git.zx2c4.com/cgit/tree/filters/email-gravatar.lua
[5] http://luajit.org/ext_ffi.html
[6] http://git.zx2c4.com/cgit/tree/cgit.mk#n30

cgit-lua: to jit or not to jit

[Jason]

On Tue, Jan 14, 2014 at 3:10 AM, demetri <demetri.spanos at gmail.com> wrote:
> 1) PUC Lua works on more machines/architectures (though Mike has
> substantially closed the gap for most architectures people care about in a
> non-embedded non-mobile environment)

Do you know what the gap is?

LuaJIT supports x86, x86_x64, arm, ppc, e500, and mips. I guess that
leaves, what? Sparc, AVR, ia64, ... other obscure ones? Hmm..

>
> 2) some Linux distributions include PUC Lua but few have LuaJIT installed
> by default, so from the user's perspective LuaJIT is one more dependency.
> Some people care about this and others don't.

Good point.

cgit-lua: to jit or not to jit

[Jason]

On Tue, Jan 14, 2014 at 3:12 AM, Tim Hill <drtimhill at gmail.com> wrote:
> LuaJIT is currently compatible with Lua 5.1, not 5.2 or the upcoming 5.3, and my understanding is the LuaJIT project has now forked from mainstream Lua and will not be staying in sync with anything past 5.1, so I would expect over time the two to diverge.

AFAIK, LuaJIT supports 5.2 currently via the
-DLUAJIT_ENABLE_LUA52COMPA flag and mentions it here
http://luajit.org/extensions.html#lua52 . OTOH, their roadmap page
says:

> As I've previously said, Lua 5.2 provides few tangible benefits.
> LuaJIT already includes the major new features, without breaking
> compatibility. Upgrading to be compatible with 5.2, just for the
> sake of a higher version number, is neither a priority nor a
> sensible move for most LuaJIT users.

So I'm not sure what to think.

cgit-lua: to jit or not to jit

[andrew.starks]

On Monday, January 13, 2014, Jason A. Donenfeld wrote:

> On Tue, Jan 14, 2014 at 3:12 AM, Tim Hill <drtimhill at gmail.com<javascript:;>>
> wrote:
> > LuaJIT is currently compatible with Lua 5.1, not 5.2 or the upcoming
> 5.3, and my understanding is the LuaJIT project has now forked from
> mainstream Lua and will not be staying in sync with anything past 5.1, so I
> would expect over time the two to diverge.
>
> AFAIK, LuaJIT supports 5.2 currently via the
> -DLUAJIT_ENABLE_LUA52COMPA flag and mentions it here
> http://luajit.org/extensions.html#lua52 . OTOH, their roadmap page
> says:
>
> > As I've previously said, Lua 5.2 provides few tangible benefits.
> > LuaJIT already includes the major new features, without breaking
> > compatibility. Upgrading to be compatible with 5.2, just for the
> > sake of a higher version number, is neither a priority nor a
> > sensible move for most LuaJIT users.
>
> So I'm not sure what to think.
>
>
I don't have LuaJit installed and would not install it (and migrate
everything I do over to it) just to use a library. By way of example, your
library may as well have been written for Python, for as much good as it
would be to me.

By contrast, if you stick to the subset of 5.2 that 5.1 supports, and / or
use a bit of the luacomp library, then anyone with lua 5.1, luajit or Lua
5.2 can use it.

The question, from a user's perspective is: what benefit are you giving me,
in exchange for locking me into luajit, as a dependency?

Even if I am using Luajit, that doesn't mean that I don't need to support
the current, mainline distribution and straight 5.1. So, I can't use your
library as a dependency, if this were the case.

It's easier for you if you like what the FFI gives you. Supporting the
common subset and using luacompat, as necessary, is the simplest, for the
user.

IMHO, of course

-Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20140113/5bee1618/attachment-0001.html>

cgit-lua: to jit or not to jit

[Jason]

On Tue, Jan 14, 2014 at 3:34 AM, William Ahern
<william at 25thandclement.com> wrote:
> The 2GB memory limit on 64-bit was a huge blocker at my work. We almost
> immeditely ran afoul of it and have chosen to stabilize on Lua 5.2 for one
> of our cloud services. (On 16-way SMP, 32GB RAM boxes it's more important to
> be able to scale up and out then to have JITd loops, especially when the
> most performance sensitive code is in C anyhow.)
>
> I suspect that if you ended up trying to write a full OpenSSL wrapper using
> LuaJIT's FFI that you wouldn't save very much time and effort. I say that as
> someone having written the most comprehensive OpenSSL bindings in Lua:
>
>         http://25thandclement.com/~william/projects/luaossl.html
>
> Things might be different if you just bound one or two interfaces. LuaJITs
> FFI excels at that sort of ad hoc interface binding. But for full-blown
> bindings things get more complicated. OpenSSL, for example has a ton of
> preprocessor generated tags. Also, string and buffer management can be made
> significantly more efficient in C, so when you're aggregating results into a
> large buffer before pushing onto the Lua stack, it's easier to get good
> performance with C code. For larger or complex modules the payoffs with
> LuaJIT FFI diminish. So it all depends on your context. How sophisticated
> will your bindings be?

In this case, it'd just be calculating an md5 of an email address to
display a gravatar. So at the moment, super simple. Good point about
the cost of aggregation for bigger uses, though.

Generally, our use case is -- cgit spits out a bunch of HTML.
Sometimes sysadmins might want to modify it on the fly based on
various pieces of information -- for example, adding a gravatar image
next to mentions of an author based on email address -- in which case,
the admin codes up a simple lua file [1]. Super super simple usage.

>
> Because Lua is emebedded in your project it's not that big of deal to just
> settle on LuaJIT. Having to deal with the different versions is more of a
> problem for a module writer, where you run into problems with differences
> across versions and implementations--e.g. GC (Lua 5.2 has ephemeron tables)
> or internals (e.g. FILE* handles).
>
>

That's a good point, okay.

[1] http://git.zx2c4.com/cgit/tree/filters/email-gravatar.lua

cgit-lua: to jit or not to jit

[Jason]

On Tue, Jan 14, 2014 at 3:37 AM, Andrew Starks <andrew.starks at trms.com> wrote:
>
> I don't have LuaJit installed and would not install it (and migrate
> everything I do over to it) just to use a library. By way of example, your
> library may as well have been written for Python, for as much good as it
> would be to me.
>
> By contrast, if you stick to the subset of 5.2 that 5.1 supports, and / or
> use a bit of the luacomp library, then anyone with lua 5.1, luajit or Lua
> 5.2 can use it.
>
> The question, from a user's perspective is: what benefit are you giving me,
> in exchange for locking me into luajit, as a dependency?
>
> Even if I am using Luajit, that doesn't mean that I don't need to support
> the current, mainline distribution and straight 5.1. So, I can't use your
> library as a dependency, if this were the case.
>
> It's easier for you if you like what the FFI gives you. Supporting the
> common subset and using luacompat, as necessary, is the simplest, for the
> user.
>
> IMHO, of course

That's a fairly compelling opinion. The only thing against it is the
temptation of using FFI in the default scripts that we ship with cgit.
But I suppose for the sake of giving users choice later on, it might
be best, as you've said, to continue to support both, and let the user
choose.

cgit-lua: to jit or not to jit

[andrew.starks]

On Monday, January 13, 2014, Jason A. Donenfeld wrote:

> On Tue, Jan 14, 2014 at 3:37 AM, Andrew Starks <andrew.starks at trms.com<javascript:;>>
> wrote:
> >
> > I don't have LuaJit installed and would not install it (and migrate
> > everything I do over to it) just to use a library. By way of example,
> your
> > library may as well have been written for Python, for as much good as it
> > would be to me.
> >
> > By contrast, if you stick to the subset of 5.2 that 5.1 supports, and /
> or
> > use a bit of the luacomp library, then anyone with lua 5.1, luajit or Lua
> > 5.2 can use it.
> >
> > The question, from a user's perspective is: what benefit are you giving
> me,
> > in exchange for locking me into luajit, as a dependency?
> >
> > Even if I am using Luajit, that doesn't mean that I don't need to support
> > the current, mainline distribution and straight 5.1. So, I can't use your
> > library as a dependency, if this were the case.
> >
> > It's easier for you if you like what the FFI gives you. Supporting the
> > common subset and using luacompat, as necessary, is the simplest, for the
> > user.
> >
> > IMHO, of course
>
> That's a fairly compelling opinion. The only thing against it is the
> temptation of using FFI in the default scripts that we ship with cgit.


This is a great point, as well. Also, there are some spiciffic use cases
where speed might be critical and luajit has some common cases where it
really shines, no doubt.

But I suppose for the sake of giving users choice later on, it might
> be best, as you've said, to continue to support both, and let the user
> choose.
>

I'll also say that I will make it a point to check this out. I hadn't heard
of the project before and it sounds interesting.

-Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20140113/65a96271/attachment.html>

cgit-lua: to jit or not to jit

[andrew.starks]

On Monday, January 13, 2014, Jason A. Donenfeld wrote:

> On Tue, Jan 14, 2014 at 3:37 AM, Andrew Starks <andrew.starks at trms.com<javascript:;>>
> wrote:
> >
> > I don't have LuaJit installed and would not install it (and migrate
> > everything I do over to it) just to use a library. By way of example,
> your
> > library may as well have been written for Python, for as much good as it
> > would be to me.
> >
> > By contrast, if you stick to the subset of 5.2 that 5.1 supports, and /
> or
> > use a bit of the luacomp library, then anyone with lua 5.1, luajit or Lua
> > 5.2 can use it.
> >
> > The question, from a user's perspective is: what benefit are you giving
> me,
> > in exchange for locking me into luajit, as a dependency?
> >
> > Even if I am using Luajit, that doesn't mean that I don't need to support
> > the current, mainline distribution and straight 5.1. So, I can't use your
> > library as a dependency, if this were the case.
> >
> > It's easier for you if you like what the FFI gives you. Supporting the
> > common subset and using luacompat, as necessary, is the simplest, for the
> > user.
> >
> > IMHO, of course
>
> That's a fairly compelling opinion. The only thing against it is the
> temptation of using FFI in the default scripts that we ship with cgit.
> But I suppose for the sake of giving users choice later on, it might
> be best, as you've said, to continue to support both, and let the user
> choose.
>

Jason,

I also just remembered this:

https://github.com/jmckaskill/luaffi

Which is a luajit compatible FFI extension for Lua 5.1 and Lua 5.2, but 5.2
is listed as beta. It might be worth a shot, if it lets you gain some of
those conveniences and keep a broad support base.

-Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20140113/1af6f73e/attachment.html>

cgit-lua: to jit or not to jit

[justin]

On 14 Jan 2014 03:52, "Andrew Starks" <andrew.starks at trms.com> wrote:
>
>
>
> On Monday, January 13, 2014, Jason A. Donenfeld wrote:
>>
>> On Tue, Jan 14, 2014 at 3:37 AM, Andrew Starks <andrew.starks at trms.com>
wrote:
>> >
>> > I don't have LuaJit installed and would not install it (and migrate
>> > everything I do over to it) just to use a library. By way of example,
your
>> > library may as well have been written for Python, for as much good as
it
>> > would be to me.
>> >
>> > By contrast, if you stick to the subset of 5.2 that 5.1 supports, and
/ or
>> > use a bit of the luacomp library, then anyone with lua 5.1, luajit or
Lua
>> > 5.2 can use it.
>> >
>> > The question, from a user's perspective is: what benefit are you
giving me,
>> > in exchange for locking me into luajit, as a dependency?
>> >
>> > Even if I am using Luajit, that doesn't mean that I don't need to
support
>> > the current, mainline distribution and straight 5.1. So, I can't use
your
>> > library as a dependency, if this were the case.
>> >
>> > It's easier for you if you like what the FFI gives you. Supporting the
>> > common subset and using luacompat, as necessary, is the simplest, for
the
>> > user.
>> >
>> > IMHO, of course
>>
>> That's a fairly compelling opinion. The only thing against it is the
>> temptation of using FFI in the default scripts that we ship with cgit.
>> But I suppose for the sake of giving users choice later on, it might
>> be best, as you've said, to continue to support both, and let the user
>> choose.
>
>
> Jason,
>
> I also just remembered this:
>
> https://github.com/jmckaskill/luaffi
>
> Which is a luajit compatible FFI extension for Lua 5.1 and Lua 5.2, but
5.2 is listed as beta. It might be worth a shot, if it lets you gain some
of those conveniences and keep a broad support base.
>

Lua 5.2 support is fine for luaffi, but both are beta, and you have to fix
the bugs yourself as it is only intermittently being developed (by me).

The most important platform that LuaJIT is not available for is probably
softfloat MIPS (eg openwrt).

Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20140114/f451921c/attachment-0001.html>

cgit-lua: to jit or not to jit

[Jason]

On Tue, Jan 14, 2014 at 4:51 AM, Andrew Starks <andrew.starks at trms.com> wrote:
>
> I also just remembered this:
>
> https://github.com/jmckaskill/luaffi
>
> Which is a luajit compatible FFI extension for Lua 5.1 and Lua 5.2, but 5.2
> is listed as beta. It might be worth a shot, if it lets you gain some of
> those conveniences and keep a broad support base.

Excellent! Thanks Andrew.

cgit-lua: to jit or not to jit

[fw]

* Jason A. Donenfeld:

> On Tue, Jan 14, 2014 at 3:10 AM, demetri <demetri.spanos at gmail.com> wrote:
>> 1) PUC Lua works on more machines/architectures (though Mike has
>> substantially closed the gap for most architectures people care about in a
>> non-embedded non-mobile environment)
>
> Do you know what the gap is?
>
> LuaJIT supports x86, x86_x64, arm, ppc, e500, and mips. I guess that
> leaves, what? Sparc, AVR, ia64, ... other obscure ones? Hmm..

ppc64 and s390x.  Neither one is particularly obscure.

cgit-lua: to jit or not to jit

[ewmailing]

> But perhaps there
> are some obvious downsides to this approach that I also am missing.

Here are a few more potential downsides not mentioned so far.

One is that iOS and Windows Store policies disallow JIT. While you can
disable this part in LuaJIT, most of the performance advantages
disappear when you do this.

Additionally, if you want/need to modify Lua, it is much easier to
modify canonical Lua as opposed to LuaJIT. It is also easy to find Lua
patches ("power patches").

On a similar vein, I would imagine auditing code for things like
security or high availability would be easier with canonical Lua.
(Look to Wikimedia Commons and Verisign talks about what they needed
to accomplish.) If you or you have users that worry about this kind of
thing, they may be more reassured if you retain compatibility with
both.

Thanks,
Eric
-- 
Beginning iPhone Games Development
http://playcontrol.net/iphonegamebook/

cgit-lua: to jit or not to jit

[chighland]

On Tue, Jan 14, 2014 at 12:41 PM, Eric Wing <ewmailing at gmail.com> wrote:
>> But perhaps there
>> are some obvious downsides to this approach that I also am missing.
>
> Here are a few more potential downsides not mentioned so far.
>
> One is that iOS and Windows Store policies disallow JIT. While you can
> disable this part in LuaJIT, most of the performance advantages
> disappear when you do this.

However, even with the JIT disabled, LuaJIT does tend to outperform
Lua for a lot of use cases (Lua wins in a few cases) just by virtue of
having an interpreter written in hand-tuned assembly.

/s/ Adam

cgit-lua: to jit or not to jit

[chighland]

On Tue, Jan 14, 2014 at 12:50 PM, Coda Highland <chighland at gmail.com> wrote:
> On Tue, Jan 14, 2014 at 12:41 PM, Eric Wing <ewmailing at gmail.com> wrote:
>>> But perhaps there
>>> are some obvious downsides to this approach that I also am missing.
>>
>> Here are a few more potential downsides not mentioned so far.
>>
>> One is that iOS and Windows Store policies disallow JIT. While you can
>> disable this part in LuaJIT, most of the performance advantages
>> disappear when you do this.
>
> However, even with the JIT disabled, LuaJIT does tend to outperform
> Lua for a lot of use cases (Lua wins in a few cases) just by virtue of
> having an interpreter written in hand-tuned assembly.
>
> /s/ Adam

Addendum: FFI still works on iOS at least, and probably WinRT.

/s/ Adam

cgit-lua: to jit or not to jit

[malkia]

One more thing for luajit vs. lua - If you ship a plugin (.so, .dll,
.dylib) using luajit, then the host application might not always work
with it. If I'm not mistaken the lower 2GB (or 4GB?) must be allocated
to luajit, (so even in 64-bit app, it needs the lower 32-bit address
space - for lua allocated objects).


On Tue, Jan 14, 2014 at 12:50 PM, Coda Highland <chighland at gmail.com> wrote:
> On Tue, Jan 14, 2014 at 12:50 PM, Coda Highland <chighland at gmail.com> wrote:
>> On Tue, Jan 14, 2014 at 12:41 PM, Eric Wing <ewmailing at gmail.com> wrote:
>>>> But perhaps there
>>>> are some obvious downsides to this approach that I also am missing.
>>>
>>> Here are a few more potential downsides not mentioned so far.
>>>
>>> One is that iOS and Windows Store policies disallow JIT. While you can
>>> disable this part in LuaJIT, most of the performance advantages
>>> disappear when you do this.
>>
>> However, even with the JIT disabled, LuaJIT does tend to outperform
>> Lua for a lot of use cases (Lua wins in a few cases) just by virtue of
>> having an interpreter written in hand-tuned assembly.
>>
>> /s/ Adam
>
> Addendum: FFI still works on iOS at least, and probably WinRT.
>
> /s/ Adam
>



-- 
Dimiter "malkia" Stanev,
ICQ: 21875894
malkia at mac.com
malkia at gmail.com