Re: Any initial thoughts on luau?

["Michal Vlasák via ntg-context" <ntg-context@ntg.nl>]

On Fri Nov 5, 2021 at 9:14 PM CET, Hans Hagen via ntg-context wrote:
> On 11/5/2021 6:23 PM, Ramkumar KB via ntg-context wrote:
> > Hello,
> > 
> > Yesterday, Roblox announced - Luau - https://luau-lang.org/why
> > <https://luau-lang.org/why> - mainly adding linting and
> > type-checking features.
> > 
> > Being compatible with Lua 5.1 is probably a bummer but nonetheless
> > would be interesting to hear about this from this community.

> for sure i need more than an email to give a real answer (and some can
> be read between the lines of the history documents / articles that we
> ship); that said:
>
> the main reasons for choosing lua for luatex has been that it is
> relatively small, has no dependencies, doesn't need tons of libraries
> to make it useful, is made for embeded usage, evolves in a proper
> academic setting, has long-term dedicated (nice) developers, is not
> part of some religious programming language war/competition, reminds
> me (in a positive way of pascal and modula), is pretty fast, ... so,
> no regrest and no need for something else
>
> [...]
>
> to start with typing: i suppose that a more explicit integer / float
> separation can give better performance, although on modern cpu's one
> can wonder (personally i don't see the current mix of integer / float
> as a benefit over all numbers being floats - which had some charm due
> to the consistency - but who an i to complain; at least we now have
> bitwise operators (for a specific application like roblox it makes
> sense but using a dedicated / patched / extended lua machinery in
> luatex doesn't); there are articles by the lua authors about typing
> but these are behind publisher firewalls
>
> [...]
>
> jitting also costs time and for a single pass process like tex there
> is no gain (we established that long ago alreadY); the only reason why
> luajittex is faster than luatex for a tex run is that the vm is faster
> due to some limitations (that one can actually hit but in context we
> got around it) ... jitting also is cpu dependent and therefore more
> fragile futurw wise (more and more complex code too)
>
> [...]
>
> if better performance is an objective: there are probably spots in the
> current code that are candidates and one can see from updates that
> occasionally optimizations happen (keep in mind that lua has to run on
> a wide range of hardware: small embedded to fast single core)
>
> so, I'm pretty okay with lua as it is now (but any influx from spin
> off languages / engines can be interesting, but i think the authors
> are aware of all that) ... and ... who knows what (side effects)
> pallene will bring
>
> (also about performance: i know pretty well how to write fast lua code
> but no one ever comes around asking so that means everyone is
> satisfied)
>
> Hans

After Ramkumar's initial e-mail, I reread some of the manuals describing
Lua 5.1 / 5. 2 / 5.3 / 5.4 / LuaJIT evolution and evaluation.

As I learned more about Lua and LuaTeX I realized that they share (or
contrast in) some core concepts (e.g. TeX's hash table vs Lua tables,
LuaTeX's sparse arrays vs Lua tables, custom allocation strategies -
huge mem array and custom management vs garbage collection).

Recently I discovered some of the things LuaJIT does (and it is in many
ways incredible piece of software).

So the idea of implementing LuaTeX in LuaJIT struck me. The reason why
LuaJIT doesn't bring that many benefits for LuaTeX specifically (apart
from the faster interpeter, is that LuaTeX heavily uses Lua C API (which
is AFAIK slow in LuaJIT since it can't (couldn't?)be JIT compiled and
there is the overhead of the API itself - the stack nature). Wile FFI in
LuaJIT is fast. And I don't necessarilly mean calling external C
functions via FFI, just defining the structures (in C syntax) and using
them transparently in LuaJIT (like "specialized and fast tables").

I wonder how possible and usable would a LuaTeX implementation in pure
(LuaJIT specific) Lua code would be, if it tried to offer the same API
as currently described in the LuaTeX manual (hence being compatible).

I can see benefits in it being pure Lua (although with some optimized
structures), most of the interesting things already happen in Lua and
this would just extend the approachability and openness of the engine.
Also the possibilities of having garbage collection (no need to worry
about memory safety), JIT being viable (since there is no use of C API
to prevent it), less platform dependence (AFAIK in some cases where PUC
Lua uses the platform C's functions LuaJIT provides portable
alternatives).

There are also arguments against LuaJIT:
 - weird 5.1 / 5.2 mix (probably a matter of preference)

 - no integer types (but actually LuaJIT internally optimizes integers
   and also provides the bit library, but the surface is "clean double
   only")

 - requires certain unidiomatic coding style for achieving performance

 - limited platform support (LuaJIT has to specifically be ported to
   each architecture, but the current support is pretty solid)

 - in the past total size of objects was limited to be 2 GiB (this is no
   longer true with GC64 mode)

 - LuaTeX specifically has had problems with LuaJIT's hash function, I
   don't know the details though

 - while the code without C API use would be JITable, the benefits may
   not be large, since TeX runs are not that long, although there are
   critical parts of code that would certainly benefit

Surely I somewhere forgot an important point for or against the
theoretical idea. But Hans, I would still appreciate your general
opinion, even though I remember some document saying that you already
disregarded this possibility.

Kind regards,
Michal Vlasák
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________