Re: Any initial thoughts on luau?

[Ramkumar KB via ntg-context <ntg-context@ntg.nl>]

[-- Attachment #1.1: Type: text/plain, Size: 10452 bytes --]

Hans / Michal,

Fascinating exchange of thoughts. Thank you so much for sharing.

On the back of my mind, probably a bittersweet moment by Roblox -
having built its foundations with Lua and deciding to fork / rewrite it...

best regards,
Ramkumar

On Sun, Nov 7, 2021 at 2:39 AM Hans Hagen via ntg-context <
ntg-context@ntg.nl> wrote:

> On 11/6/2021 5:19 PM, Michal Vlasák wrote:
> > On Fri Nov 5, 2021 at 9:14 PM CET, Hans Hagen via ntg-context wrote:
> >> On 11/5/2021 6:23 PM, Ramkumar KB via ntg-context wrote:
> >>> Hello,
> >>>
> >>> Yesterday, Roblox announced - Luau - https://luau-lang.org/why
> >>> <https://luau-lang.org/why> - mainly adding linting and
> >>> type-checking features.
> >>>
> >>> Being compatible with Lua 5.1 is probably a bummer but nonetheless
> >>> would be interesting to hear about this from this community.
> >
> >> for sure i need more than an email to give a real answer (and some can
> >> be read between the lines of the history documents / articles that we
> >> ship); that said:
> >>
> >> the main reasons for choosing lua for luatex has been that it is
> >> relatively small, has no dependencies, doesn't need tons of libraries
> >> to make it useful, is made for embeded usage, evolves in a proper
> >> academic setting, has long-term dedicated (nice) developers, is not
> >> part of some religious programming language war/competition, reminds
> >> me (in a positive way of pascal and modula), is pretty fast, ... so,
> >> no regrest and no need for something else
> >>
> >> [...]
> >>
> >> to start with typing: i suppose that a more explicit integer / float
> >> separation can give better performance, although on modern cpu's one
> >> can wonder (personally i don't see the current mix of integer / float
> >> as a benefit over all numbers being floats - which had some charm due
> >> to the consistency - but who an i to complain; at least we now have
> >> bitwise operators (for a specific application like roblox it makes
> >> sense but using a dedicated / patched / extended lua machinery in
> >> luatex doesn't); there are articles by the lua authors about typing
> >> but these are behind publisher firewalls
> >>
> >> [...]
> >>
> >> jitting also costs time and for a single pass process like tex there
> >> is no gain (we established that long ago alreadY); the only reason why
> >> luajittex is faster than luatex for a tex run is that the vm is faster
> >> due to some limitations (that one can actually hit but in context we
> >> got around it) ... jitting also is cpu dependent and therefore more
> >> fragile futurw wise (more and more complex code too)
> >>
> >> [...]
> >>
> >> if better performance is an objective: there are probably spots in the
> >> current code that are candidates and one can see from updates that
> >> occasionally optimizations happen (keep in mind that lua has to run on
> >> a wide range of hardware: small embedded to fast single core)
> >>
> >> so, I'm pretty okay with lua as it is now (but any influx from spin
> >> off languages / engines can be interesting, but i think the authors
> >> are aware of all that) ... and ... who knows what (side effects)
> >> pallene will bring
> >>
> >> (also about performance: i know pretty well how to write fast lua code
> >> but no one ever comes around asking so that means everyone is
> >> satisfied)
> >>
> >> Hans
> >
> > After Ramkumar's initial e-mail, I reread some of the manuals describing
> > Lua 5.1 / 5. 2 / 5.3 / 5.4 / LuaJIT evolution and evaluation.
> >
> > As I learned more about Lua and LuaTeX I realized that they share (or
> > contrast in) some core concepts (e.g. TeX's hash table vs Lua tables,
> > LuaTeX's sparse arrays vs Lua tables, custom allocation strategies -
> > huge mem array and custom management vs garbage collection).
> >
> > Recently I discovered some of the things LuaJIT does (and it is in many
> > ways incredible piece of software).
> >
> > So the idea of implementing LuaTeX in LuaJIT struck me. The reason why
> > LuaJIT doesn't bring that many benefits for LuaTeX specifically (apart
> > from the faster interpeter, is that LuaTeX heavily uses Lua C API (which
> > is AFAIK slow in LuaJIT since it can't (couldn't?)be JIT compiled and
> > there is the overhead of the API itself - the stack nature). Wile FFI in
> > LuaJIT is fast. And I don't necessarilly mean calling external C
> > functions via FFI, just defining the structures (in C syntax) and using
> > them transparently in LuaJIT (like "specialized and fast tables").
>
> sure, it's a nice piece of software
>
> > I wonder how possible and usable would a LuaTeX implementation in pure
> > (LuaJIT specific) Lua code would be, if it tried to offer the same API
> > as currently described in the LuaTeX manual (hence being compatible).
>
> personally i would never gamble on that, just as i never gambled on
> luajittex for the long run ... it's nice to have around as long as it
> works but just as the ffilib used in luatex is 'abandoned by where it
> came from', luajit is also stuck in 5.2 and there was even a moment that
> it looked like being quit ... for me, anything that depends on low level
> cpu specific code at compile time is not worth the effort (it's also why
> i discarded libs in luametatex that have cpu specific optimizations that
> then in turn demand running scripts to generate files for compilation
> ... all not worth the trouble)
>
> now, keep in mind that (lua)jit involves some runtime analysis and that
> is why we don't gain (we did quite some experiments at that time)
>
> also, often the runtime is not where one thinks (many jit performance
> tests are doing math and we hardly do that); also in luatex we had to
> patch the string hasher that came with luahit because it was tuned for
> urls (think of skipping the first n chars i.e. http://) while for tex
> that actually works out bad
>
> for the record: i have rewritten bits of core tex in lua but in the end
> didn't see many benefits (i'll probably pick up that thread when
> luametatex is kind of finished) .. i need to update it -)
>
> > I can see benefits in it being pure Lua (although with some optimized
> > structures), most of the interesting things already happen in Lua and
> > this would just extend the approachability and openness of the engine.
>
> as i mentioned, i will pick up that thread but still it will be via
> callbacks ... much in tex is about macro handling (efficient in c) and
> par building (efficient in c) ... crossing the c boundary is not that
> costly in the end (one can sort of measure that) ... it's the cpu cache
> (hits and misses) that matter more as lua and tex are jumping around
> memory (not crunching numbers or massaging vectors) ... some gain will
> come from the upcoming GB level 2 caches for cpu's (i guess)
>
> > Also the possibilities of having garbage collection (no need to worry
> > about memory safety), JIT being viable (since there is no use of C API
> > to prevent it), less platform dependence (AFAIK in some cases where PUC
> > Lua uses the platform C's functions LuaJIT provides portable
> > alternatives).
>
> jit is cpu dependent
>
> > There are also arguments against LuaJIT:
> >   - weird 5.1 / 5.2 mix (probably a matter of preference)
>
> and politics (one can find not so nice discussions on the web, one
> reason for not using luajit btw)
>
> >   - no integer types (but actually LuaJIT internally optimizes integers
> >     and also provides the bit library, but the surface is "clean double
> >     only")
> >
> >   - requires certain unidiomatic coding style for achieving performance
>
> indeed, and this while ffi mix is kind of not-so-lua
>
> i get the feeling that it also stimulates inefficient code writing ..
> the 'jit will do it for me' thing
>
> >   - limited platform support (LuaJIT has to specifically be ported to
> >     each architecture, but the current support is pretty solid)
> >
> >   - in the past total size of objects was limited to be 2 GiB (this is no
> >     longer true with GC64 mode)
>
> also some limitations on what goes on the stack
>
> >   - LuaTeX specifically has had problems with LuaJIT's hash function, I
> >     don't know the details though
> >
> >   - while the code without C API use would be JITable, the benefits may
> >     not be large, since TeX runs are not that long, although there are
> >     critical parts of code that would certainly benefit
>
> an alternative could be the compilation of (stable) lua code into
> modules (the lua folks have articles about that, and it seems to
> outperform luajit)
>
> > Surely I somewhere forgot an important point for or against the
> > theoretical idea. But Hans, I would still appreciate your general
> > opinion, even though I remember some document saying that you already
> > disregarded this possibility.
>
> one important point is that 'tex is also something knuthian' ...
> software legacy ... nice to keep it around, running and up to date
>
> we can pick up that thread in a year or so (or at a meeting)
>
> Hans
>
> ps. about performance: it's not the core engine that is the bottleneck,
> it's often suboptimal user styling and macro writing ... i'm often
> really surprised how fast lua is, and when it's taking time it's often
> for a reason (and yes, i spend a lot of time on benchmarking and
> considering optimizations without messing up code, kind of fun thing)
>
> -----------------------------------------------------------------
>                                            Hans Hagen | PRAGMA ADE
>                Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>         tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
> -----------------------------------------------------------------
>
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to
> the Wiki!
>
> maillist : ntg-context@ntg.nl /
> http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : http://contextgarden.net
>
> ___________________________________________________________________________________
>

[-- Attachment #1.2: Type: text/html, Size: 12782 bytes --]

[-- Attachment #2: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________