Hans / Michal, Fascinating exchange of thoughts. Thank you so much for sharing. On the back of my mind, probably a bittersweet moment by Roblox - having built its foundations with Lua and deciding to fork / rewrite it... best regards, Ramkumar On Sun, Nov 7, 2021 at 2:39 AM Hans Hagen via ntg-context < ntg-context@ntg.nl> wrote: > On 11/6/2021 5:19 PM, Michal Vlasák wrote: > > On Fri Nov 5, 2021 at 9:14 PM CET, Hans Hagen via ntg-context wrote: > >> On 11/5/2021 6:23 PM, Ramkumar KB via ntg-context wrote: > >>> Hello, > >>> > >>> Yesterday, Roblox announced - Luau - https://luau-lang.org/why > >>> - mainly adding linting and > >>> type-checking features. > >>> > >>> Being compatible with Lua 5.1 is probably a bummer but nonetheless > >>> would be interesting to hear about this from this community. > > > >> for sure i need more than an email to give a real answer (and some can > >> be read between the lines of the history documents / articles that we > >> ship); that said: > >> > >> the main reasons for choosing lua for luatex has been that it is > >> relatively small, has no dependencies, doesn't need tons of libraries > >> to make it useful, is made for embeded usage, evolves in a proper > >> academic setting, has long-term dedicated (nice) developers, is not > >> part of some religious programming language war/competition, reminds > >> me (in a positive way of pascal and modula), is pretty fast, ... so, > >> no regrest and no need for something else > >> > >> [...] > >> > >> to start with typing: i suppose that a more explicit integer / float > >> separation can give better performance, although on modern cpu's one > >> can wonder (personally i don't see the current mix of integer / float > >> as a benefit over all numbers being floats - which had some charm due > >> to the consistency - but who an i to complain; at least we now have > >> bitwise operators (for a specific application like roblox it makes > >> sense but using a dedicated / patched / extended lua machinery in > >> luatex doesn't); there are articles by the lua authors about typing > >> but these are behind publisher firewalls > >> > >> [...] > >> > >> jitting also costs time and for a single pass process like tex there > >> is no gain (we established that long ago alreadY); the only reason why > >> luajittex is faster than luatex for a tex run is that the vm is faster > >> due to some limitations (that one can actually hit but in context we > >> got around it) ... jitting also is cpu dependent and therefore more > >> fragile futurw wise (more and more complex code too) > >> > >> [...] > >> > >> if better performance is an objective: there are probably spots in the > >> current code that are candidates and one can see from updates that > >> occasionally optimizations happen (keep in mind that lua has to run on > >> a wide range of hardware: small embedded to fast single core) > >> > >> so, I'm pretty okay with lua as it is now (but any influx from spin > >> off languages / engines can be interesting, but i think the authors > >> are aware of all that) ... and ... who knows what (side effects) > >> pallene will bring > >> > >> (also about performance: i know pretty well how to write fast lua code > >> but no one ever comes around asking so that means everyone is > >> satisfied) > >> > >> Hans > > > > After Ramkumar's initial e-mail, I reread some of the manuals describing > > Lua 5.1 / 5. 2 / 5.3 / 5.4 / LuaJIT evolution and evaluation. > > > > As I learned more about Lua and LuaTeX I realized that they share (or > > contrast in) some core concepts (e.g. TeX's hash table vs Lua tables, > > LuaTeX's sparse arrays vs Lua tables, custom allocation strategies - > > huge mem array and custom management vs garbage collection). > > > > Recently I discovered some of the things LuaJIT does (and it is in many > > ways incredible piece of software). > > > > So the idea of implementing LuaTeX in LuaJIT struck me. The reason why > > LuaJIT doesn't bring that many benefits for LuaTeX specifically (apart > > from the faster interpeter, is that LuaTeX heavily uses Lua C API (which > > is AFAIK slow in LuaJIT since it can't (couldn't?)be JIT compiled and > > there is the overhead of the API itself - the stack nature). Wile FFI in > > LuaJIT is fast. And I don't necessarilly mean calling external C > > functions via FFI, just defining the structures (in C syntax) and using > > them transparently in LuaJIT (like "specialized and fast tables"). > > sure, it's a nice piece of software > > > I wonder how possible and usable would a LuaTeX implementation in pure > > (LuaJIT specific) Lua code would be, if it tried to offer the same API > > as currently described in the LuaTeX manual (hence being compatible). > > personally i would never gamble on that, just as i never gambled on > luajittex for the long run ... it's nice to have around as long as it > works but just as the ffilib used in luatex is 'abandoned by where it > came from', luajit is also stuck in 5.2 and there was even a moment that > it looked like being quit ... for me, anything that depends on low level > cpu specific code at compile time is not worth the effort (it's also why > i discarded libs in luametatex that have cpu specific optimizations that > then in turn demand running scripts to generate files for compilation > ... all not worth the trouble) > > now, keep in mind that (lua)jit involves some runtime analysis and that > is why we don't gain (we did quite some experiments at that time) > > also, often the runtime is not where one thinks (many jit performance > tests are doing math and we hardly do that); also in luatex we had to > patch the string hasher that came with luahit because it was tuned for > urls (think of skipping the first n chars i.e. http://) while for tex > that actually works out bad > > for the record: i have rewritten bits of core tex in lua but in the end > didn't see many benefits (i'll probably pick up that thread when > luametatex is kind of finished) .. i need to update it -) > > > I can see benefits in it being pure Lua (although with some optimized > > structures), most of the interesting things already happen in Lua and > > this would just extend the approachability and openness of the engine. > > as i mentioned, i will pick up that thread but still it will be via > callbacks ... much in tex is about macro handling (efficient in c) and > par building (efficient in c) ... crossing the c boundary is not that > costly in the end (one can sort of measure that) ... it's the cpu cache > (hits and misses) that matter more as lua and tex are jumping around > memory (not crunching numbers or massaging vectors) ... some gain will > come from the upcoming GB level 2 caches for cpu's (i guess) > > > Also the possibilities of having garbage collection (no need to worry > > about memory safety), JIT being viable (since there is no use of C API > > to prevent it), less platform dependence (AFAIK in some cases where PUC > > Lua uses the platform C's functions LuaJIT provides portable > > alternatives). > > jit is cpu dependent > > > There are also arguments against LuaJIT: > > - weird 5.1 / 5.2 mix (probably a matter of preference) > > and politics (one can find not so nice discussions on the web, one > reason for not using luajit btw) > > > - no integer types (but actually LuaJIT internally optimizes integers > > and also provides the bit library, but the surface is "clean double > > only") > > > > - requires certain unidiomatic coding style for achieving performance > > indeed, and this while ffi mix is kind of not-so-lua > > i get the feeling that it also stimulates inefficient code writing .. > the 'jit will do it for me' thing > > > - limited platform support (LuaJIT has to specifically be ported to > > each architecture, but the current support is pretty solid) > > > > - in the past total size of objects was limited to be 2 GiB (this is no > > longer true with GC64 mode) > > also some limitations on what goes on the stack > > > - LuaTeX specifically has had problems with LuaJIT's hash function, I > > don't know the details though > > > > - while the code without C API use would be JITable, the benefits may > > not be large, since TeX runs are not that long, although there are > > critical parts of code that would certainly benefit > > an alternative could be the compilation of (stable) lua code into > modules (the lua folks have articles about that, and it seems to > outperform luajit) > > > Surely I somewhere forgot an important point for or against the > > theoretical idea. But Hans, I would still appreciate your general > > opinion, even though I remember some document saying that you already > > disregarded this possibility. > > one important point is that 'tex is also something knuthian' ... > software legacy ... nice to keep it around, running and up to date > > we can pick up that thread in a year or so (or at a meeting) > > Hans > > ps. about performance: it's not the core engine that is the bottleneck, > it's often suboptimal user styling and macro writing ... i'm often > really surprised how fast lua is, and when it's taking time it's often > for a reason (and yes, i spend a lot of time on benchmarking and > considering optimizations without messing up code, kind of fun thing) > > ----------------------------------------------------------------- > Hans Hagen | PRAGMA ADE > Ridderstraat 27 | 8061 GH Hasselt | The Netherlands > tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl > ----------------------------------------------------------------- > > ___________________________________________________________________________________ > If your question is of interest to others as well, please add an entry to > the Wiki! > > maillist : ntg-context@ntg.nl / > http://www.ntg.nl/mailman/listinfo/ntg-context > webpage : http://www.pragma-ade.nl / http://context.aanhet.net > archive : https://bitbucket.org/phg/context-mirror/commits/ > wiki : http://contextgarden.net > > ___________________________________________________________________________________ >