Table of Contents for Programming Languages: a survey

Lua

Because it is so well-liked, Lua gets its own chapter.

Pros:

• embeddable
• really small binary (less than 0.2MB on GNU/Linux as of this writing; by comparison, Python is about 3 MB)
• Dynamic typing
• Automatic memory management
• Higher-order functions

Process:

• 'design by small committee' of 3 people, each with power to veto (cite http://www.lua.org/history.html )

Retrospectives and whitepapers:

• Lua - an extensible extension language
• The evolution of Lua
• [http://www.lua.org/doc/jucs05.pdf‎ The implementation of Lua 5.0]
• Preface to 'Programming in Lua' describes Lua's primary goals and features
• more at http://www.lua.org/docs.html
• http://www.lua.org/versions.html
• http://www.lua.org/wshop05/Ierusalimschy2.pdf
• https://cacm.acm.org/magazines/2018/11/232214-a-look-at-the-design-of-lua/fulltext

Influences: SOL, DEL, "Lua 1.0 was designed in such a way that its object constructors, being then slightly different from the current light and flexible style, incorporated the data-description syntax of SOL (hence the name Lua – sol is Portuguese for sun; lua is moon). Lua syntax for control structures was mostly borrowed from Modula (if, while, repeat/until), but also had taken influence from CLU (multiple assignments and multiple returns from function calls, as a simpler alternative to reference parameters or explicit pointers), C++ ("neat idea of allowing a local variable to be declared only where we need it"[3]), SNOBOL and AWK (associative arrays). In an article published in Dr. Dobb's Journal, Lua's creators also state that LISP and Scheme with their single, ubiquitous data structure mechanism (the list) were a major influence on their decision to develop the table as the primary data structure of Lua.[5] Lua semantics have been increasingly influenced by Scheme over time,[3] especially with the introduction of anonymous functions and full lexical scoping." -- https://en.wikipedia.org/wiki/Lua_%28programming_language%29#History

Popularity:

• Beloved by those who use it, but not popular

Tours and tutorials:

• http://learnxinyminutes.com/docs/lua/
• (LuaJIT?) https://cellux.github.io/articles/introduction-to-luajit-part-1/
• interop:
  • Creating Solid APIs with Lua by Tyler Neylon: Chapter 1. Running a Lua Script from C
  • https://www.lua.org/manual/5.1/manual.html#3

Opinions:

• Tero Piirainen of Muut notes that Muut uses Lua a lot (for Redis scripting) but finds "LUA syntax is — with all due respect — ugly and complex..." [1]
• http://blog.datamules.com/blog/2012/01/30/why-lua/
• "Lua became popular mostly based on its implementation, fast, small and self-contained portable C code, and not for the merits of the language itself" [2]
• "Before Lua 5.3 there was no integer type, this makes a lot of things harder than needed in programming, unless you just write high level code." [3]
• "Lua trivially departs from standard syntax like ==, !=, zero-based indexes, without good reasons, creating a general feeling of confusion." [4]
• "The lack by default of bit operations makes the language always in need of a trivial library." [5]
• "Tables as a single data structure are not ideal for many tasks. Especially the dualism between tables that are indexed by sequential numbers and tables acting as maps. It's a lot more kind to your users to give up with the solely academical ideal of "just one data structure", and give programmers at least an ordered and a map data type, or at least come up with something better than that. Lisp attempt at doing most things with lists was much more successful and clean." [6]
  • "...arrays in Lua: They are hashtables, but not quite (in the implementation, every hashtable has a separate array region). They look like hashtables, but not quite (there's syntactic sugar to help dealing with them). You can use hashtable-functions on them, but they don't quite work (which is why there are special functions for when you want to treat a hashtable as an array)." [7]
  • "I have a different view. Lua tables are the foundation for just about any conceivable data structure in CS. While you can do a lot with raw tables in Lua, you can also use them to build all kinds of things on top of them, like b-trees. Or an object system (which is a popular hobby among Lua aficionados)." [8] "Yeah, came here to say the same thing. Tables + metatables mean that pretty much any structure you want is available"
• "The stack-based C API is not comfortable to use" [9]
• "by default you can't type an expression in the REPL and see the value returned" [10]
• "It's a fantastic 'orchestration' language. Coroutines were also invaluable. We used them extensively to script AI behaviors. Our designers(yes, not developers) would write them with "go to X location", yield, "search for players", yield, etc. Was a really compact way to build a FSM that they could understand. " [11]
• "not being zero based is a very severe flaw where embedding into C/C++ apps is a primary use case; it also messes up luajit and terra's UX no end" [12]
• "global by default is another problem, albeit one that can be mitigated with tooling" [13]
• "what other language packs so much punch for so little weight? Despite being quite simple, the syntax still is expressive enough for a lot of DSL uses. Lua has proper tail recursion and an elegant co-routine design and it's slightly idiosyncratic lpeg library is worlds better than the regexp offerings of most languages. It has serious meta-programming power based on a much simpler model than e.g. python or ruby. Despite its warts lua still feels like a better (imperative, statement-based) scheme than scheme to me. What scheme offers a similar combination of speed and expressiveness for the same conceptual and file-size weight as lua(jit)?" [14]
• "I've problems at recognizing any significant part of Lua that is better designed than Ruby, from the POV of the language itself, pretending implementations do not exist. But this is just a personal POV so we could argue forever. I also bet that a very good subset of Ruby could be implemented with a similar-sized code base than Lua (but not at the same speed probably). " [15]
• " The problem I've with the Lua stack API is that you have to make all the mental gym about the current stack layout. And consider that I'm a fan of FORTH and Joy and other stack based languages... But when you have N parameters on the stack, you call a function from Lua, and then you have a different layout based on what such function yields, and so forth, you basically need to comment the state of the stack in order to be able to quickly understand what's going on six months later. An example is the Redis scripting.c code where we call a Lua function to sort the Lua array result." [16]
• "Another problem I've with the stack API is that it's insecure by default. If you push too many things it will result in a C stack overflow. I'm not sure how much performance this gives, I mean, the inability to grow the stack automatically looks like an optimization matter... But at least, panic when we reach the max stack instead of smashing the C stack. " [17]
• "What drove me away from Lua is its philosophy of providing minimal tools in which it is possible, but never easy or convenient, to do work in various programming modalities. Polymorphism, namespaces and packages, arrays, unicode... these things all can be done, but require significant head-standing and in many cases a lot of boilerplate. Even proto-style OO, which you would imagine would be Lua's signature feature, involves lots of rigamarole compared to languages (like Newtonscript) where it is baked in and elegant." [18]
• "Especially the deeply embedded one-based indexing and lack of a separate integer type (chars are strings, ints are floats) make it wearing to type any code that has to parse any binary data. Even bit manipulation requires going through the FFI. Considering that Lua is often touted as the optimal lightweight solution in small embedded systems, these features are must-haves. " [19]
  • "Your complaints are mostly addressed in Lua 5.3 (64 bit integers, utf8 in stdlib, bitwise ops) & many would suggest you use LPeg if you're itching for non trivial string manipulation" [20]
• "The builtin string pattern matching library attempts to mimic regular expressions, but bites the user in the leg every time they attempt to use it for anything remotely regex-y. Unfortunately most usual string manipulation seems to revolve around them and the string.gsub function. The inconvenient string library is also largely irrelevant. The developers call their language eight-bit transparent which in reality means that the programmer needs a separate Unicode library, rendering the whole standard string library mostly useless. One cannot even format UTF-8 text, because the string library doesn't differentiate between characters and bytes." [21]
  • "The patterns are a little underpowered compared to other languages, but it is still quite useful. I suggest looking into LPEG for more complicated operations" [22]
• (good for) "Running untrusted code to allow modding of games" [23]
• "I found that while the Lua core language itself is small, I had to compile in so much extra stuff to make it useful that it wasn't actually small anymore."
  • "I think the idea is that most functionality is meant to be provided by the host program. If you want a standalone scripting language, then Lua probably isn't for you." [24]
• vs. Micropython: (Micropython is) "not a good fit for embedded Linux system programming like Lua is." -- [25]
• "I really wish Lua were popular for small-scale utility scripting instead of Python and JS. Because Lua is way snappier: even with language translation, plenty of Fennel scripts will run under Python's startup time. Likewise, Lua seems like a great fit for small mobile apps and automation (dunno about threading though, but I seem to have read that Lua can do threads with a library)." [26]
• "Lua will never be more popular without a proper standard library. Leaving everything up to the user so that you have to manually implement (or hunt the web for) even the most basic features is mega annoying. How many millions of people need to rewrite string.startswith?" [27]
  • "The library thing is eh: just grab Penlight if you aren't running Lua on a microchip." [28]
• "Metatables are cool and all, but holy crap are they confusing for Lua newbies. Having an integrated class system would go a long way." [29]
• "Lua is way more common for game scripting than Python. One of the few major games which used Python I can think of was Civ IV, and Civ V switched to Lua. Lua is smaller, easier to embed interface / interact with (e.g. it's way less ceremony to expose a native function to Lua than to Python), and tends to be faster, especially when jumping back and forth between the engine and the scripting. Embedding / scripting (within a larger program) is the original use-case of Lua, not so Python. I believe it's also much easier to secure / sandbox (remove bits you don't want script writers to have access to) as well, the stdlib is smaller and I think it has less interactions between modules. Python embedding / scripting tends to be more common for software where the securing / restriction aspect is smaller but flexibility & larger embeds are necessary e.g. 3D, CAD and other "production pipeline" software tends to be scripted with Python." [30]
  • ""> I believe [Lua is] also much easier to secure / sandbox [than Python]" Yes. C code that embeds the Lua VM has total control over which functions are exposed to the Lua code. It's possible to create a VM that, for example, has no access to the filesystem. AFAIK it's not possible to do that if you embed Python. Dangerous functions like `open` are always available and sandboxing facilities have to try and prevent untrusted code from getting a reference to them. Unfortunately, there are numerous ways to work around these restrictions and obtain access to dangerous functions, e.g. via `__builtins__`. Another reason running untrusted Lua code is considered fairly safe is that the VM is well-engineered and has a history of very few bugs [31]. However, the latest 5.4.0 release seems to have more bugs than older releases. It might still possible for untrusted Lua code to use 100% CPU and hang a program, or to read data from the embedding process using a Spectre-style attack (although even that's unlikely because the Lua VM is an interpreter rather than a JIT compiler). However, it's quite possible to secure an embedded Lua VM to prevent it from doing things like accessing arbitrary files." [32]
• "I wish Lua had immutable data structures with memory sharing. It would make the designs of some things much more performant." [33]
• https://andregarzia.com/2021/01/lua-a-misunderstood-language.html
• "Walmart uses embedded lua (in a Go proxy server) to handle all of their L7 routing and to generate full responses and segmentation for high throughput analytics servers (their equiv to Google Analytics). This logic can run all the way at their edge CDN. It’s been a really powerful tool for them and it was relatively easy to embed in a multi-tenant proxy server." [34]
• http://notebook.kulchenko.com/programming/lua-good-different-bad-and-ugly-parts

Gotchas:

• "My primary woe with Lua is the absence of ‘null’. You can't preserve structure of deserialized tables if there are nil values in them, so you can't really build API middleware and such, without extra effort for that (so you'd also not be able to use functional libraries like Penlight in your middleware)." [35]
  • "Lua doesn't distinguish between, "the key is not in the table," and, "there is a value for that key, and the value is nil." Setting the value for the key to nil simply erases it from the table." [36]
• "The whole "sequence iteration stops at the first internal hole and therefore the # operator doesn't do what you think it should" gotcha is extremely user hostile." [37]
  • "Ah, yes, the table length/ iteration thing probably deserves the second place or a tie in the annoyances." [38]
• "There is one rough edge I've encountered, which is down to a serious wart in the standard library: `table.insert(tab, value)` inserts the value at the end of the array portion, which in Lua we express as `#tab + 1`. But `table.insert(tab, index, value)` is what's used to insert into another slot in the array portion. This is a serious mistake which only prevails for historical reasons. I beg of you: never shift the meaning of parameters around when you add more of them. Nothing good will ever come of this!" [39]
• "I was just talking to a friend of mine who is a game dev and used to use Lua a lot when he worked in cryengine. He complained about global variables in Lua and how Lua can return null values when a value isn’t present in a collection" [40]
• (when implementing a Lua parser) "Lua’s grammar is also pretty carefully designed to not put you into weird ambiguous situations, with only one exception I can think of. (The ambiguity of a colon-based method call in certain contexts.)" -- [41]

Types: "There are eight basic types in Lua: nil, boolean, number, string, userdata, function, thread, and table." -- http://www.lua.org/pil/2.html

Libraries:

• https://love2d.org/

Internals

Bytecode

Lua uses a register machine VM with bytecode.

Links:

• http://luaforge.net/docman/83/98/ANoFrillsIntroToLua51VMInstructions.pdf
• http://files.catwell.info/misc/mirror/lua-5.2-bytecode-vm-dirk-laurie/lua52vm.html
• Interesting things about the Lua interpreter

Local variable storage and operand stack

Lua uses the common system of stack frames, where each stack frame contains local variables[42][43]. Lua bytecode has instructions which directly access local variables (in a manner similar to registers). If closures go out of scope while they are still references by other functions, the relevant variables are moved to the heap at that time[44].

Garbage collection (GC)

• http://lua-users.org/wiki/GarbageCollection
• http://lua-users.org/wiki/GarbageCollectionTutorial
• https://poga.github.io/lua53-notes/gc.html
• Garbage collection in Lua
• https://devforum.roblox.com/t/a-beginners-guide-to-lua-garbage-collection/1756677
• Understanding Lua’s Garbage Collection: Towards a Formalized Static Analyzer

Alternate implementations and variants

• LuaJIT?
  • https://github.com/franko/luajit-lang-toolkit
  • garbage collection (GC): http://wiki.luajit.org/New-Garbage-Collector
  • FFI
    • http://luajit.org/ext_ffi_semantics.html
    • http://luajit.org/ext_ffi_tutorial.html
  • https://0xbigshaq.github.io/2022/08/22/lua-jit-intro/
  • Format for 32 bit GC references: https://github.com/LuaJIT/LuaJIT/blob/v2.1/src/lj_obj.h#L224
• embedded:
  • eLua todo cp info from proj-oot-lowEndTargets-embeddableLanguageImplementations
    • https://nodemcu.readthedocs.io/en/master/ ("NodeMCU? Lua is based on eLua" -- [45])
• Luau variant: https://luau-lang.org/ https://luau-lang.org/2021/11/03/luau-goes-open-source.html
  • https://lobste.rs/s/peamn8/roblox_s_luau_goes_open_source
  • https://news.ycombinator.com/item?id=29097692
• https://fengari.io/
• lujlu: LuaJit? VM in Lua
• LJR
  • https://sillycross.github.io/2022/11/22/2022-11-22/
    • https://news.ycombinator.com/item?id=33711583
  • https://github.com/luajit-remake/luajit-remake
  • https://lobste.rs/s/c02xce/building_fastest_lua_interpreter
  • https://hackaday.com/2022/11/26/interpreting-the-lua-interpreter/

Tools and libraries

• https://github.com/facebookarchive/luaffifb
  • http://luajit.org/ext_ffi.html

Variants

• http://moonscript.org/
• http://the-ravi-programming-language.readthedocs.io/en/latest/ravi-overview.html
• Objective Lua "is an almost pure superset of Lua that welds the Objective C object orientation system and syntax on top of the classic Lua language".

Related

• http://terralang.org/
  • https://github.com/terralang/terra
    • "Terra is a new low-level system programming language that is designed to interoperate seamlessly with the Lua programming language. It is also backwards compatible with (and embeddable in) existing C code. Like C, Terra is a monomorphic, statically-typed, compiled language with manual memory management. But unlike C, it is designed to make interaction with Lua easy. Terra code shares Lua's syntax and control-flow constructs. It is easy to call Lua functions from Terra (or Terra functions from Lua). Furthermore, you can use Lua to meta-program Terra code. The Lua meta-program handles details like conditional compilation, namespaces, and templating in Terra code that are normally special constructs in low-level languages. This coupling additionally enables more powerful features like function specialization, lisp-style macros, and manually controlled JIT compilation. Since Terra's compiler is also available at runtime, it makes it easy for libraries or embedded languages to generate low-level code dynamically."
• https://github.com/titan-lang/titan
  • "Titan is a new programming language, designed to be a statically-typed, ahead-of-time compiled sister language to Lua. It is an application programming language with a focus on performance."