notes-computer-programming-programmingLanguageDesign-prosAndCons-lua

design notes

http://lambda-the-ultimate.org/node/3894

misc

http://lua-users.org/wiki/ResumableVmPatch

" Anyway, Lua is another great example of this curse because there's no module system or standard library to speak of, everyone rolls not only their own object system but their own list functions. The reasons are different though; Lua's design goals are to be a tiny language for embedding, so they sacrifice completeness for being really easy to customize. "

silentbicycle 748 days ago

Yeah. The people who are using Lua primarily in an embedded context (and thus have a lot of dependencies, such as threading & graphic libs, already chosen for them) have VERY different priorities than the people using Lua standalone. It pulls the community in two different (and occasionally opposed) directions, and the Lua team doesn't want to alienate either camp.

(On the balance, though, it's still one of my favorites.)

randrews 748 days ago

Yes, definitely. The problem I'm running into is that the people writing the module systems are thinking in terms of running it standalone.

I want a way to get a bunch of Lua libraries and cram them into one thing I can easily embed and make available to my embedded Lua interpreter. I can't really visualize how that would work though, so I can't write it yet.

silentbicycle 748 days ago

You can compile a set of (Lua) libraries with luac, then insert the bytecode as a string literal in a C program. loadstring works on bytecode, too. With C libraries, link as usual.

There are utilities floating around to reformat the luac.out file (compiler output), but all you really need to do is put

    static const char *lua_code = "..."; /* with the body escaped */

If you don't end a line with ;, it will be concatenated to a subsequent string literal at compile time. Oh, and multiple lua files can be compiled into one luac.out.

Does that help? If you have further questions, contact info is in my profile.

randrews 748 days ago

That's the core of what I was going to do, but I want to put on top of it something like Bundler, where I can specify which libraries I want (and which ones depend on other ones) and it will build it for me.

Although I think the only one I want is Penlight, so that is probably overkill. :)

--

"

assycron • 4 days ago

Lua also has continuations and lexical scope :D 3 1 • Reply • Share ›

    Avatar
    munificent Mod assycron • 4 days ago

    I thought Lua only had coroutines, not continuations.
    •
    Reply
    •
    Share ›
        Avatar
        assycron munificent • 4 days ago
            −

        A small patch to add coroutine.clone() makes them equivalent to the continuations you would expect in scheme.

        http://lua-users.org/lists/lua...
        https://github.com/torus/lua-c..."

---

http://lua-users.org/lists/lua-l/2007-11/msg00248.html

--

http://www.freelists.org/post/luajit/Using-luajit-runtime-for-nonlua-languages,25

"

At 10:12 PM +0000 2/11/13, David Given wrote:

    I'm actually rather interested in using this technique myself... one
    thing I want to know, however is when doing lazy compilation of basic
    blocks, are there any techniques necessary to fixup calls after compilation?

David ,

For my purposes, I don't actually track basic blocks explicitly, but do have a lazy multiple-step binary interpretation process that's outlined in the last slide of the presentation. Since some RISC instructions are likely to get generated multiple times in a program, the Lua-code generated for each distinct 32-bit instruction is stored in a cache table of translated-to-Lua but not linked or located instructions. So this Lua-code table is indexed by the binary instruction code.

When a simulated program tries to execute an instruction at a particular program address, it tries to call a completely linked and located closure (for that instruction) by using the PC as an index into the "located instruction" table. If that entry is empty, it will trigger a lazy linkage operation where the link routine locates the chain of sequential instructions in that PC-indexed table and fixes up all the sequential "next instruction" links in the basic block. To make this work simply requires a couple of levels of closures in the translated per-instruction generated Lua code. The outer (first) closure binds all of the instruction operands (e.g., constant literals and register references), and the second binds the branch and next-sequential-instruction linkage information. So most of my generated code is "half-baked" until the simulated program branches to it and causes a lazy linkage :-) The basic block is "discovered" by the runtime branch-target fetch from the PC-indexed located-instruction table. The link-step loop just runs sequentially from the branch-target entry point of that metatable-supported fetch (of an empty cache entry) until it hits a branch instruction which terminates the basic block. My code fragments (in the slides) return a next-PC absolute address from the branch, so that branch instruction has to be located too, but you could alternatively have a PC-relative branch that would be location/position-independent. (If I recall correctly, that was slower in the simulator implementation; the actual RISC branch instructions are mostly encoded as PC-relative.)

I think the Lua metatable support for lazy access makes for a very clean implementation where the basic-block optimization just falls out of the normal simulator execution cycle, but as I mentioned in a previous post, I did find it speeded things up to explicitly check for a repeated loop target in the innermost simulator execution loop. Usually the other lazy table references that induce the Lua-code generation are one-time operations (unless you have self-modifying code that needs to invalidate cache entries) with insignificant cache metatable overhead.

Thanks for the insightful question, it was a point that caused me some head-scratching :-)

Craig

"

--

upvote

haberman 1 day ago

I addressed this in my last paragraph. Lua is a nice language (and I use it myself) but it has downsides too. Noticeably it is a moving target that breaks backward compatibility. And I'm not sure what the state of the art is in terms of running Lua in the browser.

reply

upvote

justincormack 1 day ago

There are a fair number of Lua in browser projects with a variety of techniques. Often game porting uses them.

reply

upvote

catch23 1 day ago

having written a fair amount of lua code, there aren't many libraries out there. Their npm equivalent is called 'luarocks'. To give you an idea of how old the libraries are: many of them mention the perl package it was translated from.

reply

upvote

TheZenPsycho? 1 day ago

The reason for that of course is that it's so trivial to embed it in a C program, that if you want a library you can just expose a C library in lua. luajit even makes this absurdly easy with its FFI api.

reply

upvote

catch23 1 day ago

Right, but libraries consist of many other things, including ORM or testing suites. There aren't many great ORM libraries written in C, and you need the test suite to work in lua because you're testing lua code. In general, the world of lua is pretty small, probably smaller than languages like Clojure or OCaml.

reply

-- " One thing Mike didn’t highlight: get a simpler language. Lua is much simpler than JS. This means you can make a simple interpreter that runs fast enough to be balanced with respect to the trace-JITted code [unlike with JS].

and a little further down:

    On the differences between JS and Lua, you can say it’s all a matter of proper design and engineering (what isn’t?), but intrinsic complexity differences in degree still cost. You can push the hard cases off the hot paths, certainly, but they take their toll. JS has more and harder hard cases than Lua. One example: Lua (without explicit metatable usage) has nothing like JS’s prototype object chain." -- http://lambda-the-ultimate.org/node/3851#comment-57671 (saying that it's easier to optimize Lua than javascript)

--

sitkack 12 hours ago

I don't know why people are leaving Python for Go when they could be using PyPy? or Shedskin. There are plenty of ways to be fast using Python.

reply

jerf 5 hours ago

It's easy to lose track of this in the torrent of "PyPy? Benchmarks Show It Sped Up 2x!!!!" and "Latest Javascript Engine 50% Faster Than The Last One!!!! OMG!! Node!!!!", but in absolute terms, the dynamic language JITs are still quite slow. It's an urban legend that they are anywhere near compiled performance. Except LuaJIT?. "Faster Python" is still slow. Unless you're in a tight loop adding numbers together, but I'm not.

Moreover, a plot of their performance over time rather strongly suggests that they've plateaued. Personally, barring any major JIT advances, I'm considering the book closed on the topic of whether JITs can take a dynamic language and make it as fast as C or C++: No.

(Recall that a Java JIT is another issue; it starts with the static typing of Java, and then JITs from there. It has a much larger array of much more powerful guarantees to work with, that the dynamic languages simply don't. Go will be running at straight-up C speeds (which it does not currently) long before Python will.)

reply

"For that, Lua has a safe environment, automatic memory management, and great facility to handle strings and other kinds of data with dynamic size. " -- http://www.lua.org/pil/p1.html

glue language, Extensibility, Simplicity, Efficiency, Portability -- http://www.lua.org/pil/p1.html

Dynamic typing to allow polymorphism, Automatic memory management to simplify interfaces. Higher-order functions and anonymous functions to allow a high degree of parametrization. -- http://www.lua.org/pil/p1.html

"

Lua & Java

What I liked about writing the Lua version was that there was so little overhead in concepts. You’ve got functions, some control structures, tables and that’s basically it. At the same time there were some annoying things. A common complaint about Lua is its relatively high verbosity - at least for a dynamic language. It’s true. Seeing long keywords like function or local everywhere hinders readability. Also, the order of definitions is important. You are not free to e.g. rearrange function definitions in source files.

In contrast to the Lua version method definitions in Java can be rearranged freely so you can put the more important stuff at the top of the source file. Just as Lua Java has the reputation of being overly verbose. In fact, Java is much more verbose than Lua so I shouldn’t really call Lua verbose in this context. On the plus side, using classes (and thereby types) to encapsulate code made it easier to understand the source code’s structure in this case despite the cognitive overhead of the class-based approach compared to the simpler “function-approach” of the Lua version.

One shouldn’t underestimate the help of a static type system when refactoring code, too. In Java it was easy because the compiler complained if parts of the code didn’t type check so you were kind of5 sure that you did not break anything. Trying to refactor the Lua version almost always lead to subtle bugs that a compiler could have found and so could have saved me a lot of time. " -- http://web.archive.org/web/20120402183419/http://www.eugenkiss.com/blog/2011-05-experience-report-a-mastermind-simulation-in-lua-java-and-haskell.html

"

State related Problems

As I already mentioned, the lion’s share of problems/bugs were state/identity related in both the Lua & Java version. Other problems were logical errors or small slips that could be fixed immediately. If your reasoning is wrong then the programming language can’t help you since it can’t guess what you really want to do. The situation is different with state/identity related bugs so I want to concentrate on these.

I pointed out the slight annoyance of not being able to freely rearrange the order of definitions in a Lua source file6. There is another reason why I don’t like it in general: You must specify an order whether you care about it or not. So when refactoring code and shuffling it around I sometimes found myself changing the order in such a way that some variables were not reasonably initialized which lead to strange behaviour but not error messages.

Even if Lua has only one mutable type - the table - I often found myself subtly overseeing this fact and thus creating weird side effects. I’m going to provide one concrete example:

--... local function _createAllCodes(codes, code, length) if length == 0 then codes[#codes+1] = {unpack(code)} else for i=0,7 do code[length] = T[i] _createAllCodes(codes, code, length-1) end end end

--- Create a list of all possible codes with a length of 'length'. local function createAllCodes(length) local codes = {} _createAllCodes(codes, {}, length) return codes end --...

The third line used to be codes[#codes+1] = code instead of codes[#codes+1] = {unpack(code)}. The former version reused references whereas the new version created new references. When I used the first version I couldn’t understand why the program was not working correctly until I, through a lot of tests and print statements, narrowed down the problem. There were a lot of problems like this one.

Much the same applies to the Java version. Here is an example:

... public Code(T... t) { this.code = t.clone(); } ...

The line this.code = t.clone(); used to be this.code = t; and, like the Lua example, lead to weird behaviour.

It says a lot about such state/identity related problems when neither I nor six additional eyes could find them. As I mentioned, the initial version7 was created in, I guess you could say, “pair programming style”. I agreed with a fellow student on the general design of the code and he sat next to me when I coded. I said out loud every function I wanted to create and how I am going to write it. He critically verified that I really did do what I said and if it made sense at all to do it this way. Still, it turned out that there were bugs8. When I explained to the others that there must be a bug in the code we went on a long bug search. Eventually, one of them told me that he found it. I was interested why his fix worked. He couldn’t explain it. He just changed some lines and the code seemed to work. I, too, couldn’t understand why this particular fix worked. I repeat: It says a lot about such state/identity related problems when you can’t often explain why a “fix” fixes the code.

Even if you pretend that you are aware of using mutable objects there will be one short moment of unconzentration and a subtle state related bug will sneak in. And boy, are they hard to find! On top of that, they have very weird and often not reproducable effects as I described earlier. I know someone who, I think, would argue that it is the programmer’s fault, that the programmer’s inability leads to this kind of bug and it is entirely not the language’s blame - the language is good enough. Well, I don’t share this opinion.

Now to a selection of some anecdotes that are only tangentially related to the simulation but all have something to do with state related bugs. As a matter of fact, I wanted to write down six anectodes but I decided that two are enough to make the point:

As a task of a university course I had to write a Java program that finds the shortest route (of cities) from one city to any other using Dijkstra’s Shortest Path Algorithm. We were given the desired output for one city as the starting point and we were given a file with the nodes and edges that were needed to solve this task. I finished the task and ran the file. To my surprise the output was wrong! I didn’t want to believe it so I ran the file again and, voila, the output was correct!? Why? I didn’t change anything! After some tests I observed that, whenever I ran the file after not having run it for some time the wrong output was produced. However, when running the file quickly in succession apart from the first time the correct output was produced. I mean, what is this!? What does such a behaviour tell me? In fact, my code was correct all along since later it turned out that there was a wrong value for an edge in the given file… But how could this weird behaviour tell you such a thing? If the program had always produced the wrong output I would have been okay with it but it didn’t9.

As you know, the simulation actuallly has been an auxiliary program for the greater university task of making a driving robot play mastermind. Well, nearing the end of the project’s deadline we had finished the program and we made a test run with the robot. We knew the secret code and the algorithm so we knew which buttons the robot had to approach and in wich order. Everything seemed to be great until the dreaded null-pointer exception peep destroyed our confidence shortly before the robot would have entered the correct sequence! We couldn’t understand it and the code didn’t give any clues either. In fact, this exception never happened again10. We weren’t the only victims of null-pointer exceptions. All the time the other groups encountered them - even in the final run. If there are possibilities and programming languages that allow us to be freed from “the null-pointer exception” why then do we still hold to it by not changing our tools?

To sum it up, I wanted to stress that bug hunting for this simulation and in general is a dominant but bad and hated part of programming. Every means to reduce bug hunting11 should be encouraged. " -- http://web.archive.org/web/20120402183419/http://www.eugenkiss.com/blog/2011-05-experience-report-a-mastermind-simulation-in-lua-java-and-haskell.html

lua vs js:

https://news.ycombinator.com/item?id=8701858

munificent 10 days ago

> Now the only thing Lua has on JS, implementation-wise, is an implementation that is very small/embeddable and very fast (LuaJIT?).

Well, and a simpler, more regular syntax. Simpler semantics. Real coroutines. Tail call elimination. Multiple returns. Better iteration syntax and protocol. Better lexical scope. Operator overloading. A well-established FFI. No weird variable hoisting, nasty implicit conversions, semicolon insertion, etc.

But, sure, aside from those things, Lua doesn't have anything on JS as a language. (Of course, as an ecosystem, JS is miles ahead.)

reply

---

weeksie 10 days ago

I immediately thought of Lua as well. A fine choice, Lua has the best C FFI I've ever worked with. I love everything about this.

reply

cheez 10 days ago

Does Lua work well with multiple threads?

reply

haberman 10 days ago

Lua has full support for creating multiple interpreter instances (ie. there are no global variables, and it's totally re-entrant). But interpreter instances are not thread-safe. So the way to support multiple threads is to have completely independent interpreters running in different threads.

reply

sharkbot 10 days ago

You can make a Lua interpreter thread safe by adding a GIL. There is implicit support for this in the code: the interpreter invokes lua_lock/lua_unlock at the proper places, the default Lua implementation just has them NOP'd out. Adding the thread primitives is up to you, however.

reply

chii 9 days ago

when you have multiple threads, isn't the idea to share data between threads? I haven't really worked with embedded lua before - does lua help with that?

reply

cheez 10 days ago

Thanks. Any idea as to the footprint of these instances?

reply

mempko 10 days ago

I use this approach in Fire★ (http://www.firestr.com). Lua vm instances are tiny. There is a reason lua is used even in systems like nintendo DS

reply

---