Luica Obscura

I’ve been fiddling around with Lua for a few months now, and it’s been an interesting experience.  Lua is an interesting language/environment for a few reasons.  One of the easiest is the size, and easy embeddability.  I actually played with Javascript engines as well, but Lua seemed to be the most capable for the size alloted.

One of the downsides of Lua is that it does not come with all the libraries known to man.  That is reserved for the well heeled Ruby, Python, and Javascript ecosystems.  But, then again, since the core Lua environment has no “batteries included”, doing a Lua distribution is a much less daunting task.  At the very least, it’s just a single .dll, lua51.dll.  That single dll is < 300K dripping wet.  To put that in perspective, I just recently visited the web page, and the combined load on my browser was about 750K of .html and pictures.  I don’t get any app development environment for that amount of code, just some pretty pictures on my screen.

Looking at the various Javascript engines, I think they’re a lot larger than that.

But, whenever I say the word “Lua” to any of my programmer friends, I get anything from quizical “tell me more” looks, to the raised eyebrows of “he’s gone off the deepend”.  I think primarily this is driven by the relative obscurity of the Lua language.  It’s not well known for driving web development, it’s primarily known for its usage in game development environments, particularly World of Warcraft.

Then along comes LuaJIT.  Anyone who’s looked at the world of dynamic programming languages knows that LuaJIT has sat near or at the top of the leader board in terms of performance.  The fact that it’s a Just In Time compiler is what does the trick.  Your more typical javascript engine would be left in the dust.  As it is, LuaJIT is about as performant as a Java VM, or even compiled C code.  For all practical purposes, I’ll just assume it’s the same as writing code in C.

But, that’s not satisfying.  Why would I switch languages just to get the same performance?  Well, the problem with C is the strength of C.  Pointers are the biggest pain, right there with memory management.  You have all the power to bend the machine to your will, but with that awesome power comes awesome responsibility, as well as buffer overflows, dangling pointers, unbalanced malloc/free, and the like.  Lua kind of eliminates most of these concerns because it has a garbage collector, and natively does not support pointers.  LuaJIT takes that Lua core, and adds a bit of danger for those who just MUST have their pointers.  It regains the pointers, with all that implies.

How to explain this really…  LuaJIT is a powerful string processor.  As such, it is much more powerful than RegEx, much more compact, and much more expressive.  Using LuaJIT, I can easily handle things like http parsing in a fraction of the space required to do so using any other means, including those various libraries that come with myriad other environments.  Since LuaJIT is a runtime compiler, my code is as fast as if I had written it natively in C.  Since I use LuaJIT as a string handler, and the Lua code itself is just a string, I can easily update my string handling capabilities without having to take down, or re-deploy the core host environment that’s running the string handler.

Well, isn’t that a funny thought?  And thus, dealing with HTTP, at first I used the http_parser, and LuaJIT interop to call that library.  But, after some wrangling, I have been able to replicate that library, entirely in Lua code, so there is no library to deploy.  At about 40K of code, this is a pretty good thing.  What does it mean?  Let’s say I have a generic network node that’s just kind of sitting out there in the world.  That node has the simple ability to listen on a port, and feed whatever comes in to a routine that will deal with it.  By default this simple node does nothing but throw the received packets away and carry on receiving.  It’s rather a black hole.  But now I want to give it the power to deal with HTTP requests.  I can send it a http parser, as luascript.  Now, when it receives a chunk, it is sent to the http parser code, when then executes.  By default, still not doing anything but parsing the http, and having knowledge of various requests.  Then I send it a bit of code to deal with file retrieve requests (a simple static web page service).

Great, starting from small little bits of code, building up to more robust services.

Let’s say I want a node to take on the face of a REST service, that is serving up something.  Same process, send it a bit of code to deal with http, then some to deal with the REST service interface.  Oops, then I found a mistake in the REST handling.  Just send a bit of code replacing the bit of code sent before.  When it’s not being used, it will simply be replaced, re-compiled on the fly, and none the wiser. 

Assuming I had about 100 such nodes distributed around the world, how could I do something like A/B testing, or rather “test in production”.  Well, send out the little bit of new code to as many nodes as you want, see how it goes, gather data, and if all is well, tell the rest of the nodes to pick up the new bit of code.  No need to take servers down, and various other bad things that occur when you have to deploy fully compiled binaries that have been tested in-house.

But, it’s just simple string handling…  There is something about the dynamic nature that makes life more interesting.  I don’t want to get all LISPish about it, but I do think this is the way of things.  Javascript proves server side dynamic script is a viable approach.  LuaJIT just makes it that much more interesting, and I hope over time it will prove to be just as capable, if not moreso.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s