Introducing LAPHLibs

Lua Application Programming Helper Libraries (LAPHLibs)

Can be found here: LAPHLibs

The LAPHLibs are a set of core routines that I have found to be useful for fairly low level programming using LuaJIT.  In particular, when programming highly performant web services, I need much finer control over memory allocations as is afforded by Lua.  It’s all pure LuaJIT code with no dependencies on any external libraries, thus they are portable to wherever LuaJIT goes.

I love the Lua string matching facilities, for example, but they necessitate copying data into a Lua string in order to use them.  In some cases, the standard C library functions are sufficient to do the job.  The LAPHLibs provide those functions.  Along with the classic string manipulation routines that are seen in C, I’ve also stuffed in some of the memory management routines, such as memmove, which is useful in some cases.  Some day, LuaJIT may provide user injectable assembly language support, and if that happens, much of these routines will become simply assembly routines, and be even more performant.

Having some hashing functions that operate on buffers instead of lua strings is also a necessity, so there is a MD5 implementation as well, which takes in a chunk of memory, and delivers a digest.

And to wrap things up, the BitBang routines are in there, to provide some low level bit twiddling support.  This is primarily beneficial when reading and writing bit based structures on a wire.

Many/most of these functions can be found in the standard C library that was used to compile LuaJIT in the first place.  My first thought might have been, ‘why not just use FFI to get at the routines that are in the standard library’?  Well, because not all platforms will implement the exact form of the routines provided here.  strlcat, for example, is primarily a BSD thing, and Microsoft has something else, and who knows what the ARM libraries provide.  By having this implemented in LuaJIT itself, I ensure the availability, and can guarantee the semantics.

I found myself asking the question, ‘why bother writing all this stuff that looks like C in Lua’?  The answer is pretty esoteric, and has to do with why Lua is interesting as a language to begin with.  One of the most challenging and error prone aspects of dealing with C has to do with memory management.  Pointers going astray has probably been the biggest source of the most egregious errors in programming history.  As a tool, LuaJIT allows a programmer to create executable code on the fly, and run it from a new chunk of  memory.  That’s another potential virus vector waiting to happen.  But, then again, Lua provides some amount of comfort and safety when it comes to memory management.  Since there is garbage collection, I don’t have to worry about balancing my malloc/free occurences.  That’s a big relief right there.

On top of core memory management, Lua also provides a very compact programming surface area.  There isn’t too much syntactic sugar, so all those conveniences that I have in C, such as auto incrementing of variables: a++, a+=1, and the like, are not available.  At first, this might seem to be a bummer, but in fact, it forces me to write code that is more explicit, easier to understand, and easier to maintain.  I don’t find myself wondering about the value of a variable in a loop if I use –a vs a–.

On top of the compactness argument, Lua is an easily parseable language.  Using the LPEG library, the parser for LuaScript is about a couple pages worth of code.  In this case, the compactness means that Lua can be translated into other target languages, such as JavaScript, or even straight C, fairly easily.  This is a tremendous advantage when trying to write a bit of core code that is supposed to work across multiple environments.

This is one area where the LAPHLibs are beneficial.  If I write my code in a fairly straight forward way, and use the LAPHLibs, when it comes time to translate the code to C, it will be a fairly straight forward conversion, as I will be able to rely on the C versions of the functions with the same names and semantics.

At any rate, this is an excursion in testing the performance limits of this language, and these libraries provide one mechanism for utilizing LuaJIT in highly performant situations.  Lua is a great language for stitching things together quickly.  With some core pieces, as provided by the LAPHLibs, that stitching becomes incrementally more interesting.

 

---