LJIT2libc – LuaJIT IS “batteries included”

I would say one of the most common complaints about any platform, framework, product is a paucity of available stuff to fiddle about with.  I’ve heard this criticism a few times leveled at the lua world in general.  Fatter frameworks are usually “batteries included”.  That’s great, when you get a whole ton of stuff that makes writing all sorts of apps relatively easy to do right off the bat.  The challenge of “batteries included” is that you get a whole ton of stuff, most of which you will never use, and some of which doesn’t have the best implementation.

Recently, I’ve been doing quite a lot off luajit bindings on the Linux platform:

If you were into anything from ASCII art graphics drivers to raw frame buffer management in a Linux system, these might start to look like batteries.  They’re not included with the luajit compiler, but they’re relatively easy to add.

But, there’s another one that I’ve been playing with recently which I think is even better:

Luajit is typically built and compiled against the libc and libm libraries.  As such, being able to access routines within those libraries comes for ‘free’ from within luajit, courtesy of the ffi capabilities.  This is very interesting because it means these routines, which are already on your system, are in fact just a stones throw away from being available within your app.
Let’s imagine you wanted to write some script like this, using the libc provided ‘puts()’ function:
puts("Hello, World!");

Well, the lua language has its own print routine, so this is a bit contrived, but let’s say you wanted to do it anyway. Well, to make this work, you need access to the function signature for the ‘puts’ routine and you need that to be accessible in the global namespace. So, it would actually look like this:

local ffi = require("ffi")

ffi.cdef("int puts(const char *);");
puts = ffi.C.puts;

puts("Hello, World!")

Great. A bit of work, and now I can program as wrecklessly as I could in C with all the batteries included in the libc and libm libraries. But, it gets tedious having to write that ffi stuff all over the place, so this is where the LJIT2libc thing comes in. It basically goes and implements a ton of the required ffi.cdef statements to make all those functions readily available to your script. An easy way to pull it into any script is to do the following:

local init = require("test_setup")()

puts("Hello, World!");

That first line ‘init = …’ in this case pulls in all of the definitions, and puts them into the global namespace, so that you can simply write your program without having to know anything about the ffi, or where functions come from or anything like that. Just start writing your code knowing that the batteries are already included.

Now, this example might seem too trivial for words, but just think about what’s in a typical libc library. It’s all the file system, sockets, system calls, math, random numbers, everything you’re likely to need to create higher level application stuff. It’s a lot of stuff that other systems end up creating from scratch, as part of their ‘batteries’.

Of course this is trivializing what’s happening in those batteries, because what you’re getting here is the raw C implementations of things, with all the headaches and dangers that are typically associated with writing code at that level. But, for those who want to have access to that level of code, and apply the safety net of lua where they see fit, this is quite a useful tool I think.

In short, batteries ARE included with every luajit. Those batteries might be just anode, cathode, and electrolyte, without the shell, but there are the raw ingredients. Having these wrappers available just makes it that much easier to think about and deal with a lot of low level stuff where I might previously had to resort to some sort of ‘package’ to achieve something as simple as traverse the file system.

So there you have it. luajit is a ‘batteries included’ system.


Taming VTables with Aplomb

If you do enough interop work, you’ll eventually run across a VTable that you’re going to have to work with.  I have previously dealt with OpenGL, which doesn’t strictly have a vtable, but has a bunch of functions which you have to lookup in order to use.  In explored the topic in this article: HeadsUp OpenGL Extension Wrangling

Recently, I have been writing code to support TLS connections in TINN.  This ultimately involves using the sspi interfaces in Windows, which leads you to the sspi.h header file which contains the following:

 

typedef struct _SECURITY_FUNCTION_TABLE_A {
    unsigned long                       dwVersion;
    ENUMERATE_SECURITY_PACKAGES_FN_A    EnumerateSecurityPackagesA;
    QUERY_CREDENTIALS_ATTRIBUTES_FN_A   QueryCredentialsAttributesA;
    ACQUIRE_CREDENTIALS_HANDLE_FN_A     AcquireCredentialsHandleA;
    FREE_CREDENTIALS_HANDLE_FN          FreeCredentialHandle;
    void *                      Reserved2;
    INITIALIZE_SECURITY_CONTEXT_FN_A    InitializeSecurityContextA;
    ACCEPT_SECURITY_CONTEXT_FN          AcceptSecurityContext;
    COMPLETE_AUTH_TOKEN_FN              CompleteAuthToken;
    DELETE_SECURITY_CONTEXT_FN          DeleteSecurityContext;
    APPLY_CONTROL_TOKEN_FN              ApplyControlToken;
    QUERY_CONTEXT_ATTRIBUTES_FN_A       QueryContextAttributesA;
    IMPERSONATE_SECURITY_CONTEXT_FN     ImpersonateSecurityContext;
    REVERT_SECURITY_CONTEXT_FN          RevertSecurityContext;
    MAKE_SIGNATURE_FN                   MakeSignature;
    VERIFY_SIGNATURE_FN                 VerifySignature;
    FREE_CONTEXT_BUFFER_FN              FreeContextBuffer;
    QUERY_SECURITY_PACKAGE_INFO_FN_A    QuerySecurityPackageInfoA;
    void *                      Reserved3;
    void *                      Reserved4;
    EXPORT_SECURITY_CONTEXT_FN          ExportSecurityContext;
    IMPORT_SECURITY_CONTEXT_FN_A        ImportSecurityContextA;
    ADD_CREDENTIALS_FN_A                AddCredentialsA ;
    void *                      Reserved8;
    QUERY_SECURITY_CONTEXT_TOKEN_FN     QuerySecurityContextToken;
    ENCRYPT_MESSAGE_FN                  EncryptMessage;
    DECRYPT_MESSAGE_FN                  DecryptMessage;
    SET_CONTEXT_ATTRIBUTES_FN_A         SetContextAttributesA;
    SET_CREDENTIALS_ATTRIBUTES_FN_A     SetCredentialsAttributesA;
    CHANGE_PASSWORD_FN_A                ChangeAccountPasswordA;
} SecurityFunctionTableA, * PSecurityFunctionTableA;

You get at this function table by making the following call:

local sspilib = ffi.load("secur32");
local VTable = sspilib.InitSecurityInterfaceA();

And then, to execute one of the functions, you could do this:

local pcPackages = ffi.new("int[1]");
local ppPackageInfo = ffi.new("PSecPkgInfoA[1]");
local result = VTable["EnumerateSecurityPackagesA"](pcPackages, ppPackageInfo);

-- Print names of all security packages
for i=0,pcPackages[0] do
  print(ffi.string(ppPackageInfo[0][i].Name));
end

Tada!! What could be simpler…

Well, this is Lua of course, so things could be made a bit simpler.

First of all, why is there even a vtable in this case? All these functions are just in the .dll file directly aren’t they? Well, there’s a bit of trickery when it comes to security packages. It turns out, it’s best not to actually load the .dll that represents the security package into the address space of the program that’s using it, directly. By calling “IniSecurityInterface()”, the actual package is loaded into a different address space, and the vtable is then used to access the functions.

You can make multiple calls to InitSecurityInterface() to get that vtable pointer, or you could stuff it into a global variable, making it available to all modules within your program, or, you could stuff it into a bit of a table wrapping and make life much easier.

-- sspi.lua
local ffi = require("ffi");

local sspi_ffi = require("sspi_ffi");
local SecError = require ("SecError");
local sspilib = ffi.load("secur32");
local SecurityPackage = require("SecurityPackage");
local Credentials = require("CredHandle");
local schannel = require("schannel");

local SecurityInterface = {
  VTable = sspilib.InitSecurityInterfaceA();
}
setmetatable(SecurityInterface, {
  __index = function(self, key)
    return self.VTable[key]
  end,
});

return {
  schannel = schannel;
	
  SecurityInterface = SecurityInterface;
  SecurityPackage = SecurityPackage;
  Credentials = Credentials;
}

With this little bit, I can now do this in my program:

local sspi = require("sspi");
local SecurityInterface = sspi.SecurityInterface;

local pcPackages = ffi.new("int[1]");
local ppPackageInfo = ffi.new("PSecPkgInfoA[1]");

local result = SecurityInterface.EnumerateSecurityPackagesA(pcPackages, ppPackageInfo);

The SecurityInterface table takes care of loading the VTable as part of it’s construction. By doing the setmetatable, and implementing the ‘__index’ metamethod, whenever a ‘.functionname’ is asked for, as with ‘.EnumerateSecurityPackagesA’, the element within the vtable with that name will be returned. Those elements so happen to be function pointers, so they will then just be executed like regular functions!

I think that’s a pretty awesome trick. The SecurityInterface table looks like a static structure with function pointers, and you just get to call those functions directly, passing in the appropriate arguments. This looks pretty much exactly like what I would expect if I were writing this in C, but I don’t have to worry about type casts and the like.

This works in this particular case because there is a single table representing the function pointers. If you were instead doing something where there were instances of an object, and an attendant vtable, you’d have to do a little bit more work to preserve the instance data, and pass it into the individual functions. Not too hard, and I actually do this trick in my Kinect interface implementation.

At any rate, that’s a relatively easy way to tackle vtables without much work. It was actually a bit surprising to me that it worked so easily, and I’ve been able to refine a pattern that I somewhat understood before, and now truly appreciate.