Loading Multiple Lua States

Here’s the thing.  Ultimately I want to use I/O completion ports on Windows to create highly scalable networking infrastructure.  I don’t want to write a lick of C code, I want to stay completely within the confines of Lua.  So what to do?  Well, first off, when you work with IOCP, you need to assign some threads to deal with the completion of various IO operations.  OK.  Easy enough, with LuaJIT, just create the interop function to the CreateThread() system call.  But wait, I want to execute some lua code in that thread…

Alrightly then, why don’t I just pass in the lua state that I have now and…

Never mind.  What I really need to do is create a thread, and within that thread I need to create a lua state object, and have that object execute the little bit of code that is needed.  With LuaJIT, this is totally possible.  Normally, when you create a lua_State, you are writing code in standard ‘C’, or whatever environment.  But, with LuaJIT< the lua51.dll is just as accessible as any other library in the world, so you can simply access it and create your state as your normally would in C.

But, there’s a rub, first you need that massive ffi definitions file that mimics the appropriate .h files.  And so, first I had to create Luaffi.lua, which is part of the BanateCoreWin32 files.  What is this file?  It is basically an amalgamation of the various header files that are used to create Lua.   Namely, it includes the contents of: luaconf.h, lua.h, lauxlib.h, and lualib.h

This was a fairly mindless task of copying over the appropriate part, deciding whether a #define was a simple alias to something, a function, or a constant, and putting ffi.cdef[[]] around all the appropriate functions.  It compiles cleanly, but that does not mean everything actually works correctly.  I’ll have to go through it a few times to ensure everything is actually correct.

There is one big Warning in there.  There is a constant, that comes from stdio: BUFSZ

On Windows, this is defined as 512, so I explicitly set the value to 512.  This value is completely dependent on your system, and should be set appropriately.  Ideally, this would be a value that could be queried in the system, because this is the most fragile bit of this interop.  It’s not used in too many places, but when it is, it will likely break things.

And so, what do you get for your troubles?

local ffi = require "ffi" lua = require "Luaffi"
function report_errors(L, status) 
    if status ~=0 then 
        print("-- ", ffi.string(lua_tostring(L, -1))) 
        lua_pop(L, 1); -- remove error message 
    end 
end
function execlua (codechunk) 
    local status, result local L = lua.luaL_newstate();  -- create state

    if L == nil then 
        print("cannot create state: not enough memory") 
        return EXIT_FAILURE; 
    end

    -- Load whatever libraries are necessary for 
    -- your code to start 
    print("luaopen_base: ", lua.luaopen_base(L)) 
    print("luaopen_io: ", lua.luaopen_io(L)); 
    print("luaopen_table", lua.luaopen_table(L)); 
    print("luaopen_string", lua.luaopen_string(L)); 
    print("luaopen_math", lua.luaopen_math(L)); 
    print("luaopen_bit", lua.luaopen_bit(L)); 
    print("luaopen_jit", lua.luaopen_jit(L)); 
    print("luaopen_ffi", lua.luaopen_ffi(L));

    --lua.luaL_openlibs(L);
    -- execute the given code chunk 
    result = luaL_dostring(L, codechunk) 
    print("Result: ", result) 

    report_errors(L, status)

    lua.lua_close(L); 
end

execlua("print('hello lua')")

That’s a fairly standard looking “main()” for using Lua.  Basically, do everything within LuaJit itself, without having to use a lick of C code.  Now that I have this basic capability, the rest of the task is fairly straight forward.

One challenge ahead is how to communicate between threads.  Well, I’m a big believer in message passing.  From years of doing multi-threaded, multi-processor programming, I know full well that I’m not good at maintaining shared memory state.  Over the years, I have found that the best way to keep things straight is to simply pass messages.  Granted, debugging an asynchronous message passing system is no walk in the park either, but it makes for much more easily scalable systems.  By using message passing, you can focus on ensuring that the messaging mechanism between processes works correctly, and forget the rest.  This style also lends itself easily to being distributed, either across processes, or across the internet, which is a good thing.

One way to pass messages between threads on Windows is to use the PostThreadMessage() function.  Each thread will be running in a little message loop, and when a message comes in, it can be placed on a queue within the lua state, and the executing code (which should be in a coroutine) can pull it out and deal with it at its leisure.

Of course, to extend more broadly, PostThreadMessage can be aliased with something more interesting, like PostIPNode, or whatever.  As long as the function can take a Blob and send it to its destination, I shouldn’t really care.

So, there it is.  First steps towards making a highly scalable Lua based processing engine.

---