Making System Programming Easier

Lua has been a great language for a great many things.  I find the core language itself adequate for many of the typical programming tasks I engage in from low level graphics, to higher level UI stitching.  As with most scripting languages, one of the great benefits comes from the language’s ability to help me stitch together bits and pieces of core OS technology.  I am not inventing TCP/IP from scratch, but rather calling into the OS specific libraries that implement networking.   Same goes for basic window and mouse/keyboard management.

There are other parts of the native system that I want to get at, and Lua, or rather LuaJIT, provides fairly easy glue mechanisms to help me get at those pieces.  One of the things I want to get at is the Heap memory management on Windows.  Heap management includes things such as HeapCreate(), HeapAlloc(), HeapFree() and the like.  What I would like to have is a way to get at those calls, and make it feel as simple and easy as dealing with any other “objects” in my Lua code.  Most notably, I want to have the garbage collector get into the act and call the appropriate functions when items I allocate go out of scope.

So, here’s what I’ve done.

First of all, I can create a heap by using this simple function:

function CreateHeap(initialSize, maxSize, options)

If you want to create a heap with a given size, and with certain restrictions, you can specify those when you call the function.  If you just want a heap that will grow to whatever size, and allow you to allocate as big as you want, you can simply call it with no parameters.

local  gheap = CreateHeap()

local blob = gheap:Alloc(1024)

print("Blob IsValid: ", blob:IsValid(), blob:Size())

blob:ReAlloc(2048, HEAP_REALLOC_IN_PLACE_ONLY)

-- Do lots of fun things with the blob, then finally
-- Get rid of it by setting the reference to nil

blob = nil

Here’s an example of creating a heap, then allocating a blob from it, resizing that blob, then finally removing any reference to the blob, which will eventually have the GC kick in and try to garbage collect the thing.

In order to make this work properly, I’ve defined a couple of structures.

ffi.cdef[[
    typedef struct {   
        HANDLE Handle;   
        int  Options;   
        size_t InitialSize;   
        size_t MaximumSize;
    } HEAP;

    typedef struct {  
        void * Data;  
        HEAP *Heap;
    } HeapBlob;
]]

Then, I define some code to go with them.  The most important piece is a metamethod that must have the __gc name.  By doing this, when the gc is about to delete this object, the function will be called, and you can do whatever cleanup you need to do, like calling the system function to free up the memory.

Heap = nil
Heap_mt = {
    __gc = function(self)
        if self.Handle == nil then return nil end
        local success = kernel32.HeapDestroy(self.Handle) ~= 0
    end,

    __index = {
        Alloc = function(self, nbytes, flags)
            flags = flags or 0
            nbytes = nbytes or 1
            local ptr = kernel32.HeapAlloc(self.Handle, flags, nbytes)

            -- If the allocation failed, then just return
            if ptr == nil then
                return nil
            end

            -- Create a blob object, and return that to the
            -- caller.
            local blob = HeapBlob(ptr, self)
            return blob
        end,

        IsValid = function(self, flags)
            flags = flags or 0
            local isValid = kernel32.HeapValidate(self.Handle, flags, nil)
            return isValid
        end,
    },
}
Heap = ffi.metatype("HEAP", Heap_mt)

Of course, if you were just doing a Alloc/Free, then you do just use ffi.gc(alloc, free), and that would be fine as well.  But, by encapsulating this all in one little object, you keep things tidy.  You want to know when the object containing the pointer has lost all references, not references to the pointer itself.  Why?  Well, consider the case where you’re passing a chunk of heap allocated memory to a system function.  You’ll want to control the lifetime of that pointer adequately such that it’s not freed by the gc prematurely.  OK, you could argue that you could easily control the lifetime of references to that pointer just as easily.  The second benefit of wrapping it up in a structure is that you have those nice convenience methods as well.  So, getting at the size, reallocating when necessary, is just a matter of making method calls on the object, and not having to worry about return values, whether you’ve givent he right command, etc.  The same benefits that accrue to any properly wrapping object in any language.

---