Experiences of a Confessed LuaJIT Binder – The all in one

It seems I’ve been writing binders, wrappers, frameworks, and the like for the entirety of my 35 years of programming.  Why is that?  Well, because the tools I use to program are often times not the same as those used to create various libraries, frameworks, and the like.  Sometimes it’s just that there is no universal bond between all those disparate libraries that you want to use, so you end up writing some wrappers, to make things look and behave similar.

I’ve written quite a lot about LuaJIT bindings, and produced rather a lot of them myself.  Most recently, I was writing a binding for the D-Bus library on Linux.  D-Bus is a communications protocol and mechanism meant to support Interprocess Communications.  As such, it’s about low latency, speed on read and write, and relative simplicity.  You can do everything from pop-up and alert, to logging commands in a persistent log.

Here I want to show an approach that I’ve slowly grown and evolved over the past few years.  This approach to writing bindings is written with a few constraints and aspirations in mind:

  • Provide a low level interface that is as true to the underlying framework as possible.
  • Porting typical C code to the wrapping should be straight forward and obvious
  • Provide an interface that supports and leverages Lua idioms and practices
  • Provide a wrapper that does not require a separate compilation step

That list of constraints is brief, but can cause enough trouble depending on how seriously you take each one of them.  So, let’s get to it.

Here I will use the LJIT2dbus project, because it gives a chance to exhibit all of the constraints listed above.  Here’s a bit of code:

 

--dbus.lua
local ffi = require("ffi")

if not DBUS_INCLUDED then

local C = {}

require ("dbus-arch-deps");
require ("dbus-address");
require ("dbus-bus");
require ("dbus-connection");
require ("dbus-errors");
require ("dbus-macros");
require ("dbus-message");
require ("dbus-misc");
require ("dbus-pending-call");
require ("dbus-protocol");
require ("dbus-server");
require ("dbus-shared");
require ("dbus-signature");
require ("dbus-syntax");
require ("dbus-threads");
require ("dbus-types");

C.TRUE = 1;
C.FALSE = 0;

Yah, ok, not so exciting, just a bunch of ‘require()’ statements pulling in other modules. What’s in one of these modules?

-- dbus-misc.lua

local ffi = require("ffi")

require("dbus-types")
require("dbus-errors")

ffi.cdef[[
char*       dbus_get_local_machine_id  (void);

void        dbus_get_version           (int *major_version_p,
                                        int *minor_version_p,
                                        int *micro_version_p);
]]

Yes, again, more require() statements. But then, there are those couple of C functions that are defined. The other files have similar stuff in them. This is the basis of the constraint related to making the code familiar to a C programmer.

Let’s look at another file which might turn out to be more illustrative:

local ffi = require("ffi")


require("dbus-macros")
require("dbus-types")
require("dbus-protocol")




ffi.cdef[[
/** Mostly-opaque type representing an error that occurred */
typedef struct DBusError DBusError;
]]

ffi.cdef[[
/**
 * Object representing an exception.
 */
struct DBusError
{
  const char *name;    /**< public error name field */
  const char *message; /**< public error message field */

  unsigned int dummy1 : 1; /**< placeholder */
  unsigned int dummy2 : 1; /**< placeholder */
  unsigned int dummy3 : 1; /**< placeholder */
  unsigned int dummy4 : 1; /**< placeholder */
  unsigned int dummy5 : 1; /**< placeholder */

  void *padding1; /**< placeholder */
};
]]


local function DBUS_ERROR_INIT()
  return ffi.new("struct DBusError", { NULL, NULL, TRUE, 0, 0, 0, 0, NULL });
end


ffi.cdef[[
void        dbus_error_init      (DBusError       *error);
void        dbus_error_free      (DBusError       *error);
void        dbus_set_error       (DBusError       *error,
                                  const char      *name,
                                  const char      *message,
                                  ...);
void        dbus_set_error_const (DBusError       *error,
                                  const char      *name,
                                  const char      *message);
void        dbus_move_error      (DBusError       *src,
                                  DBusError       *dest);
dbus_bool_t dbus_error_has_name  (const DBusError *error,
                                  const char      *name);
dbus_bool_t dbus_error_is_set    (const DBusError *error);
]]

So more require() statements, a data structure, and some C functions. And how to use it? For that, let’s look at the bottom of the dbus.lua file.

 

local Lib_dbus = ffi.load("dbus-1")


local exports = {
	Lib_dbus = Lib_dbus;
}
setmetatable(exports, {
	__index = function(self, key)
		local value = nil;
		local success = false;

		-- try looking in table of constants
		value = C[key]
		if value then
			rawset(self, key, value)
			return value;
		end


		-- try looking in the library for a function
		success, value = pcall(function() return Lib_dbus[key] end)
		if success then
			rawset(self, key, value);
			return value;
		end

		-- try looking in the ffi.C namespace, for constants
		-- and enums
		success, value = pcall(function() return ffi.C[key] end)
		--print("looking for constant/enum: ", key, success, value)
		if success then
			rawset(self, key, value);
			return value;
		end

		-- Or maybe it's a type
		success, value = pcall(function() return ffi.typeof(key) end)
		if success then
			rawset(self, key, value);
			return value;
		end

		return nil;
	end,
})

DBUS_INCLUDED = exports
end

return DBUS_INCLUDED

And just for reference, a typical usage of the same:

local dbus = require("dbus")
  local err = dbus.DBusError();
  dbus.dbus_error_init(err);
  local bus = dbus.dbus_bus_get(dbus.DBUS_BUS_SESSION, err);

  if (dbus.dbus_bus_name_has_owner(bus, SYSNOTE_NAME, err) == 0) then
    io.stderr:write("Name has no owner on the bus!\n");
    return EXIT_FAILURE;
  end

First case is the creation of the ‘local err’ value.

  local err = dbus.DBusError();
  dbus.dbus_error_init(err);

The dbus.lua file does not have a function called ‘DBusError()’. All it has done is load a bunch of type and function declarations, to be used by the LuaJIT ffi mechanism. So, how do we get a function of that name? It doesn’t even exist in one of the required modules.

The trick here is in the ‘__index’ function of the dbus.lua table. The way the Lua language works, any time you make what looks like an access to the member of a table, if it can’t be found in the table, and if the __index function is implemented, it will get called, with the key passed in as a parameter.

In this case, the ‘__index’ function implements a series of lookups, trying to find the value that is associated with the specified key. First it tries looking in a table of constants. Then it tries looking in the actual library, for a function with the specified name. If it doesn’t find the value as a function, it will try to find it as a constant in the C namespace. This will find any enum, or static const int values that have been defined in ffi.cdef[[]] blocks. Finally, if it doesn’t find the key as one of the constants, it tries to figure out if maybe it’s a type (‘ffi.typeof’). This is in the case of DBusError, this one will succeed, and we’ll get back a type.

As it turns out, the types returned from the LuaJIT ffi can be used as constructors.
So, what we really have is this:

local errType = dbus.DBusError;
local err = errType();

The fact that you can just do it all on one line is a short hand.

This is a great convenience. Also, once the value is found, it is stuck into the dbus table itself, so that the next time it’s used, this long lookup won’t occur. The type is already known, and will just be retrieved from the dbus (exports) table directly.

Well, this works for the other kinds of types as well. For example, the ‘dbus_error_init()’ function is defined in a ffi.cdef[[]] block, and that’s about all. So, when re reference it through dbus.dbus_error_init(), we’re going to look it up in the library, and try to use the function found there. And again, once it’s found, it will be stuffed into the dbus (exports) table, for future reference.

This works out great, in that it’s fairly minimal amount of work to get all interesting things defined in ffi.cdef blocks, then just use this __index lookup trick to actually return the values.

I’ve come to this style because otherwise you end up doing a lot more work trying to make the constants, types, enums, and functions universally accessible from anywhere within your program, without resorting to global values. Of course you can make the __index lookup as complex as you like. You can lazily load in modules, for example.

That’s it for this round. An all in one interface that gives you access to constants, enums, and functions in a ffi.cdef wrapped library.


Note To Self – Enumerating bit flags

I’ve been trawling through the Linux V4L2 group of libraries of late as part of LLUI.  v4l2 is one of those sprawling libraries that does all things for all people in terms of video on Linux machines.  It’s roughly equivalent to oh so many similar things from the past on the Windows side.  This is one of the libraries you might utilize if you were to get into streaming from your webcam programmatically.  Of course, you could just read from it directly with libusb, but then you lose out on all the nifty format conversions, and I miss this chance to write another pointless reminder for my later coding self.

So, what’s got me so bothered this time around?  Well, lets say I’m just parusing my system, turning everything into a database as I go along.  I’d like to get a hold of my webcam, and see what it’s capable of.  There’s a call for that of course.  Once you make the appropriate IOCtl call, you end up with a struct that looks like this:

 
[soucecode]
struct v4l2_capability {
uint8_t driver[16];
uint8_t card[32];
uint8_t bus_info[32];
uint32_t version;
uint32_t capabilities;
uint32_t device_caps;
uint32_t reserved[3];
};
[/sourcecode]

The driver, card, and bus_info fields are pretty straight forward as they are simple ‘null terminated’ strings, so you have print them out if you like. It’s that ‘capabilities’ field that gives me fits. This is one of those combined bit flags sort of things. The value can be a combination of any of the numerous ‘capability’ flags, which are these:

-- Values for 'capabilities' field
caps = {
	V4L2_CAP_VIDEO_CAPTURE		= 0x00000001 ; -- Is a video capture device */
	V4L2_CAP_VIDEO_OUTPUT		= 0x00000002; -- Is a video output device */
	V4L2_CAP_VIDEO_OVERLAY		= 0x00000004; -- Can do video overlay */
	V4L2_CAP_VBI_CAPTURE		= 0x00000010; -- Is a raw VBI capture device */
	V4L2_CAP_VBI_OUTPUT			= 0x00000020; -- Is a raw VBI output device */
	V4L2_CAP_SLICED_VBI_CAPTURE	= 0x00000040; -- Is a sliced VBI capture device */
	V4L2_CAP_SLICED_VBI_OUTPUT	= 0x00000080; -- Is a sliced VBI output device */
	V4L2_CAP_RDS_CAPTURE		= 0x00000100; -- RDS data capture */
	V4L2_CAP_VIDEO_OUTPUT_OVERLAY	= 0x00000200; -- Can do video output overlay */
	V4L2_CAP_HW_FREQ_SEEK		= 0x00000400; -- Can do hardware frequency seek  */
	V4L2_CAP_RDS_OUTPUT			= 0x00000800; -- Is an RDS encoder */

	V4L2_CAP_VIDEO_CAPTURE_MPLANE	= 0x00001000;
	V4L2_CAP_VIDEO_OUTPUT_MPLANE	= 0x00002000;
	V4L2_CAP_VIDEO_M2M_MPLANE		= 0x00004000;
	V4L2_CAP_VIDEO_M2M				= 0x00008000;

	V4L2_CAP_TUNER			= 0x00010000; -- has a tuner */
	V4L2_CAP_AUDIO			= 0x00020000; -- has audio support */
	V4L2_CAP_RADIO			= 0x00040000; -- is a radio device */
	V4L2_CAP_MODULATOR		= 0x00080000; -- has a modulator */

	V4L2_CAP_READWRITE              = 0x01000000; -- read/write systemcalls */
	V4L2_CAP_ASYNCIO                = 0x02000000; -- async I/O */
	V4L2_CAP_STREAMING              = 0x04000000; -- streaming I/O ioctls */
}

For the embedded webcam in my laptop, the reported value is: 0x04000001;

Of course, when you’re doing something programmatically, and you just want to check whether a particular flag is set or not, you can just do:

canStream = band(V4L2_CAP_STREAMING, 0x04000001) ~= 0

Very common, and probably some of the most common code you’ll see anywhere. But what else? For various reasons, I want to create the string values for those bit fields, and use those values as keys to tables, or just to print, or to send somewhere, or display, or what have you.

I’ve seen enough ‘C’ code deal with this there is a common patter. First create the #define, or enum statement which encapsulates the values for all the flags. Then, to get the values as strings, create a completely separate string table, which does the mapping of the nice tight enum values and the string values. Then write a little lookup function which can go from the value to the string.

Well, here’s one of those things I love about Lua. In this case, the program IS the database. No need for those parallel representations. Here’s some code:

local pow = math.pow
local bit = require("bit")
local lshift, rshift, band, bor = bit.lshift, bit.rshift, bit.band, bit.bor

local function getValueName(value, tbl)
	for k,v in pairs(tbl) do
		if v == value then
			return k;
		end
	end

	return nil;
end

local function enumbits(bitsValue, tbl, bitsSize)
	local function name_gen(params, state)

		if state >= params.bitsSize then return nil; end

		while(true) do
			local mask = pow(2,state)
			local maskedValue = band(mask, params.bitsValue)
--print(string.format("(%2d) MASK [%x] - %#x", state, mask, maskedValue))			
			if maskedValue ~= 0 then
				return state + 1, getValueName(maskedValue, params.tbl) or "UNKNOWN"
			end

			state = state + 1;
			if state >= params.bitsSize then return nil; end
		end

		return nil;
	end

	return name_gen, {bitsValue = bitsValue, tbl = tbl, bitsSize = bitsSize or 32}, 0
end

return enumbits

The function “getValueName()” at the top there simply does a reverse lookup in the table. That is, given a value, return the string that represents that value (is the string key for that value).

Next, the “enumbits()” function is an enumerator. It will iterate over the bit flags, returning a string name for all the ones that are set to ‘1’, and nothing for any of the other bits. Here’s an example:

local bit = require("bit")
local lshift, rshift, band, bor = bit.lshift, bit.rshift, bit.band, bit.bor

local enumbits = require("enumbits")

local testtbl = {
	LOWEST 	= 0x0001;
	MEDIUM 	= 0x0002;
	HIGHEST = 0x0004;
	MIGHTY 	= 0x0008;
	SLUGGO 	= 0x0010;
	MUGGO 	= 0x0020;
	BUGGO 	= 0x0040;
	PUGGO 	= 0x0080;
}


local caps = {
	V4L2_CAP_VIDEO_CAPTURE		= 0x00000001 ; -- Is a video capture device */
	V4L2_CAP_VIDEO_OUTPUT		= 0x00000002; -- Is a video output device */
	V4L2_CAP_VIDEO_OVERLAY		= 0x00000004; -- Can do video overlay */
	V4L2_CAP_VBI_CAPTURE		= 0x00000010; -- Is a raw VBI capture device */
	V4L2_CAP_VBI_OUTPUT			= 0x00000020; -- Is a raw VBI output device */
	V4L2_CAP_SLICED_VBI_CAPTURE	= 0x00000040; -- Is a sliced VBI capture device */
	V4L2_CAP_SLICED_VBI_OUTPUT	= 0x00000080; -- Is a sliced VBI output device */
	V4L2_CAP_RDS_CAPTURE		= 0x00000100; -- RDS data capture */
	V4L2_CAP_VIDEO_OUTPUT_OVERLAY	= 0x00000200; -- Can do video output overlay */
	V4L2_CAP_HW_FREQ_SEEK		= 0x00000400; -- Can do hardware frequency seek  */
	V4L2_CAP_RDS_OUTPUT			= 0x00000800; -- Is an RDS encoder */

	V4L2_CAP_VIDEO_CAPTURE_MPLANE	= 0x00001000;
	V4L2_CAP_VIDEO_OUTPUT_MPLANE	= 0x00002000;
	V4L2_CAP_VIDEO_M2M_MPLANE		= 0x00004000;
	V4L2_CAP_VIDEO_M2M				= 0x00008000;

	V4L2_CAP_TUNER			= 0x00010000; -- has a tuner */
	V4L2_CAP_AUDIO			= 0x00020000; -- has audio support */
	V4L2_CAP_RADIO			= 0x00040000; -- is a radio device */
	V4L2_CAP_MODULATOR		= 0x00080000; -- has a modulator */

	V4L2_CAP_READWRITE              = 0x01000000; -- read/write systemcalls */
	V4L2_CAP_ASYNCIO                = 0x02000000; -- async I/O */
	V4L2_CAP_STREAMING              = 0x04000000; -- streaming I/O ioctls */
}

local function printBits(bitsValue, tbl)
	tbl = tbl or testtbl
	for _, name in enumbits(bitsValue, tbl) do
		io.write(string.format("%s, ",name))
	end
	print()
end

-- single bits
printBits(lshift(1,0))
printBits(lshift(1,1))
printBits(lshift(1,31))

-- combined bits
printBits(0x0045)
printBits(0x04000001, caps)

With that last test case, what you’ll get is the output:

V4L2_CAP_VIDEO_CAPTURE, V4L2_CAP_STREAMING

Well that’s handy, particularly when you’re doing some debugging. Just a simple 20 line iterator, and you’re in business, printing flag fields like a boss! That is, if you’re in the lua environment, or any dynamic programming environment that supports iteration of a dictionary.

So, this note to future self is about pointing out the fact that even bitflags are nothing than a very compact form of database. Unpacking them into human readable, programmable form, requires just the right routine, and away you go, you never have to bother with dealing with this little item again. Great for debugging, great for sticking keys in tables, great for displaying on controls!


On the insideous deception of simplicity

I find that in programming, there is much tribal knowledge. This is true for every framework, language, OS, blogsphere, and the like. As such, it is the hapless seeker of knowledge who will often fall pray to false assumptions, miscalculations, missteps, and generally an unhappy time of it.

Case in point. A while back, I was working on schedlua, and it came time to add epoll to get some async socket action going.  Well, if you browse the interwebs, looking for things related to epoll, you’ll eventually come across ‘struct epoll_event’.  As I might be dealing with in some ‘binding’ environment, I have need to clearly define what this data structure is.

Here is a typical answer:

struct epoll_event {
    uint32_t events; /* epoll events (bit mask) */
    epoll_data_t data; /* User data */
};

Well that seems easy enough. When I first played with epoll, it was on a Raspberry Pi, and I probably used such a simple definition. I probably copied it from some other code, probably didn’t think much about it at all.

Then, when I did schedlua, things didn’t turn out so well. I used the same simple definition, and somehow I found my data structures were being overwritten, or just had wrong values. What gives? Well, it was actually pointed out to me by someone in the LuaJIT community (Mike Pall) that this particular data structure is defined thus:

typedef union epoll_data {
	void *ptr;
	int fd;
	uint32_t u32;
	uint64_t u64;
} epoll_data_t;

struct epoll_event {
	uint32_t events;
	epoll_data_t data;
}
#ifdef __x86_64__
__attribute__ ((__packed__))
#endif
;

That is, specifically when you’re talking about 64-bit machines, you have to use that packed attribute so that the structure ends up having the same layout as it would have on a 32-bit machine. I guess the Linux kernel itself cares about this alignment for this particular case.

Of course, if I weren’t a lazy programmer, I would have easily seen this if I examined the particulars in the header file it comes from (/usr/include/sys/epoll.h).

The Windows OS is littered with all sorts of this kind of goodness due to its long legacy of compatibility (all the way back to 16-bit computers). It’s just one of those things you have to be aware of.

More recently I’ve been working on the LJIT2libc project. As I wrote in: LJIT2libc – LuaJIT IS “batteries included”, the luajit program, as compared to the liblua51 library, provides access to all of libc, and libm. the only catch is you need the ffi.cdef definitions available to use it.  Well, that’s a heck of a lot of definitions!!  But, I am naive and undaunted by the enormity of the task, so I embarked…

Talk about esoteric tribal knowledge!  Do yourself a favor and browse through the code of one of these ‘libc’ libraries some time.  I used musl as my guide, because it’s modern and fairly well written in my opinion.  At the very least, you can gain an appreciation for the magic and seamless masking of platform specifics this library presents, by browsing the header file structure.  As libc is utilized on multiple different platforms, architectures, OSs, and the like, it has to cover the differences between all of those, and present something that is relatively the same for the programmer who chooses to utilize the library.  My epoll data structures is but one example.  Here’s more esoteric ones:

When you do something as simple as: #include <limits.h> in your C program, what do you get?

On ARM, you get some standard stuff, but the definition of max integer values looks like this:

#if defined(_POSIX_SOURCE) || defined(_POSIX_C_SOURCE) \
 || defined(_XOPEN_SOURCE) || defined(_GNU_SOURCE) || defined(_BSD_SOURCE)
#define PAGE_SIZE 4096
#define LONG_BIT 32
#endif

#define LONG_MAX  0x7fffffffL
#define LLONG_MAX  0x7fffffffffffffffLL

And if you’re dealing with a 64-bit platform, you’re likely to get something like this:

#if defined(_POSIX_SOURCE) || defined(_POSIX_C_SOURCE) \
 || defined(_XOPEN_SOURCE) || defined(_GNU_SOURCE) || defined(_BSD_SOURCE)
#define PAGE_SIZE 4096
#define LONG_BIT 64
#endif

#define LONG_MAX  0x7fffffffffffffffL
#define LLONG_MAX  0x7fffffffffffffffLL

Did you blink? Did you catch that change? LONG_MAX isn’t always LONG_MAX. Same is true for the language specific ‘size_t’. How big are these? What’s their range? It depends.

So, for something as simple as integer values and ranges, that tribal knowledge comes in handy. Of course the well groomed programmer wouldn’t make mistakes related to any assumptions around the range of these values, but the poor lazy slobs, such as myself, who don’t know all the details and assumptions, will make their own assumptions, and the bugs will come… eventually.

Doing LJIT2libc gives me an appreciation for how hard it is to actually create a library such as this to satisfy as broad an audience as it does. If I could only match the headers in detail, then LJIT2libc will be applicable to all platforms where luajit lives. That’s probably a good thing.

Pursuing this path for libc makes me appreciate even more the work of Justin Cormack in creating ljsyscall.  I first leveraged ljsyscall back when I was doing LuaJIT on the Raspberry Pi.  Back then it was getting some ioctl calls right so I could ready from joysticks, keyboards, and mice.  ljsyscall had all that covered.  Why it’s so much more special, and cool for lazy programmers such as myself, is it covers a wide swath of system programming.  Whereas libc tries to make a standard set of library function available to multiple environments, ljsyscall tries to make those multiple environments look relative the same.  For example, it provides a programming interface that is the same across Linux, FreeBsd, rum kernels, and other forms of unices that are similar, but each with their own tribal knowledge.  Quite a feat I’d say, and something I can really appreciate.  For me, ljsyscall has become the face of ‘UNIX’, at least when you’re programming using LuaJIT.

And so it goes.  It’s the small and innocuous which will trip you up time and time again.  Getting the smallest details right at the very start, sussing out that tribal knowledge, checking those facts again and again, are what will keep the simple things simple, and allow you to build cathedrals on granite rather than sand.


Isn’t the whole system just a database? – libdrm

Do enough programming, and everything looks like a database of one form or another. Case in point, when you want to get keyboard and mouse input, you first have to query the system to see which of the /dev/input/eventxxx devices you want to open for your particular needs. Yes, there are convenient shortcuts, but that’s beside the point.

Same goes with other devices in the system. This time around, I want to find the drm device which represents the graphics card in my system (from the libdrm perspective).

In LJIT2libudev there are already objects that make it convenient to enumerate all the devices in the system using a simple iterator:

local ctxt, err = require("UDVContext")()
for _, dev in ctxt:devices() do
    print(dev)
end

Well, that’s find and all, but let’s get specific. To use the libdrm library, I very specifically need one of the active devices in the ‘drm’ subsystem. I could write this:

local ctxt, err = require("UDVContext")()

local function getActiveDrm()
  local function isActiveDrm(dev)
    if dev.IsInitialized and dev:getProperty("subsystem") == "drm" then
      return true;
    end

    return false;
  end

  for _, dev in ctxt:devices() do
    if isActiveDrm(dev) then
      return dev;
    end
  end

  return nil;
end

local device = getActiveDrm()

Yah, that would work. Then of course, when I want to change the criteria for finding the device I’m looking for, I would change up this code a bit. The core iterator is the key starting point at least. The ‘isActiveDrm()’ is a function which acts as a predicate to filter through the results, and only return the ones I want.

Since this is Lua though, and since there is a well throught out functional programming library already (luafun), this could be made even easier:

local function isActiveDrm(dev)
  if dev.IsInitialized and dev:getProperty("subsystem") == "drm" then
    return true;
  end

  return false;
end

local device = head(filter(isActiveDrm, ctxt:devices()))
assert(device, "could not find active drm device")

In this case, we let the luafun ‘filter’ and ‘head’ functions do their job of dealing with the predicate, and taking the first one off the iterator that matches and returning it. Now, changing my criteria is fairly straight forward. Just change out the predicate, and done. This is kind of nice, particularly with Lua, because that predicate is just some code, it could be generated at runtime because we’re in script right?

So, how about this version:

-- File: IsActiveDrmDevice.lua
-- predicate to determine if a device is a DRM device and it's active
return function(dev)
  if dev.IsInitialized and dev:getProperty("subsystem") == "drm" then
    return true;
  end

  return false;
end

-- File: devices_where.lua
#!/usr/bin/env luajit

-- devices_where.lua
-- print devices in the system, filtered by a supplied predicate
-- generates output which is a valid lua table
package.path = package.path..";../?.lua"

local fun = require("fun")()
local utils = require("utils")

local ctxt, err = require("UDVContext")()
assert(ctxt ~= nil, "Error creating context")

if #arg < 1 then
  error("you must specify a predicate")
end

local predicate = require(arg[1])

print("{")
	
each(utils.printDevice, filter(predicate, ctxt:devices()))

print("}")


-- Actual usage from the command line
./devices_where.lua isActiveDrmDevice

In this case, the ‘query’ has been generalized enough such that you can pass a predicate as a filename (minus the ‘.lua’). The code for the predicate will be compiled in, and used as the predicate for the filter() function. Well, that’s pretty nifty I think. And since the query itself again is just a bit of code, that can be changed on the fly as well. I can easily see a system where lua is the query language, and the entire machine is the database.

The tarantool database is written in Lua, and I believe the luafun code is used there. Tarantool is not a system database, but the fact that it’s written in Lua itself is interesting, and just proves the case that Lua is a good language for doing some database work.

I have found that tackling the lowest level enumeration by putting a Lua iterator on top of it makes life a whole lot easier. With many of the libraries that you run across, they spend a fair amount of resources/code on trying to make things look like a database. In the case of libudev, there are functions for iterating their internal hash table of values, routines for creating ‘enumerators’ which are essentially queries, routines for getting properties, routines for turning properties into more accessible strings, routines for turning the ‘FLAGS’ property into individual values, and the like, and then there’s the memory management routines (ref, unref). A lot of that stuff either goes away, or is handled much more succinctly when you’re using a language such as Lua, or JavaScript, or Python, Ruby, whatever, as long as it’s modern, dynamic, and has decent enough higher level memory managed libraries.

And thus, the whole system, from log files, to perf counters, to device lists, is a database, waiting to be harvested, and made readily available.


Dyanmic Programming in C

Certainly there must be a picture?
test_keyboard

This is a simple test app using the graphicc drawproc library.  The app is pretty simple.  It displays a background image, and as you drag the mouse over the image, the location and size of that semi-translucent square is displayed at the bottom of the window on that status line.
Here is the entirety of the program

/*
test_keyboard

Do some simple mouse and keyboard tracking

*/
#include "drawproc.h"
#include "guistyle.h"
#include &lt;stdio.h&gt;

static GUIStyle styler;
static const int gMaxMode = 3;
static int gMode = 0;

pb_rgba fb;
pb_rect keyRect = { 0, 0, 34, 34 };

void  mousePressed()
{
	gMode++;
	if (gMode &gt;= gMaxMode) {
		gMode = 0;
	}
}

void  mouseMoved()
{
	keyRect.x = mouseX - keyRect.width / 2;
	keyRect.y = mouseY - keyRect.height / 2;
}

void  keyReleased()
{
	switch (keyCode)
	{
	case VK_SPACE:
		write_PPM_binary("test_keyboard.ppm", gpb);
		break;

	case VK_RIGHT: // increase width of keyRect
		keyRect.width += 1;
		break;

	case VK_LEFT:
		keyRect.width -= 1;
		if (keyRect.width &lt; 4) keyRect.width = 4;
		break;
	case VK_UP:
		keyRect.height += 1;
		break;
	case VK_DOWN:
		keyRect.height -= 1;
		if (keyRect.height &lt; 4) keyRect.height = 4;
		break;
	}

	keyRect.x = mouseX - keyRect.width / 2;
	keyRect.y = mouseY - keyRect.height / 2;

}

// Draw information about the mouse
// location, buttons pressed, etc
void drawMouseInfo()
{
	// draw a white banner across the bottom
	noStroke();
	fill(pWhite);
	rect(0, fb.frame.height + 2, width, 24);


	// draw the key rectangle
	fill(RGBA(255, 238, 200, 180));
	stroke(pDarkGray);
	rect(keyRect.x, keyRect.y, keyRect.width, keyRect.height);

	// select verdana font
	setFont(verdana17);
	char infobuff[256];
	sprintf_s(infobuff, "Mouse X: %3d Y: %3d    Key: (%3f, %3f)(%3.0f, %3.0f)", mouseX, mouseY, keyRect.x, 
		keyRect.y, keyRect.width, keyRect.height);
	fill(pBlack);
	textAlign(TX_LEFT, TX_TOP);
	text(infobuff, 0, fb.frame.height + 2);

}

void draw()
{
	background(pLightGray);
	backgroundImage(&fb);

	drawMouseInfo();
}

void setup()
{
	int ret = PPM_read_binary("c:/repos/graphicc/Test/windows-keyboard-60-keys.ppm", &fb);

	size(fb.frame.width+4, fb.frame.height+4+30);
	background(pLightGray);
}



Of particular note, and what this article is about, are the routines to track the mouse and keyboard actions. Let’s take the mouse movement first of all.

void  mousePressed()
{
	gMode++;
	if (gMode &gt;= gMaxMode) {
		gMode = 0;
	}
}

void  mouseMoved()
{
	keyRect.x = mouseX - keyRect.width / 2;
	keyRect.y = mouseY - keyRect.height / 2;
}

Basically, every time the mouse moves, the location of the ‘keyRect’ is updated, to reflect the new mouse position. In this case, we have the reported mouse position at the center of that translucent square.

Alright, but who’s calling these routines in the first place? For that we look at the mouse handling within drawproc.cpp


LRESULT CALLBACK mouseHandler(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
	switch (message)
	{
		case WM_MOUSEWHEEL:
			if (gmouseOnWheelHandler != nullptr) {
				gmouseOnWheelHandler(); // hWnd, message, wParam, lParam);
			}
		break;


		case WM_MOUSEMOVE:
			mouseX = GET_X_LPARAM(lParam);
			mouseY = GET_Y_LPARAM(lParam);

			if (isMousePressed) {
				if (gmouseOnDraggedHandler != nullptr) {
					gmouseOnDraggedHandler();
				}
			} else if (gmouseOnMovedHandler != nullptr) {
				gmouseOnMovedHandler();
			}
		break;

		case WM_LBUTTONDOWN:
		case WM_MBUTTONDOWN:
		case WM_RBUTTONDOWN:
			isMousePressed = true;
			mouseButton = wParam;

			if (gOnMousePressedHandler != nullptr) {
				gOnMousePressedHandler();
			}
		break;

		case WM_LBUTTONUP:
		case WM_MBUTTONUP:
		case WM_RBUTTONUP:
			isMousePressed = false;

			if (gmouseReleasedHandler != nullptr) {
				gmouseReleasedHandler();
			}
		break;

		default:
			return DefWindowProc(hWnd, message, wParam, lParam);
	}

	return 0;
}

Good good, looks like a typical “Windows” mouse handling routine. But, what of these various global variables?

		case WM_MOUSEMOVE:
			mouseX = GET_X_LPARAM(lParam);
			mouseY = GET_Y_LPARAM(lParam);

			if (isMousePressed) {
				if (gmouseOnDraggedHandler != nullptr) {
					gmouseOnDraggedHandler();
				}
			} else if (gmouseOnMovedHandler != nullptr) {
				gmouseOnMovedHandler();
			}
		break;

In this case, we have mouseX, mouseY, isMousePressed, gmouseOnDraggedHandler , and gmouseOnMovedHandler. That’s a lot of global state. Of particular interest are the two that point to functions, which are gmouseOnDraggedHandler, and gmouseOnMovedHandler. These routines are called whenever there is mouse movement. The one for dragging is called if the additional global variable ‘isMousePressed’ is true.

This is nice because we’re already getting a chance at being ‘dynamic’ because the function calls are not hard coded, which means, those pointers, although they may have started with some value at compile time, can changed dynamically any time during runtime. So, let’s see how they might get set.

Before any actions occur, as part of the initialization for the drawproc routines, there is this series of calls:

void init()
{
	// Setup text
	font_t_init(&gfont, verdana12);
	gTextSize = 10;
	gTextAlignX = TX_LEFT;
	gTextAlignY = TX_TOP;


	
	HMODULE modH = GetModuleHandle(NULL);

	// Keyboard handling routines
	setOnKeyPressedHandler((EventObserverHandler)GetProcAddress(modH, "keyPressed"));
	setOnKeyReleasedHandler((EventObserverHandler)GetProcAddress(modH, "keyReleased"));
	setOnKeyTypedHandler((EventObserverHandler)GetProcAddress(modH, "keyTyped"));

	setKeyboardHandler(keyHandler);


	// Mouse Handling Routines
	setOnMousePressedHandler((EventObserverHandler)GetProcAddress(modH, "mousePressed"));
	setOnMouseReleasedHandler((EventObserverHandler)GetProcAddress(modH, "mouseReleased"));
	setOnMouseMovedHandler((EventObserverHandler)GetProcAddress(modH, "mouseMoved"));
	setOnMouseDraggedHandler((EventObserverHandler)GetProcAddress(modH, "mouseDragged"));

	setMouseHandler(mouseHandler);
}

First of all, ‘GetModuleHandle’ is the way within Windows to get a handle on one of the libraries (.dll file) which may have been loaded into the program. If you pass “NULL” as the parameter, it will get the handle for the currently running program (the .exe file) itself. This is valuable to have because with this handle, you can make the ‘GetProcAddress’ call, with the name of a function, and get a pointer to that function. Once we have that pointer, we can set it on the global variable, and the rest we’ve already seen.

So that’s it, that’s the trick. The typical way of compiling with the drawproc library is to statically link your code with it. The drawproc library comes with a ‘main’, so all you have to do is implement various bits and pieces that might not already be there.

As we saw from the test_keyboard code, mousePressed, and mouseMoved are the only two of the 4 mouse routines that were implemented. How is that possible? Well, the others will be NULL, as they won’t be found in the library, and since they’re NULL, they simply won’t be called.

There is one trick though to making this work. The GetProcAddress will only work if the routines were marked for export when compiled. This is achieved using the following in the drawproc.h header:

#define DPROC_API		__declspec(dllexport)

// IO Event Handlers
DPROC_API void keyPressed();
DPROC_API void keyReleased();
DPROC_API void keyTyped();


DPROC_API void mousePressed();
DPROC_API void mouseMoved();
DPROC_API void mouseDragged();
DPROC_API void mouseReleased();

This will essentially create function declarations for the functions we intend to implement. If your test code includes ‘drawproc.h’, and then you implement any of these routines, they will automatically be ‘exported’ when the whole is compiled. This will make it possible for them to be picked up during the ‘init()’ process, and thereby setting those global variables. This is also true of the ‘setup()’ and ‘draw()’ routines.

This is a great little trick, when you’re linking statically against the library. Another way to do it would be to put your code into a separate .dll, and run a drawproc.exe host program, giving the name of your .dll as a parameter. Then the library could be dynamically loaded, and the same hookup of routines could occur. The advantage of that setup is that you don’t need to create an entirely new .exe file for each of your test programs. Just produce the .dll. The disadvantage is you get stuck in a state when the runtime changes, and your .dll files no longer match.

So, there you have it, some amount of ‘dynamic’ programming in the little graphics engine that could, using ‘C’, which is certainly capable of supporting this pattern of dynamic programming.


LAPHLibs gets a makeover

Quite a while ago (it looks like about 3 years), I create the LAPHLibs repository.  It was an outgrowth of various projects I was doing, and an experiment in open licensing.  The repo is full of routines varying from hash functions to bit banging.  Not a ‘library’ as such, but just a curation of things that are all pure LuaJIT code based.

Well, after I spun it out, I didn’t show it much love.  I made a couple of updates here and there as I found fixes while using the routines in other projects.  Recently though, I found that this is the most linked to project of all my github based projects.  As such, I thought it might be useful to give it a makeover because having all that bad code out there doesn’t really speak well of the Lua language, nor my learnings of it over the past few years.

So, I recently spent a few hours cleaning things up.  Most of the changes are documented in the new CHANGELOG.md file.

If you are one of the ten people who so happens to read this blog, and are a user of bits and pieces from that library, you might want to take a look.

One of the biggest sins that I fixed is the fact that in a lot of cases, I was polluting the global namespace with my functions.  It was inconsistent.  Some functions were global, some local, sometimes even in the same file!  Now everything is local, and there are proper exports.

I also got rid of a few things, like the implementation of strtoul().  The native tonumber() function is much more correct, and deals with all cases I had implemented.

There are a few places where I was doing idiomatic classes, and I cleaned those up by adding proper looking ‘constructors’.

Overall, the set of routines stands a little taller than it did before.  I can’t say when’s the next time I’ll do another overhaul.  I did want to play around a bit more with the bit banging stuff, and perhaps I can add a little bit more from projects I’ve done recently, like the schedlua scheduler and the like.

Bottom line, sometimes it’s actually worth revisiting old code, if for no other reason than to recognize the sins of your past and correct them if possible.


schedlua – async io

And so, finally we come to the point. Thus far, we looked at the simple scheduler, the core signaling, and the predicate and alarm functions built atop that. That’s all great stuff for fairly straight forward apps that need some amount of concurrency. The best part though is when you can do concurrent networking stuff.

Here’s the setup; I want to issue 20 concurrent GET requests to 20 different sites, and get results back. I want the program to halt once all the tasks have been completed.

--test_linux_net.lua
package.path = package.path..";../?.lua"

local ffi = require("ffi")

local Kernel = require("kernel"){exportglobal = true}
local predicate = require("predicate")(Kernel, true)
local AsyncSocket = require("AsyncSocket")

local sites = require("sites");

-- list of tasks
local taskList = {}


local function httpRequest(s, sitename)
	local request = string.format("GET / HTTP/1.1\r\nUser-Agent: schedlua (linux-gnu)\r\nAccept: */*\r\nHost: %s\r\nConnection: close\r\n\r\n", sitename);
	return s:write(request, #request);
end

local function httpResponse(s)
	local BUFSIZ = 512;
	local buffer = ffi.new("char[512+1]");
	local bytesRead = 0
	local err = nil;
	local cumulative = 0

	repeat
		bytesRead, err = s:read(buffer, BUFSIZ);

		if bytesRead then
			cumulative = cumulative + bytesRead;
		else
			print("read, error: ", err)
			break;
		end
	until bytesRead < 1

	return cumulative;
end


local function siteGET(sitename)
	print("siteGET, BEGIN: ", sitename);

	local s = AsyncSocket();

	local success, err = s:connect(sitename, 80);  

	if success then
		httpRequest(s, sitename);
		httpResponse(s);
	else
		print("connect, error: ", err, sitename);
	end

	s:close();

	print("siteGET, FINISHED: ", sitename)
end


local function allProbesFinished()
	for idx, t in ipairs(taskList) do
		if t:getStatus() ~= "dead" then
			return false;
		end
	end

	return true;
end

local function main()
	for count=1,20 do
		table.insert(taskList, Kernel:spawn(siteGET, sites[math.random(#sites)]))
		Kernel:yield();
	end

	when(allProbesFinished, halt);
end

run(main)

Step by step. The httpRequest() function takes a socket, and does the most bare mimimal HTTP GET request, assuming the socket is already connected to the site.

Similarly, the httpResponse() function gets a response back from the server, and reads as much as it can until the socket is closed (because the Connection: close header was sent).

That’s about the most basic of HTTP request/response pairs you can have, ignoring doing any parsing of the returned data.

Alright, so let’s wrap those two up into a function called siteGET(). siteGET(sitename) takes the name of a site, creates a socket, connects it to the site, and then issues the httpRequest(), and then the httpResponse(). Very simple. What I like about this is that the httpRequest(); httpResponse() sequence is executed in serial as far as I’m concerned. I don’t have to be worried about the httpResponse() being issued before the request completes. Furthermore, if I didn’t use a spawn(), I could simply execute the code directly and be none the wiser.

I want to execute these siteGET()s concurrently though, so within main(), I start up 20 of these tasks, and let them go. Then comes the waiting part:

local function allProbesFinished()
	for idx, t in ipairs(taskList) do
		if t:getStatus() ~= "dead" then
			return false;
		end
	end

	return true;
end

	when(allProbesFinished, halt);

Going back to our knowledge of predicates, we know that the ‘when’ function takes a predicate (function that returns true/false), and will execute the second function when the predicate returns true.

OK, so we just need to come up with a predicate which tells us that all the tasks have completed. Easy enough as a list of the tasks is generated when they are spawned. So, we just go through that list and see if any of them are still running. If there is a single one that is still running, the predicate will return false, and ‘halt()’ will not be called. As soon as the last task finished, the predicate will return true, and the halt() function will be called.

Of course, most things in schedlua are convenient compositions of deeper things (with signals being at the core).

Instead of using the ‘when’ function, you could write the code more directly like this:

	while true
		if allProbesFinished() then
			halt();
			break;
		end
		yield();
	end

That doesn’t quite look as nice as just using the when() function I think. Also, you’re sitting in the main() function, which is no big deal as there’s nothing else trying to execute after this, but it just doesn’t seem as clean. Furthermore, the ‘when’ function might have some magic in its implementation, such as a better understanding of the state of tasks, or special knowledge of the scheduler, or who knows what. At any rate, either way essentially implements a barrier, and the technique can be used anywhere you want to perform an action after some set of tasks has completed. The allProbesFinished() function can be generalized to wait on any list of tasks, maybe call it “waitForTasks()” or some such thing.

At any rate, that completes the primitives that are baked into the core schedlua package. Everything from signals, to predicates, alarms, and finally async io. Of course this is Linux, so async io works with any file descriptor, not just network sockets, so file management or device communications in general can be thrown into the mix.

Now that the basics work, it’s a matter of cleaning up, writing more test cases, fixing bugs, reorganizing, and optimizing resource usage a bit. In general though, the constructs are there, and it’s ready to be used for real applications.