Spelunking Linux – Decomposing systemd

Honestly, I don’t know what all the fuss is about. What is systemd?  It’s that bit of code that gets things going on your Linux machine once the kernel has loaded itself.  You know, dealing with bringing up services, communicating between services, running the udev and dbus stuff, etc.

So, I wrote an ffi wrapper for the libsystemd.so library This has proven to be handy, as usual, I can essentially write what looks like standard C code, but it’s actually LuaJIT goodness.

--[[
	Test using SDJournal as a cursor over the journal entries

	In this case, we want to try the various cursor positioning
	operations to ensure the work correctly.
--]]
package.path = package.path..";../src/?.lua"

local SDJournal = require("SDJournal")
local sysd = require("systemd_ffi")

local jnl = SDJournal()

-- move forward a few times
for i=1,10 do
	jnl:next();
end

-- get the cursor label for this position
local label10 = jnl:positionLabel()
print("Label 10: ", label10)

-- seek to the beginning, print that label
jnl:seekHead();
jnl:next();
local label1 = jnl:positionLabel();
print("Label 1: ", label1);


-- seek to label 10 again
jnl:seekLabel(label10)
jnl:next();
local label3 = jnl:positionLabel();
print("Label 3: ", label3)
print("label 3 == label 10: ", label3 == label10)

In this case, a simple journal object which makes it relatively easy to browse through the systemd journals that are laying about. That’s handy. Combined with the luafun functions, browsing through journals suddenly becomes a lot easier, with the full power of lua to form very interesting queries, or other operations.

--[[
	Test cursoring over journal, turning each entry
	into a lua table to be used with luafun filters and whatnot
--]]
package.path = package.path..";../src/?.lua"

local SDJournal = require("SDJournal")
local sysd = require("systemd_ffi")
local fun = require("fun")()

-- Feed this routine a table with the names of the fields
-- you are interested in seeing in the final output table
local function selection(fields, aliases)
	return function(entry)
		local res = {}
		for _, k in ipairs(fields) do
			if entry[k] then
				res[k] = entry[k];
			end
		end
		return res;
	end
end

local function  printTable(entry)
	print(entry)
	each(print, entry)
end

local function convertCursorToTable(cursor)
	return cursor:currentValue();
end


local function printJournalFields(selector, flags)
	flags = flags or 0
	local jnl1 = SDJournal();

	if selector then
		each(printTable, map(selector, map(convertCursorToTable, jnl1:entries())))
	else
		each(printTable, map(convertCursorToTable, jnl1:entries()))	
	end
end

-- print all fields, but filter the kind of journal being looked at
--printJournalFields(nil, sysd.SD_JOURNAL_CURRENT_USER)
--printJournalFields(nil, sysd.SD_JOURNAL_SYSTEM)

-- printing specific fields
--printJournalFields(selection({"_HOSTNAME", "SYSLOG_FACILITY"}));
printJournalFields(selection({"_EXE", "_CMDLINE"}));

-- to print all the fields available per entry
--printJournalFields();

In this case, we have a simple journal printer, which will take a list of fields, as well as a selection of the kinds of journals to look at. That’s quite useful as you can easily generate JSON or XML, or Lua tables on the output end, without much work. You can easily select which fields you want to display, and you could even change the names along the way. You have the full power of lua at your disposal to do whatever you want with the data.

In this case, the SDJournal object is pretty straight forward. It simply wraps the various ‘sd_xxx’ calls within the library to get its work done. What about some other cases? Does the systemd library need to be used for absolutely everything that it does? The answer is ‘no’, you can do a lot of the work yourself, because at the end of the day, the passive part of systemd is just a bunch of file system manipulation.

Here’s where it gets interesting in terms of decomposition.

Within the libsystemd library, there is the sd_get_machine_names() function:

_public_ int sd_get_machine_names(char ***machines) {
        char **l = NULL, **a, **b;
        int r;

        assert_return(machines, -EINVAL);

        r = get_files_in_directory("/run/systemd/machines/", &l);
        if (r < 0)
                return r;

        if (l) {
                r = 0;

                /* Filter out the unit: symlinks */
                for (a = l, b = l; *a; a++) {
                        if (startswith(*a, "unit:") || !machine_name_is_valid(*a))
                                free(*a);
                        else {
                                *b = *a;
                                b++;
                                r++;
                        }
                }

                *b = NULL;
        }

        *machines = l;
        return r;
}

The lua wrapper for this would simply be:

ffi.cdef("int sd_get_machine_names(char ***machines)")

Great, for those who already know this call, you can allocate a char * array, get the array of string values, and party on. But what about the lifetime of those strings, and if you’re doing it as an iterator, when do you ever free stuff, and isn’t this all wasteful?

So, looking at that code in the library, you might think, ‘wait a minute, I could just replicate that in Lua, and get it done without doing any ffi stuff at all!

local fun = require("fun")

local function isNotUnit(name)
	return not strutil.startswith(name, "unit:")
end

function SDMachine.machineNames(self)
	return fun.filter(isNotUnit, fsutil.files_in_directory("/run/systemd/machines/"))
end

OK, that looks simple. But what’s happening with that ‘files_in_directory()’ function? Well, that’s the meat and potatoes of this operation.

local function nil_iter()
    return nil;
end

-- This is a 'generator' which will continue
-- the iteration over files
local function gen_files(dir, state)
    local de = nil

    while true do
       de = libc.readdir(dir)
    
        -- if we've run out of entries, then return nil
        if de == nil then return nil end

    -- check the entry to see if it's an actual file, and not
    -- a directory or link
        if dirutil.dirent_is_file(de) then
            break;
        end
    end

    
    local name = ffi.string(de.d_name);

    return de, name
end

local function files_in_directory(path)
    local dir = libc.opendir(path)

    if not dir==nil then return nil_iter, nil, nil; end

    -- make sure to do the finalizer
    -- for garbage collection
    ffi.gc(dir, libc.closedir);

    return gen_files, dir, initial;
end

In this case, files_in_directory() takes a string path, like “/run/systemd/machines”, and just iterates over the directory, returning only the files found there. It’s convenient in that it will skip so called ‘hidden’ files, and things that are links. This simple technique/function can be the cornerstone of a lot of things that view files in Linux. The function leverages the libc opendir(), readdir(), functions, so there’s nothing new here, but it wraps it all up in this convenient iterator, which is nice.

systemd is about a whole lot more than just browsing directories, but that’s certain a big part of it. When you break it down like this, you begin to realize that you don’t actually need to use a ton of stuff from the library. In fact, it’s probably better and less resource intensive to just ‘go direct’ where it makes sense. In this case, it was just implementing a few choice routines to make file iteration work the same as it does in systemd. As this binding evolves, I’m sure there is other low lying fruit that I’ll be able to pluck to make it even more interesting, useful, and independent of the libsystemd library.


Experiences of a Confessed LuaJIT Binder – The all in one

It seems I’ve been writing binders, wrappers, frameworks, and the like for the entirety of my 35 years of programming.  Why is that?  Well, because the tools I use to program are often times not the same as those used to create various libraries, frameworks, and the like.  Sometimes it’s just that there is no universal bond between all those disparate libraries that you want to use, so you end up writing some wrappers, to make things look and behave similar.

I’ve written quite a lot about LuaJIT bindings, and produced rather a lot of them myself.  Most recently, I was writing a binding for the D-Bus library on Linux.  D-Bus is a communications protocol and mechanism meant to support Interprocess Communications.  As such, it’s about low latency, speed on read and write, and relative simplicity.  You can do everything from pop-up and alert, to logging commands in a persistent log.

Here I want to show an approach that I’ve slowly grown and evolved over the past few years.  This approach to writing bindings is written with a few constraints and aspirations in mind:

  • Provide a low level interface that is as true to the underlying framework as possible.
  • Porting typical C code to the wrapping should be straight forward and obvious
  • Provide an interface that supports and leverages Lua idioms and practices
  • Provide a wrapper that does not require a separate compilation step

That list of constraints is brief, but can cause enough trouble depending on how seriously you take each one of them.  So, let’s get to it.

Here I will use the LJIT2dbus project, because it gives a chance to exhibit all of the constraints listed above.  Here’s a bit of code:

 

--dbus.lua
local ffi = require("ffi")

if not DBUS_INCLUDED then

local C = {}

require ("dbus-arch-deps");
require ("dbus-address");
require ("dbus-bus");
require ("dbus-connection");
require ("dbus-errors");
require ("dbus-macros");
require ("dbus-message");
require ("dbus-misc");
require ("dbus-pending-call");
require ("dbus-protocol");
require ("dbus-server");
require ("dbus-shared");
require ("dbus-signature");
require ("dbus-syntax");
require ("dbus-threads");
require ("dbus-types");

C.TRUE = 1;
C.FALSE = 0;

Yah, ok, not so exciting, just a bunch of ‘require()’ statements pulling in other modules. What’s in one of these modules?

-- dbus-misc.lua

local ffi = require("ffi")

require("dbus-types")
require("dbus-errors")

ffi.cdef[[
char*       dbus_get_local_machine_id  (void);

void        dbus_get_version           (int *major_version_p,
                                        int *minor_version_p,
                                        int *micro_version_p);
]]

Yes, again, more require() statements. But then, there are those couple of C functions that are defined. The other files have similar stuff in them. This is the basis of the constraint related to making the code familiar to a C programmer.

Let’s look at another file which might turn out to be more illustrative:

local ffi = require("ffi")


require("dbus-macros")
require("dbus-types")
require("dbus-protocol")




ffi.cdef[[
/** Mostly-opaque type representing an error that occurred */
typedef struct DBusError DBusError;
]]

ffi.cdef[[
/**
 * Object representing an exception.
 */
struct DBusError
{
  const char *name;    /**< public error name field */
  const char *message; /**< public error message field */

  unsigned int dummy1 : 1; /**< placeholder */
  unsigned int dummy2 : 1; /**< placeholder */
  unsigned int dummy3 : 1; /**< placeholder */
  unsigned int dummy4 : 1; /**< placeholder */
  unsigned int dummy5 : 1; /**< placeholder */

  void *padding1; /**< placeholder */
};
]]


local function DBUS_ERROR_INIT()
  return ffi.new("struct DBusError", { NULL, NULL, TRUE, 0, 0, 0, 0, NULL });
end


ffi.cdef[[
void        dbus_error_init      (DBusError       *error);
void        dbus_error_free      (DBusError       *error);
void        dbus_set_error       (DBusError       *error,
                                  const char      *name,
                                  const char      *message,
                                  ...);
void        dbus_set_error_const (DBusError       *error,
                                  const char      *name,
                                  const char      *message);
void        dbus_move_error      (DBusError       *src,
                                  DBusError       *dest);
dbus_bool_t dbus_error_has_name  (const DBusError *error,
                                  const char      *name);
dbus_bool_t dbus_error_is_set    (const DBusError *error);
]]

So more require() statements, a data structure, and some C functions. And how to use it? For that, let’s look at the bottom of the dbus.lua file.

 

local Lib_dbus = ffi.load("dbus-1")


local exports = {
	Lib_dbus = Lib_dbus;
}
setmetatable(exports, {
	__index = function(self, key)
		local value = nil;
		local success = false;

		-- try looking in table of constants
		value = C[key]
		if value then
			rawset(self, key, value)
			return value;
		end


		-- try looking in the library for a function
		success, value = pcall(function() return Lib_dbus[key] end)
		if success then
			rawset(self, key, value);
			return value;
		end

		-- try looking in the ffi.C namespace, for constants
		-- and enums
		success, value = pcall(function() return ffi.C[key] end)
		--print("looking for constant/enum: ", key, success, value)
		if success then
			rawset(self, key, value);
			return value;
		end

		-- Or maybe it's a type
		success, value = pcall(function() return ffi.typeof(key) end)
		if success then
			rawset(self, key, value);
			return value;
		end

		return nil;
	end,
})

DBUS_INCLUDED = exports
end

return DBUS_INCLUDED

And just for reference, a typical usage of the same:

local dbus = require("dbus")
  local err = dbus.DBusError();
  dbus.dbus_error_init(err);
  local bus = dbus.dbus_bus_get(dbus.DBUS_BUS_SESSION, err);

  if (dbus.dbus_bus_name_has_owner(bus, SYSNOTE_NAME, err) == 0) then
    io.stderr:write("Name has no owner on the bus!\n");
    return EXIT_FAILURE;
  end

First case is the creation of the ‘local err’ value.

  local err = dbus.DBusError();
  dbus.dbus_error_init(err);

The dbus.lua file does not have a function called ‘DBusError()’. All it has done is load a bunch of type and function declarations, to be used by the LuaJIT ffi mechanism. So, how do we get a function of that name? It doesn’t even exist in one of the required modules.

The trick here is in the ‘__index’ function of the dbus.lua table. The way the Lua language works, any time you make what looks like an access to the member of a table, if it can’t be found in the table, and if the __index function is implemented, it will get called, with the key passed in as a parameter.

In this case, the ‘__index’ function implements a series of lookups, trying to find the value that is associated with the specified key. First it tries looking in a table of constants. Then it tries looking in the actual library, for a function with the specified name. If it doesn’t find the value as a function, it will try to find it as a constant in the C namespace. This will find any enum, or static const int values that have been defined in ffi.cdef[[]] blocks. Finally, if it doesn’t find the key as one of the constants, it tries to figure out if maybe it’s a type (‘ffi.typeof’). This is in the case of DBusError, this one will succeed, and we’ll get back a type.

As it turns out, the types returned from the LuaJIT ffi can be used as constructors.
So, what we really have is this:

local errType = dbus.DBusError;
local err = errType();

The fact that you can just do it all on one line is a short hand.

This is a great convenience. Also, once the value is found, it is stuck into the dbus table itself, so that the next time it’s used, this long lookup won’t occur. The type is already known, and will just be retrieved from the dbus (exports) table directly.

Well, this works for the other kinds of types as well. For example, the ‘dbus_error_init()’ function is defined in a ffi.cdef[[]] block, and that’s about all. So, when re reference it through dbus.dbus_error_init(), we’re going to look it up in the library, and try to use the function found there. And again, once it’s found, it will be stuffed into the dbus (exports) table, for future reference.

This works out great, in that it’s fairly minimal amount of work to get all interesting things defined in ffi.cdef blocks, then just use this __index lookup trick to actually return the values.

I’ve come to this style because otherwise you end up doing a lot more work trying to make the constants, types, enums, and functions universally accessible from anywhere within your program, without resorting to global values. Of course you can make the __index lookup as complex as you like. You can lazily load in modules, for example.

That’s it for this round. An all in one interface that gives you access to constants, enums, and functions in a ffi.cdef wrapped library.


Spelunking Linux – Yes, the system truly is a database

In this article: Isn’t the whole system just a database? – libdrm, I explored a little bit of the database nature of Linux by using libudev to enumerate and open libdrm devices.  After that, I spent some time bringing up a USB module: LJIT2libusb.  libusb is a useful cross platform library that makes it relatively easy to gain access to the usb functions on multiple platforms.  It can enumerate devices, deal with hot plug notifications, open up, read, write, etc.

At its core, on Linux at least, libusb tries to leverage the uvdev capabilities of the target system, if those capabilities are there.  This means that device enumeration and hot plugging actually use the libuvdev stuff.  In fact, the code for enumerating those usb devices in libusb looks like this:

 

	udev_enumerate_add_match_subsystem(enumerator, "usb");
	udev_enumerate_add_match_property(enumerator, "DEVTYPE", "usb_device");
	udev_enumerate_scan_devices(enumerator);
	devices = udev_enumerate_get_list_entry(enumerator);

There’s more stuff of course, to turn that into data structures which are appropriate for use within the libusb view of the world. But, here’s the equivalent using LLUI and the previously developed UVDev stuff:

local function isUsbDevice(dev)
	if dev.IsInitialized and dev:getProperty("subsystem") == "usb" and
		dev:getProperty("devtype") == "usb_device" then
		return true;
	end

	return false;
end

each(print, filter(isUsbDevice, ctxt:devices()))

It’s just illustrative, but it’s fairly simple to understand I think. The ‘ctxt:devices()’ is an iterator over all the devices in the system. The ‘filter’ function is part of the luafun functional programming routines available to Lua. the ‘isUsbDevice’ is a predicate function, which returns ‘true’ when the device in question matches what it believes makes a device a ‘usb’ device. In this case, its the subsystem and dev_type properties which are used.

Being able to easily query devices like this makes life a heck of a lot easier. No funky code polluting my pure application. Just these simple query predicates written in Lua, and I’m all set. So, instead of relying on libusb to enumerate my usb devices, I can just enumerate them directly using uvdev, which is what the library does anyway. Enumeration and hotplug handing is part of the library. The other part is the actual send and receiving of data. For that, the libusb library is still primarily important, as replacing that code will take some time.

Where else can this great query capability be applied? Well, libudev is just a nice wrapper atop sysfs, which is that virtual file system built into Linux for gaining access to device information and control of the same. There’s all sorts of stuff in there. So, let’s say you want to list all the block devices?

local function isBlockDevice(dev)
	if dev.IsInitialized and dev:getProperty("subsystem") == "block" then
		return true;
	end

	return false;
end

That will get all the devices which are in the subsystem “block”. That includes physical disks, virtual disks, partitions, and the like. If you’re after just the physical ones, then you might use something like this:

local function isPhysicalBlockDevice(dev)
	if dev.IsInitialized and dev:getProperty("subsystem") == "block" and
		dev:getProperty("devtype") == "disk" and
		dev:getProperty("ID_BUS") ~= nil then
		return true;
	end

	return false;
end

Here, a physical device is indicated by subsystem == ‘block’ and devtype == ‘disk’ and the ‘ID_BUS’ property exists, assuming any physical disk would show up on one of the system’s buses. This won’t catch a SD card though. For that, you’d use the first one, and then look for a property related to being an SD card. Same goes for ‘cd’ vs ramdisk, or whatever. You can make these queries as complex or simple as you want.

Once you have a device, you can simply open it using the “SysName” parameter, handed to an fopen() call.

I find this to be a great way to program. It makes the creation of utilities such as ‘lsblk’ relatively easy. You would just look for all the block devices and their partitions, and put them into a table. Then separately, you would have a display routine, which would consume the table and generate whatever output you want. I find this much better than the typical Linux tools which try to do advanced display using the terminal window. That’s great as far as it goes, but not so great if what you really want is a nice html page generated for some remote viewing.

At any rate, this whole libudev exploration is a great thing. You can list all devices easily, getting every bit of information you care to examine. Since it’s all scriptable, it’s fairly easy to taylor your queries on the fly, looking at, discovering, and the like. I discovered that the thumb print reader in my old laptop was made by Broadcom, and my webcam by 3M? It’s just so much fun.

Well there you have it. The more you spelunk, the more you know, and the more you can fiddle about.


New Baby in the house!!

Of course you’d expect to see a picture here, or on Facebook, or Instagram, or whatever, but we don’t roll like that.

New baby girl arrived 9/30 at 9:30am.  September 28th was apparently a ‘supermoon’, and we’ve been having ‘blood moon’, and the pope was visiting and the president of China was visiting, and…

Seems like an interesting confluence, or near confluence.  I especially like the 9/30, 09:30 thing.  That will make it easier to remember.

Having newborns around means a couple of things.  First and foremost, not a lot of sleep, although this one seems to sleep at least a couple hours here and there.  Seems like we’re constantly up from 10pm to 4 am, although it’s probably not true.  The second thing it means is that a lot of fairly shoddy code gets written.  During those wee hours, between feedings, changes, and general soothing walks around the house, one or the other of the laptops is actually on my lap, and I’m slinging out code.  It’s funny though.  Something that might take 2 hours to figure out during these late night sessions can take a mere minute to realize in the bright light of regular day hours.  Nonetheless, I’m heeding the call of our President and making it possible for everyone to write code!

And so it goes.  A new reason to write inspired code.  There’s someone new to reap the benefits of my labors.


Note To Self – Enumerating bit flags

I’ve been trawling through the Linux V4L2 group of libraries of late as part of LLUI.  v4l2 is one of those sprawling libraries that does all things for all people in terms of video on Linux machines.  It’s roughly equivalent to oh so many similar things from the past on the Windows side.  This is one of the libraries you might utilize if you were to get into streaming from your webcam programmatically.  Of course, you could just read from it directly with libusb, but then you lose out on all the nifty format conversions, and I miss this chance to write another pointless reminder for my later coding self.

So, what’s got me so bothered this time around?  Well, lets say I’m just parusing my system, turning everything into a database as I go along.  I’d like to get a hold of my webcam, and see what it’s capable of.  There’s a call for that of course.  Once you make the appropriate IOCtl call, you end up with a struct that looks like this:

 
[soucecode]
struct v4l2_capability {
uint8_t driver[16];
uint8_t card[32];
uint8_t bus_info[32];
uint32_t version;
uint32_t capabilities;
uint32_t device_caps;
uint32_t reserved[3];
};
[/sourcecode]

The driver, card, and bus_info fields are pretty straight forward as they are simple ‘null terminated’ strings, so you have print them out if you like. It’s that ‘capabilities’ field that gives me fits. This is one of those combined bit flags sort of things. The value can be a combination of any of the numerous ‘capability’ flags, which are these:

-- Values for 'capabilities' field
caps = {
	V4L2_CAP_VIDEO_CAPTURE		= 0x00000001 ; -- Is a video capture device */
	V4L2_CAP_VIDEO_OUTPUT		= 0x00000002; -- Is a video output device */
	V4L2_CAP_VIDEO_OVERLAY		= 0x00000004; -- Can do video overlay */
	V4L2_CAP_VBI_CAPTURE		= 0x00000010; -- Is a raw VBI capture device */
	V4L2_CAP_VBI_OUTPUT			= 0x00000020; -- Is a raw VBI output device */
	V4L2_CAP_SLICED_VBI_CAPTURE	= 0x00000040; -- Is a sliced VBI capture device */
	V4L2_CAP_SLICED_VBI_OUTPUT	= 0x00000080; -- Is a sliced VBI output device */
	V4L2_CAP_RDS_CAPTURE		= 0x00000100; -- RDS data capture */
	V4L2_CAP_VIDEO_OUTPUT_OVERLAY	= 0x00000200; -- Can do video output overlay */
	V4L2_CAP_HW_FREQ_SEEK		= 0x00000400; -- Can do hardware frequency seek  */
	V4L2_CAP_RDS_OUTPUT			= 0x00000800; -- Is an RDS encoder */

	V4L2_CAP_VIDEO_CAPTURE_MPLANE	= 0x00001000;
	V4L2_CAP_VIDEO_OUTPUT_MPLANE	= 0x00002000;
	V4L2_CAP_VIDEO_M2M_MPLANE		= 0x00004000;
	V4L2_CAP_VIDEO_M2M				= 0x00008000;

	V4L2_CAP_TUNER			= 0x00010000; -- has a tuner */
	V4L2_CAP_AUDIO			= 0x00020000; -- has audio support */
	V4L2_CAP_RADIO			= 0x00040000; -- is a radio device */
	V4L2_CAP_MODULATOR		= 0x00080000; -- has a modulator */

	V4L2_CAP_READWRITE              = 0x01000000; -- read/write systemcalls */
	V4L2_CAP_ASYNCIO                = 0x02000000; -- async I/O */
	V4L2_CAP_STREAMING              = 0x04000000; -- streaming I/O ioctls */
}

For the embedded webcam in my laptop, the reported value is: 0x04000001;

Of course, when you’re doing something programmatically, and you just want to check whether a particular flag is set or not, you can just do:

canStream = band(V4L2_CAP_STREAMING, 0x04000001) ~= 0

Very common, and probably some of the most common code you’ll see anywhere. But what else? For various reasons, I want to create the string values for those bit fields, and use those values as keys to tables, or just to print, or to send somewhere, or display, or what have you.

I’ve seen enough ‘C’ code deal with this there is a common patter. First create the #define, or enum statement which encapsulates the values for all the flags. Then, to get the values as strings, create a completely separate string table, which does the mapping of the nice tight enum values and the string values. Then write a little lookup function which can go from the value to the string.

Well, here’s one of those things I love about Lua. In this case, the program IS the database. No need for those parallel representations. Here’s some code:

local pow = math.pow
local bit = require("bit")
local lshift, rshift, band, bor = bit.lshift, bit.rshift, bit.band, bit.bor

local function getValueName(value, tbl)
	for k,v in pairs(tbl) do
		if v == value then
			return k;
		end
	end

	return nil;
end

local function enumbits(bitsValue, tbl, bitsSize)
	local function name_gen(params, state)

		if state >= params.bitsSize then return nil; end

		while(true) do
			local mask = pow(2,state)
			local maskedValue = band(mask, params.bitsValue)
--print(string.format("(%2d) MASK [%x] - %#x", state, mask, maskedValue))			
			if maskedValue ~= 0 then
				return state + 1, getValueName(maskedValue, params.tbl) or "UNKNOWN"
			end

			state = state + 1;
			if state >= params.bitsSize then return nil; end
		end

		return nil;
	end

	return name_gen, {bitsValue = bitsValue, tbl = tbl, bitsSize = bitsSize or 32}, 0
end

return enumbits

The function “getValueName()” at the top there simply does a reverse lookup in the table. That is, given a value, return the string that represents that value (is the string key for that value).

Next, the “enumbits()” function is an enumerator. It will iterate over the bit flags, returning a string name for all the ones that are set to ‘1’, and nothing for any of the other bits. Here’s an example:

local bit = require("bit")
local lshift, rshift, band, bor = bit.lshift, bit.rshift, bit.band, bit.bor

local enumbits = require("enumbits")

local testtbl = {
	LOWEST 	= 0x0001;
	MEDIUM 	= 0x0002;
	HIGHEST = 0x0004;
	MIGHTY 	= 0x0008;
	SLUGGO 	= 0x0010;
	MUGGO 	= 0x0020;
	BUGGO 	= 0x0040;
	PUGGO 	= 0x0080;
}


local caps = {
	V4L2_CAP_VIDEO_CAPTURE		= 0x00000001 ; -- Is a video capture device */
	V4L2_CAP_VIDEO_OUTPUT		= 0x00000002; -- Is a video output device */
	V4L2_CAP_VIDEO_OVERLAY		= 0x00000004; -- Can do video overlay */
	V4L2_CAP_VBI_CAPTURE		= 0x00000010; -- Is a raw VBI capture device */
	V4L2_CAP_VBI_OUTPUT			= 0x00000020; -- Is a raw VBI output device */
	V4L2_CAP_SLICED_VBI_CAPTURE	= 0x00000040; -- Is a sliced VBI capture device */
	V4L2_CAP_SLICED_VBI_OUTPUT	= 0x00000080; -- Is a sliced VBI output device */
	V4L2_CAP_RDS_CAPTURE		= 0x00000100; -- RDS data capture */
	V4L2_CAP_VIDEO_OUTPUT_OVERLAY	= 0x00000200; -- Can do video output overlay */
	V4L2_CAP_HW_FREQ_SEEK		= 0x00000400; -- Can do hardware frequency seek  */
	V4L2_CAP_RDS_OUTPUT			= 0x00000800; -- Is an RDS encoder */

	V4L2_CAP_VIDEO_CAPTURE_MPLANE	= 0x00001000;
	V4L2_CAP_VIDEO_OUTPUT_MPLANE	= 0x00002000;
	V4L2_CAP_VIDEO_M2M_MPLANE		= 0x00004000;
	V4L2_CAP_VIDEO_M2M				= 0x00008000;

	V4L2_CAP_TUNER			= 0x00010000; -- has a tuner */
	V4L2_CAP_AUDIO			= 0x00020000; -- has audio support */
	V4L2_CAP_RADIO			= 0x00040000; -- is a radio device */
	V4L2_CAP_MODULATOR		= 0x00080000; -- has a modulator */

	V4L2_CAP_READWRITE              = 0x01000000; -- read/write systemcalls */
	V4L2_CAP_ASYNCIO                = 0x02000000; -- async I/O */
	V4L2_CAP_STREAMING              = 0x04000000; -- streaming I/O ioctls */
}

local function printBits(bitsValue, tbl)
	tbl = tbl or testtbl
	for _, name in enumbits(bitsValue, tbl) do
		io.write(string.format("%s, ",name))
	end
	print()
end

-- single bits
printBits(lshift(1,0))
printBits(lshift(1,1))
printBits(lshift(1,31))

-- combined bits
printBits(0x0045)
printBits(0x04000001, caps)

With that last test case, what you’ll get is the output:

V4L2_CAP_VIDEO_CAPTURE, V4L2_CAP_STREAMING

Well that’s handy, particularly when you’re doing some debugging. Just a simple 20 line iterator, and you’re in business, printing flag fields like a boss! That is, if you’re in the lua environment, or any dynamic programming environment that supports iteration of a dictionary.

So, this note to future self is about pointing out the fact that even bitflags are nothing than a very compact form of database. Unpacking them into human readable, programmable form, requires just the right routine, and away you go, you never have to bother with dealing with this little item again. Great for debugging, great for sticking keys in tables, great for displaying on controls!


On the insideous deception of simplicity

I find that in programming, there is much tribal knowledge. This is true for every framework, language, OS, blogsphere, and the like. As such, it is the hapless seeker of knowledge who will often fall pray to false assumptions, miscalculations, missteps, and generally an unhappy time of it.

Case in point. A while back, I was working on schedlua, and it came time to add epoll to get some async socket action going.  Well, if you browse the interwebs, looking for things related to epoll, you’ll eventually come across ‘struct epoll_event’.  As I might be dealing with in some ‘binding’ environment, I have need to clearly define what this data structure is.

Here is a typical answer:

struct epoll_event {
    uint32_t events; /* epoll events (bit mask) */
    epoll_data_t data; /* User data */
};

Well that seems easy enough. When I first played with epoll, it was on a Raspberry Pi, and I probably used such a simple definition. I probably copied it from some other code, probably didn’t think much about it at all.

Then, when I did schedlua, things didn’t turn out so well. I used the same simple definition, and somehow I found my data structures were being overwritten, or just had wrong values. What gives? Well, it was actually pointed out to me by someone in the LuaJIT community (Mike Pall) that this particular data structure is defined thus:

typedef union epoll_data {
	void *ptr;
	int fd;
	uint32_t u32;
	uint64_t u64;
} epoll_data_t;

struct epoll_event {
	uint32_t events;
	epoll_data_t data;
}
#ifdef __x86_64__
__attribute__ ((__packed__))
#endif
;

That is, specifically when you’re talking about 64-bit machines, you have to use that packed attribute so that the structure ends up having the same layout as it would have on a 32-bit machine. I guess the Linux kernel itself cares about this alignment for this particular case.

Of course, if I weren’t a lazy programmer, I would have easily seen this if I examined the particulars in the header file it comes from (/usr/include/sys/epoll.h).

The Windows OS is littered with all sorts of this kind of goodness due to its long legacy of compatibility (all the way back to 16-bit computers). It’s just one of those things you have to be aware of.

More recently I’ve been working on the LJIT2libc project. As I wrote in: LJIT2libc – LuaJIT IS “batteries included”, the luajit program, as compared to the liblua51 library, provides access to all of libc, and libm. the only catch is you need the ffi.cdef definitions available to use it.  Well, that’s a heck of a lot of definitions!!  But, I am naive and undaunted by the enormity of the task, so I embarked…

Talk about esoteric tribal knowledge!  Do yourself a favor and browse through the code of one of these ‘libc’ libraries some time.  I used musl as my guide, because it’s modern and fairly well written in my opinion.  At the very least, you can gain an appreciation for the magic and seamless masking of platform specifics this library presents, by browsing the header file structure.  As libc is utilized on multiple different platforms, architectures, OSs, and the like, it has to cover the differences between all of those, and present something that is relatively the same for the programmer who chooses to utilize the library.  My epoll data structures is but one example.  Here’s more esoteric ones:

When you do something as simple as: #include <limits.h> in your C program, what do you get?

On ARM, you get some standard stuff, but the definition of max integer values looks like this:

#if defined(_POSIX_SOURCE) || defined(_POSIX_C_SOURCE) \
 || defined(_XOPEN_SOURCE) || defined(_GNU_SOURCE) || defined(_BSD_SOURCE)
#define PAGE_SIZE 4096
#define LONG_BIT 32
#endif

#define LONG_MAX  0x7fffffffL
#define LLONG_MAX  0x7fffffffffffffffLL

And if you’re dealing with a 64-bit platform, you’re likely to get something like this:

#if defined(_POSIX_SOURCE) || defined(_POSIX_C_SOURCE) \
 || defined(_XOPEN_SOURCE) || defined(_GNU_SOURCE) || defined(_BSD_SOURCE)
#define PAGE_SIZE 4096
#define LONG_BIT 64
#endif

#define LONG_MAX  0x7fffffffffffffffL
#define LLONG_MAX  0x7fffffffffffffffLL

Did you blink? Did you catch that change? LONG_MAX isn’t always LONG_MAX. Same is true for the language specific ‘size_t’. How big are these? What’s their range? It depends.

So, for something as simple as integer values and ranges, that tribal knowledge comes in handy. Of course the well groomed programmer wouldn’t make mistakes related to any assumptions around the range of these values, but the poor lazy slobs, such as myself, who don’t know all the details and assumptions, will make their own assumptions, and the bugs will come… eventually.

Doing LJIT2libc gives me an appreciation for how hard it is to actually create a library such as this to satisfy as broad an audience as it does. If I could only match the headers in detail, then LJIT2libc will be applicable to all platforms where luajit lives. That’s probably a good thing.

Pursuing this path for libc makes me appreciate even more the work of Justin Cormack in creating ljsyscall.  I first leveraged ljsyscall back when I was doing LuaJIT on the Raspberry Pi.  Back then it was getting some ioctl calls right so I could ready from joysticks, keyboards, and mice.  ljsyscall had all that covered.  Why it’s so much more special, and cool for lazy programmers such as myself, is it covers a wide swath of system programming.  Whereas libc tries to make a standard set of library function available to multiple environments, ljsyscall tries to make those multiple environments look relative the same.  For example, it provides a programming interface that is the same across Linux, FreeBsd, rum kernels, and other forms of unices that are similar, but each with their own tribal knowledge.  Quite a feat I’d say, and something I can really appreciate.  For me, ljsyscall has become the face of ‘UNIX’, at least when you’re programming using LuaJIT.

And so it goes.  It’s the small and innocuous which will trip you up time and time again.  Getting the smallest details right at the very start, sussing out that tribal knowledge, checking those facts again and again, are what will keep the simple things simple, and allow you to build cathedrals on granite rather than sand.


Drawproc – Like Processing, but in C++

Triangle Strips using drawproc

trianglestrip

So, what’s the code look like to create this gem?

#include "drawproc.h"

int x;
int y;
float outsideRadius = 150;
float insideRadius = 100;


void setup() {
	size(640, 360);
	background(204);
	x = width / 2;
	y = height / 2;
}


void drawStrip()
{
	int numPoints = int(map(mouseX, 0, width, 6, 60));
	float angle = 0;
	float angleStep = 180.0 / numPoints;

	beginShape(GR_TRIANGLE_STRIP);
	for (int i = 0; i <= numPoints; i++) {
		float px = x + cos(radians(angle)) * outsideRadius;
		float py = y + sin(radians(angle)) * outsideRadius;
		angle += angleStep;
		vertex(px, py);
		px = x + cos(radians(angle)) * insideRadius;
		py = y + sin(radians(angle)) * insideRadius;
		vertex(px, py);
		angle += angleStep;
	}
	endShape();
}

void draw() {
  background(204);
  drawStrip();
}

If you’ve done any coding in Processing, you can look at the example that inspired this bit of code here: Triangle Strip

What’s notable about it is the similarity to the Java or even the JavaScript version (if processing.js). It takes about a 10 minute conversion to go from Processing to using drawproc. So, what is drawproc?

Drawproc is an application and library which facilitates the creation of interactive graphics. It is the culmination of taking the work from graphicc and encapsulating in such a way that makes it easy to use in multiple situations.

So, how does it work?  Basically, there is the drawproc.exe application.  This application contains a main(), and a primary event loop which takes care of capturing mouse and keyboard events, and issuing “draw()” calls.  Previously (Dyanmic Programming in C) I explained how that dynamic bit of machinery works. All that machinery is at work here with the addition of one more dynamic programming item.

bool InitializeInstance(const char *moduleName)
{

	// Get pointers to client setup and loop routines
	clientModule = LoadLibrary(moduleName);

	printf("modH: 0x%p\n", clientModule);

	SetupHandler procAddr = (SetupHandler)GetProcAddress(clientModule, "setup");
	printf("proc Address: 0x%p\n", procAddr);

	if (procAddr != NULL) {
		setSetupRoutine(procAddr);
	}

	LoopHandler loopAddr = (LoopHandler)GetProcAddress(clientModule, "draw");
	printf("loop Addr: 0x%p\n", loopAddr);

	if (loopAddr != NULL) {
		setLoopRoutine(loopAddr);
	}

	if ((procAddr == nullptr) && (loopAddr == nullptr))
	{
		return false;
	}

	gClock = dproc_clock_new();

	return true;
}

When invoking drawproc, you give a name of a module, which is a .dll file compiled against the .exe. Typical execution looks like this:

c:\tools>drawproc trianglestrip.dll

That ‘trianglestrip.dll’ is passed along to the InitializeInstance() call, the module is loaded, and the ‘setup()’ and ‘draw()’ functions are looked for. If neither of them is found, or the .dll doesn’t load, then the program quits. At this point, everything is the same as if you had linked the drawing module into the drawproc.exe program directly. The advantage is you have a simple small (~200K for debug version) executable (drawproc.exe) which is very slow changing. Then you have the modules, which can be numerous, and dynamic. You can create modules independently of the drawproc.exe and run as you wish. You could even write a single module which loads .lua, or any other embedded scripting environment, and write your code using that scripting language instead.

How do you create these modules? Well, you just write your code, make reference to the header files within drawproc, and use drawproc.lib as the library reference. All the relevant symbols within drawproc are exported, so this just works. At the end of the day, the drawproc.exe looks just like any other .dll that might be out there.

In case you’re still reading, here’s another picture.

SineConsoleBanate CAD 2011

This one is interesting because it’s actually an animation (SineConsole).  A few years back, when I was experimenting with BanateCAD, I had done something similar, all in Lua Banate CAD 2011.

Why bother with all this though?  Why C? What’s the point?  I had this interesting conversation last week with a co-worker.  We were discussing whether people who are coming into becoming software programmers would be better served by learning C#, or C/C++.  I think my answer was C#, simply because it seems more in fashion and more applicable to other dynamic languages than does C/C++.  But, here we’re doing a lot of ‘dynamic’ with standard C/C++.  Really the answer to that question is “you will need to learn and use many languages, frameworks, and tools in your software development.  Learning some C will likely serve you well, but be prepared to learn many different things.’

drawproc being written in C/C++ is great because it makes programming graphics fairly simple (because of the Processing mimicry).  Using the Processing API makes the graphics stuff really easy.  At the same time, since it’s written in C/C++, gaining access to the lowest level stuff of the platform is really easy as well.  For example, integrating with the Microsoft Kinect sensor is as easy as just using the Microsoft Provided SDK directly.  No shim, no translation layer, no ‘binding’ to get in the way.  That’s a very good thing.  Also, as time goes on, doing the accelerated this and that, throwing in networking and the like will be a relative no brainer.

So, there you have it.  drawproc is a new standalone tool which can be used for fiddling about with graphics.  For those who are into such things, it’s a nice tool to play with.


Follow

Get every new post delivered to your Inbox.

Join 56 other followers