Spelunking Linux – what is this auxv thing anyway

While spelunking Linux, trying to find an easier way to do this or that, I ran across this vDSO thing (virtual ELF Dynamic Shared Object). What?

Ok, it’s like this. I was implementing direct calling of syscalls on Linux, and reading up on how the C libraries do things. On Linux, when you wan to talk to the kernel, you typically go through syscalls, or ioctl calls, or netlinks. With syscall, it’s actually a fairly expensive process, switching from userspace to kernel space, issuing and responding to interrupts, etc. In some situations this could be a critical performance hit. So, to make things easier/faster, some of the system calls are implemented in this little ELF package (vDSO). This little elf package (a dynamic link library) is loaded into every application on Linux. Then, the C library can decide to make calls into that library, instead of syscalls, thus saving a lot of overhead and speeding things up. Not all systems have this capability, but many do.

Alright, so how does the C runtime know whether the capability is there or now, and where this little library is, and how to get at functions therewith? In steps our friend auxv. In the GNU C library, there is a lone call:

unsigned long getauxval(unsigned long);

What values can you get out of this thing? Well, the constants can be found in the elf.h file, and look like:

#define AT_PLATFORM 15
#define AT_PAGESZ 6

And about 30 others. How you use each of these depends on the type of the value that you are looking up. For instance, the AT_PLATFORM returns a pointer to a null terminated string. The AT_PAGESZ returns an integer which represents the memory page size of the machine you’re running on.

OK, so what’s the lua version?

static const int AT_NULL = 0;
static const int AT_IGNORE = 1;
static const int AT_EXECFD = 2;
static const int AT_PHDR = 3;
static const int AT_PHENT = 4;
static const int AT_PHNUM = 5;
static const int AT_PAGESZ = 6;
static const int AT_BASE = 7;
static const int AT_FLAGS = 8;
static const int AT_ENTRY = 9;
static const int AT_NOTELF = 10;
static const int AT_UID = 11;
static const int AT_EUID = 12;
static const int AT_GID = 13;
static const int AT_EGID = 14;
static const int AT_CLKTCK = 17;
static const int AT_PLATFORM = 15;
static const int AT_HWCAP = 16;
static const int AT_FPUCW = 18;
static const int AT_DCACHEBSIZE = 19;
static const int AT_ICACHEBSIZE = 20;
static const int AT_UCACHEBSIZE = 21;
static const int AT_IGNOREPPC = 22;
static const int AT_SECURE = 23;
static const int AT_BASE_PLATFORM = 24;
static const int AT_RANDOM = 25;
static const int AT_HWCAP2 = 26;
static const int AT_EXECFN = 31;
static const int AT_SYSINFO = 32;
static const int AT_SYSINFO_EHDR = 33;
static const int AT_L1I_CACHESHAPE = 34;
static const int AT_L1D_CACHESHAPE = 35;
static const int AT_L2_CACHESHAPE = 36;

unsigned long getauxval(unsigned long);

With this, I can then write code that looks like the following:

local function getStringAuxVal(atype)
	local res = libc.getauxval(atype)

	if res == 0 then 
		return false, "type not found"

	local str = ffi.string(ffi.cast("char *", res));
	return str

local function getIntAuxValue(atype)
	local res = libc.getauxval(atype)

	if res == 0 then 
		return false, "type not found"

	return tonumber(res);

local function getPtrAuxValue(atype)
	local res = libc.getauxval(atype)

	if res == 0 then 
		return false, "type not found"

	return ffi.cast("intptr_t", res);

-- convenience functions
local function getExecPath()
	return getStringAuxVal(libc.AT_EXECFN);

local function getPlatform()
	return getStringAuxVal(libc.AT_PLATFORM);

local function getPageSize()
	return getIntAuxValue(libc.AT_PAGESZ);

local function getRandom()
	return getPtrAuxValue(libc.AT_RANDOM);

	Some test cases
print(" Platform: ", getPlatform());
print("Exec Path: ", getExecPath());
print("Page Size: ", getPageSize());
print("   Random: ", getRandom());

And so on and so forth, assuming you have the proper ‘libc’ luajit ffi binding, which gives you access to constants through the ‘libc.’ mechanism.

OK, fine, if I’m a C programmer, and I just want to port some code that’s already doing this sort of thing. By the way, the one value that we’re interested in is: AT_SYSINFO_EHDR. That contains a pointer to the beginning of our vDSO. Then you can call functions directly from there (there’s an API for that).

But, if I’m a lua programmer, I’ve come to expect more out of my environment, largely because I’m lazy and don’t like so much typing. Upon further examination, you can get this information yourself. If you’re hard core, you can look at the top of memory in your program, and map that location to a pointer you can fiddle with directly. Otherwise, you can actually get this information from a ‘file’ on Linux.

Turns out that you can get this info from ‘/proc/self/auxv’, if you’re running this command about your current process (which you most likely are). So, now what can I do with that? Well, the lua way would be the following:

-- auxv_iter.lua
local ffi = require("ffi")
local libc = require("libc")

local E = {}

-- This table maps the constant values for the various
-- AT_* types to their symbolic names.  This table is used
-- to both generate cdefs, as well and hand back symbolic names
-- for the keys.
local auxtbl = {
	[0] =  "AT_NULL";
	[1] =  "AT_IGNORE";
	[2] = "AT_EXECFD";
	[3] = "AT_PHDR";
	[4] = "AT_PHENT";
	[5] = "AT_PHNUM";
	[6] = "AT_PAGESZ";
	[7] = "AT_BASE";
	[8] = "AT_FLAGS";
	[9] = "AT_ENTRY";
	[10] = "AT_NOTELF";
	[11] = "AT_UID";
	[12] = "AT_EUID";
	[13] = "AT_GID";
	[14] = "AT_EGID";
	[17] = "AT_CLKTCK";
	[15] = "AT_PLATFORM";
	[16] = "AT_HWCAP";
	[18] = "AT_FPUCW";
	[22] = "AT_IGNOREPPC";
	[23] = "AT_SECURE";
	[25] = "AT_RANDOM";
	[26] = "AT_HWCAP2";
	[31] = "AT_EXECFN";
	[32] = "AT_SYSINFO";
	[33] = "AT_SYSINFO_EHDR";
	[34] = "AT_L1I_CACHESHAPE";
	[35] = "AT_L1D_CACHESHAPE";
	[36] = "AT_L2_CACHESHAPE";

-- Given a auxv key(type), and the value returned from reading
-- the file, turn the value into a lua specific type.
-- string pointers --> string
-- int values -> number
-- pointer values -> intptr_t

local function auxvaluefortype(atype, value)
	if atype == libc.AT_EXECFN or atype == libc.AT_PLATFORM then
		return ffi.string(ffi.cast("char *", value))

	if atype == libc.AT_UID or atype == libc.AT_EUID or
		atype == libc.AT_GID or atype == libc.AT_EGID or 
		atype == libc.AT_FLAGS or atype == libc.AT_PAGESZ or
		atype == libc.AT_HWCAP or atype == libc.AT_CLKTCK or 
		atype == libc.AT_PHENT or atype == libc.AT_PHNUM then

		return tonumber(value)

	if atype == libc.AT_SECURE then
		if value == 0 then 
			return false
			return true;

	return ffi.cast("intptr_t", value);

-- iterate over the auxv values at the specified path
-- if no path is specified, use '/proc/self/auxv' to get
-- the values for the currently running program
local function auxviterator(path)
	path = path or "/proc/self/auxv"
	local fd = libc.open(path, libc.O_RDONLY);

	local params = {
		fd = fd;
		keybuff = ffi.new("intptr_t[1]");
		valuebuff = ffi.new("intptr_t[1]");
		buffsize = ffi.sizeof(ffi.typeof("intptr_t"));

	local function gen_value(param, state)
		local res1 = libc.read(param.fd, param.keybuff, param.buffsize)
		local res2 = libc.read(param.fd, param.valuebuff, param.buffsize)
		if param.keybuff[0] == 0 then
			return nil;

		local atype = tonumber(param.keybuff[0])
		return state, atype, auxvaluefortype(atype, param.valuebuff[0])

	return gen_value, params, 0


-- generate ffi.cdef calls to turn the symbolic type names
-- into constant integer values
local cdefsGenerated = false;

local function gencdefs()
	for k,v in pairs(auxtbl) do		
		-- since we don't know if this is already defined, we wrap
		-- it in a pcall to catch the error
		pcall(function() ffi.cdef(string.format("static const int %s = %d;", v,k)) end)
	cdefsGenerated = true;

-- get a single value for specified key.  A path can be specified
-- as well (default it '/proc/self/auxv')
-- this is most like the gnuC getauxval() function
local function getOne(key, path)
	-- iterate over the values, looking for the one we want
	for _, atype, value in auxviterator(path) do
		if atype == key then
			return value;

	return nil;

E.gencdefs = gencdefs;
E.keyvaluepairs = auxviterator;	
E.keynames = auxtbl;
E.getOne = getOne;

setmetatable(E, {
	-- we allow the user to specify one of the symbolic constants
	-- when doing a 'getOne()'.  This indexing allows for the creation
	-- and use of those constants if they haven't already been specified
	__index = function(self, key)
		if not cdefsGenerated then

		local success, value = pcall(function() return ffi.C[key] end)
		if success then
			rawset(self, key, value);
			return value;

		return nil;


return E

In a nutshell, this is all you need for all the lua based auxv goodness in your life. Here are a couple of examples of usage in action:

local init = require("test_setup")()
local auxv_util = require("auxv_iter")
local apairs = auxv_util.keyvaluepairs;
local keynames = auxv_util.keynames;
local auxvGetOne = auxv_util.getOne;

print("==== Iterate All ====")
local function printAll()
	for _, key, value in apairs(path) do
		io.write(string.format("%20s[%2d] : ", keynames[key], key))

-- print all the entries

-- try to get a specific one
print("==== Get Singles ====")
print(" Platform: ", auxvGetOne(auxv_util.AT_PLATFORM))
print("Page Size: ", auxvGetOne(auxv_util.AT_PAGESZ))

The output from printAll() might look like this:

==== Iterate All ====
     AT_SYSINFO_EHDR[33] : 140721446887424LL
            AT_HWCAP[16] : 3219913727
           AT_PAGESZ[ 6] : 4096
           AT_CLKTCK[17] : 100
             AT_PHDR[ 3] : 4194368LL
            AT_PHENT[ 4] : 56
            AT_PHNUM[ 5] : 10
             AT_BASE[ 7] : 140081410764800LL
            AT_FLAGS[ 8] : 0
            AT_ENTRY[ 9] : 4208720LL
              AT_UID[11] : 1000
             AT_EUID[12] : 1000
              AT_GID[13] : 1000
             AT_EGID[14] : 1000
           AT_SECURE[23] : false
           AT_RANDOM[25] : 140721446787113LL
           AT_EXECFN[31] : /usr/local/bin/luajit
         AT_PLATFORM[15] : x86_64

The printAll() function uses the auxv iteration function, which in turns reads the key/value pairs directly from the /proc/self/auxv file. No need for the GNU C lib function at all. It goes further and turns the raw ‘unsigned long’ values into the appropriate data type based on what kind of data the key specified represents. So, you get lua string, and not just a pointer to a C string.

In the second example, getting singles, the output is simply this:

 Platform: 	x86_64
Page Size: 	4096

The code for this goes into a bit of trickery that’s possible with luajit. first of all, notice the use of ‘auxv_util.AT_PAGESZ’. There is nothing in the auxv_iter.lua file that supports this value directly. There is the table of names, and then there’s that ‘setmetatable’ at the end of things. Here’s where the trickery happens. Basically, this function is called whenever you put a ‘.’ after a table to try and access something, and that something isn’t in the table. You get a chance to make something up and return it. Ini this case, we first call ‘gencdefs()’ if that hasn’t already been called. This will generate ‘static const int XXX’ for all the values in the table of names, so that we can then do a lookup of the values in the ‘ffi.C.’ namespace, using the name. If we find a name, then we add it to the table, so next time the lookup will succeed, and we won’t end up calling the __index function.

At any rate, we now have the requisite value to lookup. Then we just roll through the iterator, and return when we’ve got the value we were looking for. The conversion to the appropriate lua type is automatic.

And there you have it! From relative obscurity, to complete usability, in one iterator. Being able to actually get function pointers in the vDSO is the next step. That will require another API wrapper, or worst case, and all encompassing ELF parser…

cUrling up to the net – a LuaJIT binding

I have need to connect to some Azure services from Linux, using C++. There are a few C/C++ libraries around that will make life relatively easy in this regard, but I thought I’d go with an option that has me learn about something I don’t use that often, but will be fairly powerful and complete.  I chose to use cURL, or libcurl.so to be more precise.  Why?  cURL has been around for ages, has continued to evolve, and makes it fairly easy to do anything from ftp down/upload to https connections.  It has tons of options and knobs, including dealing with authentication, SSL, or just acting like an ordinary socket of you prefer that.

First I created a luajit binding to libcurl.  This just follows my usual learning pattern.  In order to conquer an API, you must first render it useful from Lua.  I did my typical two part binding, first a fairly faithful low level binding, then added some luajit idioms atop.  In this particular case, there’s not a big database to query, although there are quite a few options that get listed in a table:



This is an excerpt from the original curl.h header file. There is a listing of some 200 options which can be set on a curl connection (depending on the context of the connection). This CINIT is a macro that sets the value of an enum appropriately. Well, those macros don’t work in the luajit ffi.cdef call, so I needed a way to convert these. I could have just run the lot through the gcc pre-processor, and that would give me the values I needed, but I thought I’d take a different approach.

I wrote a bit of script to scan the curl.h file looking for the CINIT lines, and turn them into something interesting.

function startswith(s, prefix)
    return string.find(s, prefix, 1, true) == 1

local function writeClean(filename)
	for line in io.lines(filename) do
		if startswith(line, "CINIT") then
			name, tp, num = line:match("CINIT%((%g+),%s*(%g+),%s*(%d+)")
			print(string.format("\t%-25s = {'%s', %s},", name, tp, num))


	COOKIE                    = {'OBJECTPOINT', 22},
	HTTPHEADER                = {'OBJECTPOINT', 23},
	HTTPPOST                  = {'OBJECTPOINT', 24},
	SSLCERT                   = {'OBJECTPOINT', 25},
	KEYPASSWD                 = {'OBJECTPOINT', 26},
	CRLF                      = {'LONG', 27},

Well, that’s nice. Now I have it as a useful table. I can write another script to turn that inti enums, or anything else.

local ffi = require("ffi")

local filename = arg[1] or "CurlOptions"

local options = require(filename)

typedef enum {
	CURLOPTTYPE_LONG          = 0,
	CURLOPTTYPE_OFF_T         = 30000

local function CINIT(na,t,nu) 
	return string.format("\tCURLOPT_%s = CURLOPTTYPE_%s+%d,", na, t, nu)

local tbl = {}
local function addenum(name, type, number)
	table.insert(tbl, CINIT(name, type, number));

table.insert(tbl, "local ffi = require('ffi')");
table.insert(tbl, "ffi.cdef[[\ntypedef enum {")

for k,v in pairs(options) do
	addenum(k, v[1], v[2]);

table.insert(tbl, "} CURLoption;]]");

local tblstr = table.concat(tbl,'\n')
-- now get the definitions as a giant string
-- and execute it
local defs = loadstring(tblstr);

It’s actually easier than this. I threw in the loadstring just to ensure the output was valid. Yah, ok basics.

libcurl is an extremely convenient library. As such, it can be a challenge to use. Fortunately, it has an ‘easy’ interface as well. Here, I chose to wrap the easy interface in an object like wrapper to make it even easier. Here’s how you use it:


That will retrieve the entirety of any given url, and dump the output to stdout. Well, that’s somewhat useful, and its only one line of code. This simple interface will become more sophisticated over time, including being the basis for REST calls, but for now, it suffices.

So, there you have it. libcurl seems to be a viable choice for web access from within lua. You could of corse just do it all with pure lua code, but, you probably wont be wrong to leverage libcurl.

Experiences of a Confessed LuaJIT Binder – The all in one

It seems I’ve been writing binders, wrappers, frameworks, and the like for the entirety of my 35 years of programming.  Why is that?  Well, because the tools I use to program are often times not the same as those used to create various libraries, frameworks, and the like.  Sometimes it’s just that there is no universal bond between all those disparate libraries that you want to use, so you end up writing some wrappers, to make things look and behave similar.

I’ve written quite a lot about LuaJIT bindings, and produced rather a lot of them myself.  Most recently, I was writing a binding for the D-Bus library on Linux.  D-Bus is a communications protocol and mechanism meant to support Interprocess Communications.  As such, it’s about low latency, speed on read and write, and relative simplicity.  You can do everything from pop-up and alert, to logging commands in a persistent log.

Here I want to show an approach that I’ve slowly grown and evolved over the past few years.  This approach to writing bindings is written with a few constraints and aspirations in mind:

  • Provide a low level interface that is as true to the underlying framework as possible.
  • Porting typical C code to the wrapping should be straight forward and obvious
  • Provide an interface that supports and leverages Lua idioms and practices
  • Provide a wrapper that does not require a separate compilation step

That list of constraints is brief, but can cause enough trouble depending on how seriously you take each one of them.  So, let’s get to it.

Here I will use the LJIT2dbus project, because it gives a chance to exhibit all of the constraints listed above.  Here’s a bit of code:


local ffi = require("ffi")

if not DBUS_INCLUDED then

local C = {}

require ("dbus-arch-deps");
require ("dbus-address");
require ("dbus-bus");
require ("dbus-connection");
require ("dbus-errors");
require ("dbus-macros");
require ("dbus-message");
require ("dbus-misc");
require ("dbus-pending-call");
require ("dbus-protocol");
require ("dbus-server");
require ("dbus-shared");
require ("dbus-signature");
require ("dbus-syntax");
require ("dbus-threads");
require ("dbus-types");

C.TRUE = 1;
C.FALSE = 0;

Yah, ok, not so exciting, just a bunch of ‘require()’ statements pulling in other modules. What’s in one of these modules?

-- dbus-misc.lua

local ffi = require("ffi")


char*       dbus_get_local_machine_id  (void);

void        dbus_get_version           (int *major_version_p,
                                        int *minor_version_p,
                                        int *micro_version_p);

Yes, again, more require() statements. But then, there are those couple of C functions that are defined. The other files have similar stuff in them. This is the basis of the constraint related to making the code familiar to a C programmer.

Let’s look at another file which might turn out to be more illustrative:

local ffi = require("ffi")


/** Mostly-opaque type representing an error that occurred */
typedef struct DBusError DBusError;

 * Object representing an exception.
struct DBusError
  const char *name;    /**< public error name field */
  const char *message; /**< public error message field */

  unsigned int dummy1 : 1; /**< placeholder */
  unsigned int dummy2 : 1; /**< placeholder */
  unsigned int dummy3 : 1; /**< placeholder */
  unsigned int dummy4 : 1; /**< placeholder */
  unsigned int dummy5 : 1; /**< placeholder */

  void *padding1; /**< placeholder */

local function DBUS_ERROR_INIT()
  return ffi.new("struct DBusError", { NULL, NULL, TRUE, 0, 0, 0, 0, NULL });

void        dbus_error_init      (DBusError       *error);
void        dbus_error_free      (DBusError       *error);
void        dbus_set_error       (DBusError       *error,
                                  const char      *name,
                                  const char      *message,
void        dbus_set_error_const (DBusError       *error,
                                  const char      *name,
                                  const char      *message);
void        dbus_move_error      (DBusError       *src,
                                  DBusError       *dest);
dbus_bool_t dbus_error_has_name  (const DBusError *error,
                                  const char      *name);
dbus_bool_t dbus_error_is_set    (const DBusError *error);

So more require() statements, a data structure, and some C functions. And how to use it? For that, let’s look at the bottom of the dbus.lua file.


local Lib_dbus = ffi.load("dbus-1")

local exports = {
	Lib_dbus = Lib_dbus;
setmetatable(exports, {
	__index = function(self, key)
		local value = nil;
		local success = false;

		-- try looking in table of constants
		value = C[key]
		if value then
			rawset(self, key, value)
			return value;

		-- try looking in the library for a function
		success, value = pcall(function() return Lib_dbus[key] end)
		if success then
			rawset(self, key, value);
			return value;

		-- try looking in the ffi.C namespace, for constants
		-- and enums
		success, value = pcall(function() return ffi.C[key] end)
		--print("looking for constant/enum: ", key, success, value)
		if success then
			rawset(self, key, value);
			return value;

		-- Or maybe it's a type
		success, value = pcall(function() return ffi.typeof(key) end)
		if success then
			rawset(self, key, value);
			return value;

		return nil;



And just for reference, a typical usage of the same:

local dbus = require("dbus")
  local err = dbus.DBusError();
  local bus = dbus.dbus_bus_get(dbus.DBUS_BUS_SESSION, err);

  if (dbus.dbus_bus_name_has_owner(bus, SYSNOTE_NAME, err) == 0) then
    io.stderr:write("Name has no owner on the bus!\n");
    return EXIT_FAILURE;

First case is the creation of the ‘local err’ value.

  local err = dbus.DBusError();

The dbus.lua file does not have a function called ‘DBusError()’. All it has done is load a bunch of type and function declarations, to be used by the LuaJIT ffi mechanism. So, how do we get a function of that name? It doesn’t even exist in one of the required modules.

The trick here is in the ‘__index’ function of the dbus.lua table. The way the Lua language works, any time you make what looks like an access to the member of a table, if it can’t be found in the table, and if the __index function is implemented, it will get called, with the key passed in as a parameter.

In this case, the ‘__index’ function implements a series of lookups, trying to find the value that is associated with the specified key. First it tries looking in a table of constants. Then it tries looking in the actual library, for a function with the specified name. If it doesn’t find the value as a function, it will try to find it as a constant in the C namespace. This will find any enum, or static const int values that have been defined in ffi.cdef[[]] blocks. Finally, if it doesn’t find the key as one of the constants, it tries to figure out if maybe it’s a type (‘ffi.typeof’). This is in the case of DBusError, this one will succeed, and we’ll get back a type.

As it turns out, the types returned from the LuaJIT ffi can be used as constructors.
So, what we really have is this:

local errType = dbus.DBusError;
local err = errType();

The fact that you can just do it all on one line is a short hand.

This is a great convenience. Also, once the value is found, it is stuck into the dbus table itself, so that the next time it’s used, this long lookup won’t occur. The type is already known, and will just be retrieved from the dbus (exports) table directly.

Well, this works for the other kinds of types as well. For example, the ‘dbus_error_init()’ function is defined in a ffi.cdef[[]] block, and that’s about all. So, when re reference it through dbus.dbus_error_init(), we’re going to look it up in the library, and try to use the function found there. And again, once it’s found, it will be stuffed into the dbus (exports) table, for future reference.

This works out great, in that it’s fairly minimal amount of work to get all interesting things defined in ffi.cdef blocks, then just use this __index lookup trick to actually return the values.

I’ve come to this style because otherwise you end up doing a lot more work trying to make the constants, types, enums, and functions universally accessible from anywhere within your program, without resorting to global values. Of course you can make the __index lookup as complex as you like. You can lazily load in modules, for example.

That’s it for this round. An all in one interface that gives you access to constants, enums, and functions in a ffi.cdef wrapped library.

Spelunking Linux – Yes, the system truly is a database

In this article: Isn’t the whole system just a database? – libdrm, I explored a little bit of the database nature of Linux by using libudev to enumerate and open libdrm devices.  After that, I spent some time bringing up a USB module: LJIT2libusb.  libusb is a useful cross platform library that makes it relatively easy to gain access to the usb functions on multiple platforms.  It can enumerate devices, deal with hot plug notifications, open up, read, write, etc.

At its core, on Linux at least, libusb tries to leverage the uvdev capabilities of the target system, if those capabilities are there.  This means that device enumeration and hot plugging actually use the libuvdev stuff.  In fact, the code for enumerating those usb devices in libusb looks like this:


	udev_enumerate_add_match_subsystem(enumerator, "usb");
	udev_enumerate_add_match_property(enumerator, "DEVTYPE", "usb_device");
	devices = udev_enumerate_get_list_entry(enumerator);

There’s more stuff of course, to turn that into data structures which are appropriate for use within the libusb view of the world. But, here’s the equivalent using LLUI and the previously developed UVDev stuff:

local function isUsbDevice(dev)
	if dev.IsInitialized and dev:getProperty("subsystem") == "usb" and
		dev:getProperty("devtype") == "usb_device" then
		return true;

	return false;

each(print, filter(isUsbDevice, ctxt:devices()))

It’s just illustrative, but it’s fairly simple to understand I think. The ‘ctxt:devices()’ is an iterator over all the devices in the system. The ‘filter’ function is part of the luafun functional programming routines available to Lua. the ‘isUsbDevice’ is a predicate function, which returns ‘true’ when the device in question matches what it believes makes a device a ‘usb’ device. In this case, its the subsystem and dev_type properties which are used.

Being able to easily query devices like this makes life a heck of a lot easier. No funky code polluting my pure application. Just these simple query predicates written in Lua, and I’m all set. So, instead of relying on libusb to enumerate my usb devices, I can just enumerate them directly using uvdev, which is what the library does anyway. Enumeration and hotplug handing is part of the library. The other part is the actual send and receiving of data. For that, the libusb library is still primarily important, as replacing that code will take some time.

Where else can this great query capability be applied? Well, libudev is just a nice wrapper atop sysfs, which is that virtual file system built into Linux for gaining access to device information and control of the same. There’s all sorts of stuff in there. So, let’s say you want to list all the block devices?

local function isBlockDevice(dev)
	if dev.IsInitialized and dev:getProperty("subsystem") == "block" then
		return true;

	return false;

That will get all the devices which are in the subsystem “block”. That includes physical disks, virtual disks, partitions, and the like. If you’re after just the physical ones, then you might use something like this:

local function isPhysicalBlockDevice(dev)
	if dev.IsInitialized and dev:getProperty("subsystem") == "block" and
		dev:getProperty("devtype") == "disk" and
		dev:getProperty("ID_BUS") ~= nil then
		return true;

	return false;

Here, a physical device is indicated by subsystem == ‘block’ and devtype == ‘disk’ and the ‘ID_BUS’ property exists, assuming any physical disk would show up on one of the system’s buses. This won’t catch a SD card though. For that, you’d use the first one, and then look for a property related to being an SD card. Same goes for ‘cd’ vs ramdisk, or whatever. You can make these queries as complex or simple as you want.

Once you have a device, you can simply open it using the “SysName” parameter, handed to an fopen() call.

I find this to be a great way to program. It makes the creation of utilities such as ‘lsblk’ relatively easy. You would just look for all the block devices and their partitions, and put them into a table. Then separately, you would have a display routine, which would consume the table and generate whatever output you want. I find this much better than the typical Linux tools which try to do advanced display using the terminal window. That’s great as far as it goes, but not so great if what you really want is a nice html page generated for some remote viewing.

At any rate, this whole libudev exploration is a great thing. You can list all devices easily, getting every bit of information you care to examine. Since it’s all scriptable, it’s fairly easy to taylor your queries on the fly, looking at, discovering, and the like. I discovered that the thumb print reader in my old laptop was made by Broadcom, and my webcam by 3M? It’s just so much fun.

Well there you have it. The more you spelunk, the more you know, and the more you can fiddle about.

Note To Self – Enumerating bit flags

I’ve been trawling through the Linux V4L2 group of libraries of late as part of LLUI.  v4l2 is one of those sprawling libraries that does all things for all people in terms of video on Linux machines.  It’s roughly equivalent to oh so many similar things from the past on the Windows side.  This is one of the libraries you might utilize if you were to get into streaming from your webcam programmatically.  Of course, you could just read from it directly with libusb, but then you lose out on all the nifty format conversions, and I miss this chance to write another pointless reminder for my later coding self.

So, what’s got me so bothered this time around?  Well, lets say I’m just parusing my system, turning everything into a database as I go along.  I’d like to get a hold of my webcam, and see what it’s capable of.  There’s a call for that of course.  Once you make the appropriate IOCtl call, you end up with a struct that looks like this:

struct v4l2_capability {
uint8_t driver[16];
uint8_t card[32];
uint8_t bus_info[32];
uint32_t version;
uint32_t capabilities;
uint32_t device_caps;
uint32_t reserved[3];

The driver, card, and bus_info fields are pretty straight forward as they are simple ‘null terminated’ strings, so you have print them out if you like. It’s that ‘capabilities’ field that gives me fits. This is one of those combined bit flags sort of things. The value can be a combination of any of the numerous ‘capability’ flags, which are these:

-- Values for 'capabilities' field
caps = {
	V4L2_CAP_VIDEO_CAPTURE		= 0x00000001 ; -- Is a video capture device */
	V4L2_CAP_VIDEO_OUTPUT		= 0x00000002; -- Is a video output device */
	V4L2_CAP_VIDEO_OVERLAY		= 0x00000004; -- Can do video overlay */
	V4L2_CAP_VBI_CAPTURE		= 0x00000010; -- Is a raw VBI capture device */
	V4L2_CAP_VBI_OUTPUT			= 0x00000020; -- Is a raw VBI output device */
	V4L2_CAP_SLICED_VBI_CAPTURE	= 0x00000040; -- Is a sliced VBI capture device */
	V4L2_CAP_SLICED_VBI_OUTPUT	= 0x00000080; -- Is a sliced VBI output device */
	V4L2_CAP_RDS_CAPTURE		= 0x00000100; -- RDS data capture */
	V4L2_CAP_VIDEO_OUTPUT_OVERLAY	= 0x00000200; -- Can do video output overlay */
	V4L2_CAP_HW_FREQ_SEEK		= 0x00000400; -- Can do hardware frequency seek  */
	V4L2_CAP_RDS_OUTPUT			= 0x00000800; -- Is an RDS encoder */

	V4L2_CAP_VIDEO_M2M_MPLANE		= 0x00004000;
	V4L2_CAP_VIDEO_M2M				= 0x00008000;

	V4L2_CAP_TUNER			= 0x00010000; -- has a tuner */
	V4L2_CAP_AUDIO			= 0x00020000; -- has audio support */
	V4L2_CAP_RADIO			= 0x00040000; -- is a radio device */
	V4L2_CAP_MODULATOR		= 0x00080000; -- has a modulator */

	V4L2_CAP_READWRITE              = 0x01000000; -- read/write systemcalls */
	V4L2_CAP_ASYNCIO                = 0x02000000; -- async I/O */
	V4L2_CAP_STREAMING              = 0x04000000; -- streaming I/O ioctls */

For the embedded webcam in my laptop, the reported value is: 0x04000001;

Of course, when you’re doing something programmatically, and you just want to check whether a particular flag is set or not, you can just do:

canStream = band(V4L2_CAP_STREAMING, 0x04000001) ~= 0

Very common, and probably some of the most common code you’ll see anywhere. But what else? For various reasons, I want to create the string values for those bit fields, and use those values as keys to tables, or just to print, or to send somewhere, or display, or what have you.

I’ve seen enough ‘C’ code deal with this there is a common patter. First create the #define, or enum statement which encapsulates the values for all the flags. Then, to get the values as strings, create a completely separate string table, which does the mapping of the nice tight enum values and the string values. Then write a little lookup function which can go from the value to the string.

Well, here’s one of those things I love about Lua. In this case, the program IS the database. No need for those parallel representations. Here’s some code:

local pow = math.pow
local bit = require("bit")
local lshift, rshift, band, bor = bit.lshift, bit.rshift, bit.band, bit.bor

local function getValueName(value, tbl)
	for k,v in pairs(tbl) do
		if v == value then
			return k;

	return nil;

local function enumbits(bitsValue, tbl, bitsSize)
	local function name_gen(params, state)

		if state >= params.bitsSize then return nil; end

		while(true) do
			local mask = pow(2,state)
			local maskedValue = band(mask, params.bitsValue)
--print(string.format("(%2d) MASK [%x] - %#x", state, mask, maskedValue))			
			if maskedValue ~= 0 then
				return state + 1, getValueName(maskedValue, params.tbl) or "UNKNOWN"

			state = state + 1;
			if state >= params.bitsSize then return nil; end

		return nil;

	return name_gen, {bitsValue = bitsValue, tbl = tbl, bitsSize = bitsSize or 32}, 0

return enumbits

The function “getValueName()” at the top there simply does a reverse lookup in the table. That is, given a value, return the string that represents that value (is the string key for that value).

Next, the “enumbits()” function is an enumerator. It will iterate over the bit flags, returning a string name for all the ones that are set to ‘1’, and nothing for any of the other bits. Here’s an example:

local bit = require("bit")
local lshift, rshift, band, bor = bit.lshift, bit.rshift, bit.band, bit.bor

local enumbits = require("enumbits")

local testtbl = {
	LOWEST 	= 0x0001;
	MEDIUM 	= 0x0002;
	HIGHEST = 0x0004;
	MIGHTY 	= 0x0008;
	SLUGGO 	= 0x0010;
	MUGGO 	= 0x0020;
	BUGGO 	= 0x0040;
	PUGGO 	= 0x0080;

local caps = {
	V4L2_CAP_VIDEO_CAPTURE		= 0x00000001 ; -- Is a video capture device */
	V4L2_CAP_VIDEO_OUTPUT		= 0x00000002; -- Is a video output device */
	V4L2_CAP_VIDEO_OVERLAY		= 0x00000004; -- Can do video overlay */
	V4L2_CAP_VBI_CAPTURE		= 0x00000010; -- Is a raw VBI capture device */
	V4L2_CAP_VBI_OUTPUT			= 0x00000020; -- Is a raw VBI output device */
	V4L2_CAP_SLICED_VBI_CAPTURE	= 0x00000040; -- Is a sliced VBI capture device */
	V4L2_CAP_SLICED_VBI_OUTPUT	= 0x00000080; -- Is a sliced VBI output device */
	V4L2_CAP_RDS_CAPTURE		= 0x00000100; -- RDS data capture */
	V4L2_CAP_VIDEO_OUTPUT_OVERLAY	= 0x00000200; -- Can do video output overlay */
	V4L2_CAP_HW_FREQ_SEEK		= 0x00000400; -- Can do hardware frequency seek  */
	V4L2_CAP_RDS_OUTPUT			= 0x00000800; -- Is an RDS encoder */

	V4L2_CAP_VIDEO_M2M_MPLANE		= 0x00004000;
	V4L2_CAP_VIDEO_M2M				= 0x00008000;

	V4L2_CAP_TUNER			= 0x00010000; -- has a tuner */
	V4L2_CAP_AUDIO			= 0x00020000; -- has audio support */
	V4L2_CAP_RADIO			= 0x00040000; -- is a radio device */
	V4L2_CAP_MODULATOR		= 0x00080000; -- has a modulator */

	V4L2_CAP_READWRITE              = 0x01000000; -- read/write systemcalls */
	V4L2_CAP_ASYNCIO                = 0x02000000; -- async I/O */
	V4L2_CAP_STREAMING              = 0x04000000; -- streaming I/O ioctls */

local function printBits(bitsValue, tbl)
	tbl = tbl or testtbl
	for _, name in enumbits(bitsValue, tbl) do
		io.write(string.format("%s, ",name))

-- single bits

-- combined bits
printBits(0x04000001, caps)

With that last test case, what you’ll get is the output:


Well that’s handy, particularly when you’re doing some debugging. Just a simple 20 line iterator, and you’re in business, printing flag fields like a boss! That is, if you’re in the lua environment, or any dynamic programming environment that supports iteration of a dictionary.

So, this note to future self is about pointing out the fact that even bitflags are nothing than a very compact form of database. Unpacking them into human readable, programmable form, requires just the right routine, and away you go, you never have to bother with dealing with this little item again. Great for debugging, great for sticking keys in tables, great for displaying on controls!

Isn’t the whole system just a database? – libdrm

Do enough programming, and everything looks like a database of one form or another. Case in point, when you want to get keyboard and mouse input, you first have to query the system to see which of the /dev/input/eventxxx devices you want to open for your particular needs. Yes, there are convenient shortcuts, but that’s beside the point.

Same goes with other devices in the system. This time around, I want to find the drm device which represents the graphics card in my system (from the libdrm perspective).

In LJIT2libudev there are already objects that make it convenient to enumerate all the devices in the system using a simple iterator:

local ctxt, err = require("UDVContext")()
for _, dev in ctxt:devices() do

Well, that’s find and all, but let’s get specific. To use the libdrm library, I very specifically need one of the active devices in the ‘drm’ subsystem. I could write this:

local ctxt, err = require("UDVContext")()

local function getActiveDrm()
  local function isActiveDrm(dev)
    if dev.IsInitialized and dev:getProperty("subsystem") == "drm" then
      return true;

    return false;

  for _, dev in ctxt:devices() do
    if isActiveDrm(dev) then
      return dev;

  return nil;

local device = getActiveDrm()

Yah, that would work. Then of course, when I want to change the criteria for finding the device I’m looking for, I would change up this code a bit. The core iterator is the key starting point at least. The ‘isActiveDrm()’ is a function which acts as a predicate to filter through the results, and only return the ones I want.

Since this is Lua though, and since there is a well throught out functional programming library already (luafun), this could be made even easier:

local function isActiveDrm(dev)
  if dev.IsInitialized and dev:getProperty("subsystem") == "drm" then
    return true;

  return false;

local device = head(filter(isActiveDrm, ctxt:devices()))
assert(device, "could not find active drm device")

In this case, we let the luafun ‘filter’ and ‘head’ functions do their job of dealing with the predicate, and taking the first one off the iterator that matches and returning it. Now, changing my criteria is fairly straight forward. Just change out the predicate, and done. This is kind of nice, particularly with Lua, because that predicate is just some code, it could be generated at runtime because we’re in script right?

So, how about this version:

-- File: IsActiveDrmDevice.lua
-- predicate to determine if a device is a DRM device and it's active
return function(dev)
  if dev.IsInitialized and dev:getProperty("subsystem") == "drm" then
    return true;

  return false;

-- File: devices_where.lua
#!/usr/bin/env luajit

-- devices_where.lua
-- print devices in the system, filtered by a supplied predicate
-- generates output which is a valid lua table
package.path = package.path..";../?.lua"

local fun = require("fun")()
local utils = require("utils")

local ctxt, err = require("UDVContext")()
assert(ctxt ~= nil, "Error creating context")

if #arg < 1 then
  error("you must specify a predicate")

local predicate = require(arg[1])

each(utils.printDevice, filter(predicate, ctxt:devices()))


-- Actual usage from the command line
./devices_where.lua isActiveDrmDevice

In this case, the ‘query’ has been generalized enough such that you can pass a predicate as a filename (minus the ‘.lua’). The code for the predicate will be compiled in, and used as the predicate for the filter() function. Well, that’s pretty nifty I think. And since the query itself again is just a bit of code, that can be changed on the fly as well. I can easily see a system where lua is the query language, and the entire machine is the database.

The tarantool database is written in Lua, and I believe the luafun code is used there. Tarantool is not a system database, but the fact that it’s written in Lua itself is interesting, and just proves the case that Lua is a good language for doing some database work.

I have found that tackling the lowest level enumeration by putting a Lua iterator on top of it makes life a whole lot easier. With many of the libraries that you run across, they spend a fair amount of resources/code on trying to make things look like a database. In the case of libudev, there are functions for iterating their internal hash table of values, routines for creating ‘enumerators’ which are essentially queries, routines for getting properties, routines for turning properties into more accessible strings, routines for turning the ‘FLAGS’ property into individual values, and the like, and then there’s the memory management routines (ref, unref). A lot of that stuff either goes away, or is handled much more succinctly when you’re using a language such as Lua, or JavaScript, or Python, Ruby, whatever, as long as it’s modern, dynamic, and has decent enough higher level memory managed libraries.

And thus, the whole system, from log files, to perf counters, to device lists, is a database, waiting to be harvested, and made readily available.


Get every new post delivered to your Inbox.

Join 55 other followers