A Day in the Life of Tech – August 2015

It’s been an interesting past couple of weeks in my tech universe.  Just last week, Mike Pall, long time maintainer of the LuaJIT project, declared he’s looking for new maintainers for the LuaJIT project.  Lots of people from the LuaJIT community stepped forward to throw their hat in the arena to be on the maintainer’s bandwagon.  Yours truly expressed some desire to help out where possible, although I don’t fancy myself as a ‘maintainer’, but rather an occasional contributor.

At roughly the same time, another Michael Neidermayer, maintainer if ffmpeg,  a fairly high profile project, declared his desire to leave his project.

I half expected the commentary around these departures to be on the order of “who is John Galt”, but nope, just a couple of guys moving on to bigger and better things, either due to burn out, disillusionment, or simply because doing the same thing for a long time just starts to lose interest.

The LuaJIT one is interesting as I had a conversation with a coworker just a week previous discussing reluctance to use LuaJIT due to the fact it is maintained by a single entity.  I take Mike’s turning over of the project to ‘the community’ to be a good thing.  More people involved, less likely for the project to disappear.  It won’t quite have the same genius focus that it had before, but at the same time, it will get some new perspectives, and perhaps some new features that are really needed that Mike was neither interested in, or had the time to implement.  We’ll see.  I intend to write up some documentation, and possibly contribute some test cases.

This reminds me that these things are just tools, and not crusades in and of themselves.  I love using LuaJIT because it solves a great many challenges when rapidly prototyping and stitching together disparate systems.  But, I’m not a LuaJIT acolite (counter to appearances).

In another project, I recently purchased the brand spanking new Odroid XU4 from Hardkernel.  I bought the cloudshell as well because it’s a fairly simple effective case that comes complete with a 2.5″ display, and connection for a SATA/SSD drive.  I’ve had interesting debates with colleagues about the usefulness of such modules in the fact of fairly solid computing commodity products.  The most interesting is probably discussing the reality of NAS boxes, which this thing is meant to be able to support the creation of.  The bottom line is, for $79, this is a simple easy compute module, like the Raspberry Pi, although much more powerful.  It could possibly be used as a replacement for many a primary compute device, or could just be a good way to do a little bit of prototyping and experimentation.

The Odroid isn’t necessarily meant to be a finished end user consumer product.  They are solidly single board developer computers, and as such, they are just another tool in the arsenal of a developer.  Need to do a little Linux?  Just turn this thing on, SSH into and there you go.  No need for a VM on some desktop machine, or your Mac, just throw $79 at the problem and carry on.

Ubiquitous computing, and by that I mean cheap and easy and powerful enough, will bring in many possibilities.

On another front, I had the opportunity to fly by 3DR quadcopter a couple of times in the last week.  First was to show my mother and sister.  My mother’s response “Nice toy for rich boys”.  My sister was a bit more enthusiastic with “that’s very cool”.  Non-human occupied aerial devices are simply going to proliferate, because they’re cheap enough, and easy enough to fly either by humans or machines, that it’s an almost no brainer.  Legislation can’t keep up, and by the time it catches up it won’t be able to put the genie back in the bottle.  We’ll no doubt develop automated air traffic control systems for these things, so that aerial delivery of small items can become a reality.

Although I started with the 3DR Iris+, I’m pretty sure my next rig will come from Quad Questions.

The Sparrow racing quad kit they build seems quite interesting.  I now know that to get beyond the standard video clip of a slow motion rise above nice landscapes, you want a FPV rig, preferably built out of carbon fiber, with very little frills beyond that.  Perhaps I’ll save up for this coming Christmas/birthday and see what I can get.  I’m sure I’ll get an assembled machine instead of a kit, because kits have a way of never actually getting completed.

Other than that, the release of the new Tessels is imminent, and Microsoft is making great strides with event hubs on Azure, consuming 10s of millions of iOT like events on a daily basis.  This space is going to get interesting fairly quickly.

And that’s what this day in tech looks like.  More progress on hardware, shifting ownership in software, and something interesting always just around the corner.


Dyanmic Programming in C

Certainly there must be a picture?
test_keyboard

This is a simple test app using the graphicc drawproc library.  The app is pretty simple.  It displays a background image, and as you drag the mouse over the image, the location and size of that semi-translucent square is displayed at the bottom of the window on that status line.
Here is the entirety of the program

/*
test_keyboard

Do some simple mouse and keyboard tracking

*/
#include "drawproc.h"
#include "guistyle.h"
#include <stdio.h>

static GUIStyle styler;
static const int gMaxMode = 3;
static int gMode = 0;

pb_rgba fb;
pb_rect keyRect = { 0, 0, 34, 34 };

void  mousePressed()
{
	gMode++;
	if (gMode >= gMaxMode) {
		gMode = 0;
	}
}

void  mouseMoved()
{
	keyRect.x = mouseX - keyRect.width / 2;
	keyRect.y = mouseY - keyRect.height / 2;
}

void  keyReleased()
{
	switch (keyCode)
	{
	case VK_SPACE:
		write_PPM_binary("test_keyboard.ppm", gpb);
		break;

	case VK_RIGHT: // increase width of keyRect
		keyRect.width += 1;
		break;

	case VK_LEFT:
		keyRect.width -= 1;
		if (keyRect.width < 4) keyRect.width = 4;
		break;
	case VK_UP:
		keyRect.height += 1;
		break;
	case VK_DOWN:
		keyRect.height -= 1;
		if (keyRect.height < 4) keyRect.height = 4;
		break;
	}

	keyRect.x = mouseX - keyRect.width / 2;
	keyRect.y = mouseY - keyRect.height / 2;

}

// Draw information about the mouse
// location, buttons pressed, etc
void drawMouseInfo()
{
	// draw a white banner across the bottom
	noStroke();
	fill(pWhite);
	rect(0, fb.frame.height + 2, width, 24);


	// draw the key rectangle
	fill(RGBA(255, 238, 200, 180));
	stroke(pDarkGray);
	rect(keyRect.x, keyRect.y, keyRect.width, keyRect.height);

	// select verdana font
	setFont(verdana17);
	char infobuff[256];
	sprintf_s(infobuff, "Mouse X: %3d Y: %3d    Key: (%3f, %3f)(%3.0f, %3.0f)", mouseX, mouseY, keyRect.x, 
		keyRect.y, keyRect.width, keyRect.height);
	fill(pBlack);
	textAlign(TX_LEFT, TX_TOP);
	text(infobuff, 0, fb.frame.height + 2);

}

void draw()
{
	background(pLightGray);
	backgroundImage(&fb);

	drawMouseInfo();
}

void setup()
{
	int ret = PPM_read_binary("c:/repos/graphicc/Test/windows-keyboard-60-keys.ppm", &fb);

	size(fb.frame.width+4, fb.frame.height+4+30);
	background(pLightGray);
}



Of particular note, and what this article is about, are the routines to track the mouse and keyboard actions. Let’s take the mouse movement first of all.

void  mousePressed()
{
	gMode++;
	if (gMode >= gMaxMode) {
		gMode = 0;
	}
}

void  mouseMoved()
{
	keyRect.x = mouseX - keyRect.width / 2;
	keyRect.y = mouseY - keyRect.height / 2;
}

Basically, every time the mouse moves, the location of the ‘keyRect’ is updated, to reflect the new mouse position. In this case, we have the reported mouse position at the center of that translucent square.

Alright, but who’s calling these routines in the first place? For that we look at the mouse handling within drawproc.cpp


LRESULT CALLBACK mouseHandler(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
	switch (message)
	{
		case WM_MOUSEWHEEL:
			if (gmouseOnWheelHandler != nullptr) {
				gmouseOnWheelHandler(); // hWnd, message, wParam, lParam);
			}
		break;


		case WM_MOUSEMOVE:
			mouseX = GET_X_LPARAM(lParam);
			mouseY = GET_Y_LPARAM(lParam);

			if (isMousePressed) {
				if (gmouseOnDraggedHandler != nullptr) {
					gmouseOnDraggedHandler();
				}
			} else if (gmouseOnMovedHandler != nullptr) {
				gmouseOnMovedHandler();
			}
		break;

		case WM_LBUTTONDOWN:
		case WM_MBUTTONDOWN:
		case WM_RBUTTONDOWN:
			isMousePressed = true;
			mouseButton = wParam;

			if (gOnMousePressedHandler != nullptr) {
				gOnMousePressedHandler();
			}
		break;

		case WM_LBUTTONUP:
		case WM_MBUTTONUP:
		case WM_RBUTTONUP:
			isMousePressed = false;

			if (gmouseReleasedHandler != nullptr) {
				gmouseReleasedHandler();
			}
		break;

		default:
			return DefWindowProc(hWnd, message, wParam, lParam);
	}

	return 0;
}

Good good, looks like a typical “Windows” mouse handling routine. But, what of these various global variables?

		case WM_MOUSEMOVE:
			mouseX = GET_X_LPARAM(lParam);
			mouseY = GET_Y_LPARAM(lParam);

			if (isMousePressed) {
				if (gmouseOnDraggedHandler != nullptr) {
					gmouseOnDraggedHandler();
				}
			} else if (gmouseOnMovedHandler != nullptr) {
				gmouseOnMovedHandler();
			}
		break;

In this case, we have mouseX, mouseY, isMousePressed, gmouseOnDraggedHandler , and gmouseOnMovedHandler. That’s a lot of global state. Of particular interest are the two that point to functions, which are gmouseOnDraggedHandler, and gmouseOnMovedHandler. These routines are called whenever there is mouse movement. The one for dragging is called if the additional global variable ‘isMousePressed’ is true.

This is nice because we’re already getting a chance at being ‘dynamic’ because the function calls are not hard coded, which means, those pointers, although they may have started with some value at compile time, can changed dynamically any time during runtime. So, let’s see how they might get set.

Before any actions occur, as part of the initialization for the drawproc routines, there is this series of calls:

void init()
{
	// Setup text
	font_t_init(&gfont, verdana12);
	gTextSize = 10;
	gTextAlignX = TX_LEFT;
	gTextAlignY = TX_TOP;


	
	HMODULE modH = GetModuleHandle(NULL);

	// Keyboard handling routines
	setOnKeyPressedHandler((EventObserverHandler)GetProcAddress(modH, "keyPressed"));
	setOnKeyReleasedHandler((EventObserverHandler)GetProcAddress(modH, "keyReleased"));
	setOnKeyTypedHandler((EventObserverHandler)GetProcAddress(modH, "keyTyped"));

	setKeyboardHandler(keyHandler);


	// Mouse Handling Routines
	setOnMousePressedHandler((EventObserverHandler)GetProcAddress(modH, "mousePressed"));
	setOnMouseReleasedHandler((EventObserverHandler)GetProcAddress(modH, "mouseReleased"));
	setOnMouseMovedHandler((EventObserverHandler)GetProcAddress(modH, "mouseMoved"));
	setOnMouseDraggedHandler((EventObserverHandler)GetProcAddress(modH, "mouseDragged"));

	setMouseHandler(mouseHandler);
}

First of all, ‘GetModuleHandle’ is the way within Windows to get a handle on one of the libraries (.dll file) which may have been loaded into the program. If you pass “NULL” as the parameter, it will get the handle for the currently running program (the .exe file) itself. This is valuable to have because with this handle, you can make the ‘GetProcAddress’ call, with the name of a function, and get a pointer to that function. Once we have that pointer, we can set it on the global variable, and the rest we’ve already seen.

So that’s it, that’s the trick. The typical way of compiling with the drawproc library is to statically link your code with it. The drawproc library comes with a ‘main’, so all you have to do is implement various bits and pieces that might not already be there.

As we saw from the test_keyboard code, mousePressed, and mouseMoved are the only two of the 4 mouse routines that were implemented. How is that possible? Well, the others will be NULL, as they won’t be found in the library, and since they’re NULL, they simply won’t be called.

There is one trick though to making this work. The GetProcAddress will only work if the routines were marked for export when compiled. This is achieved using the following in the drawproc.h header:

#define DPROC_API		__declspec(dllexport)

// IO Event Handlers
DPROC_API void keyPressed();
DPROC_API void keyReleased();
DPROC_API void keyTyped();


DPROC_API void mousePressed();
DPROC_API void mouseMoved();
DPROC_API void mouseDragged();
DPROC_API void mouseReleased();

This will essentially create function declarations for the functions we intend to implement. If your test code includes ‘drawproc.h’, and then you implement any of these routines, they will automatically be ‘exported’ when the whole is compiled. This will make it possible for them to be picked up during the ‘init()’ process, and thereby setting those global variables. This is also true of the ‘setup()’ and ‘draw()’ routines.

This is a great little trick, when you’re linking statically against the library. Another way to do it would be to put your code into a separate .dll, and run a drawproc.exe host program, giving the name of your .dll as a parameter. Then the library could be dynamically loaded, and the same hookup of routines could occur. The advantage of that setup is that you don’t need to create an entirely new .exe file for each of your test programs. Just produce the .dll. The disadvantage is you get stuck in a state when the runtime changes, and your .dll files no longer match.

So, there you have it, some amount of ‘dynamic’ programming in the little graphics engine that could, using ‘C’, which is certainly capable of supporting this pattern of dynamic programming.


graphicc – Scalable Fonts

The initial text support in graphicc was based on some fixed sized bitmap fonts.  That’s a good starting point, and good enough for a lot of UI work, and perhaps even some simple graphs.  But it’s a less satisfying answer when you really want to get into the nitty gritty of things.

HellaWerld

Here’s the segoeui.ttf font being displayed in the current graphicc .  Why display in such an unreadable way?  To emphasize that it looks pretty smooth if you don’t look too close…

When it comes to truetype font rendering, the most common goto source is probably the freetype library.  It does a great job, has existed forever, and is used by everyone who doesn’t want to roll their own.  It does great rendering, using the hinting found in the truetype font files and all that.  All that greatness comes at a cost though.  The freetype library is a bit largish for such a small library as graphicc.  So, I went with a different approach.

Here I’m leveraging the stb_trutype.h file found in the public domain stb codes.  The library is good enough to render what you see here, using any of the truetype fonts found on your system.  It won’t do all the fancy things that freetype will, but it at least gives you access to the bulk of the font information, so you can do simple things like kerning.

Whats more, you get access to all the vertex information, so if you want to, you can actually render polygon outlines and the like, and not just take the pre-rendered bitmaps that it offers up.  In fact, one of the strongest parts of these routines is the polygon rendering code.  It makes sense to lift this code up and make it the general polygon rendering code for the entirety of graphicc.  At the moment, there are two or three polygon renderers in the codebase (including samples).  Having one routine which is compact, and capable of dealing with text, and anything else less complex than that, would be a big win for a tiny library such as this.  It will take some work to refactor in such a way that this can become a reality, but it’s probably well worth the effort.

Another benefit of that separation will be the fact that I’ll be able to apply a transform to the vertices for a glyph, and do the cheesy rotated text tricks.
Curious about the code? It’s roughly this

struct ttfset fontset;
struct stbtt_fontinfo finfo;

void setup()
{
	size(1024, 768);

	//char *filename = "c:/windows/fonts/arial.ttf";
	char *filename = "c:/windows/fonts/segoeui.ttf";

	if (!ttfset_open(fontset, filename)) {
		return ;
	}

	finfo = ttfset_font_from_index(fontset, 0);
}

void drawText()
{
	int i, j;
	int ascent, descent, lineGap, baseline;
	float scale, xpos = 2; // leave a little padding in case the character extends left
	float ypos = 2;

	char *text = "Hella Werld!";
	unsigned char *bitmap;

	stbtt_GetFontVMetrics(&finfo, &ascent, &descent, &lineGap);

	for (int pixfactor = 3; pixfactor < 9; pixfactor++) {

		int pixsize = pow((float)2, pixfactor);
		scale = stbtt_ScaleForPixelHeight(&finfo, pixsize);
		baseline = (int)(ascent*scale);

		int idx = 0;
		while (text[idx]) {
			int advance, lsb, x0, y0, x1, y1;
			float x_shift = xpos - (float)floor(xpos);
			stbtt_GetCodepointHMetrics(&finfo, text[idx], &advance, &lsb);
			stbtt_GetCodepointBitmapBoxSubpixel(&finfo, text[idx], scale, scale, x_shift, 0, &x0, &y0, &x1, &y1);

			int w, h;
			bitmap = stbtt_GetCodepointBitmap(&finfo, 0, scale, text[idx], &w, &h, 0, 0);

			raster_rgba_blend_alphamap(gpb, xpos, ypos + baseline + y0, bitmap, w, h, pYellow);

			//printf("%d %d %d", baseline, y0, y1);
			xpos += (advance * scale);
			if (text[idx + 1])
				xpos += scale*stbtt_GetCodepointKernAdvance(&finfo, text[idx], text[idx + 1]);
			idx++;

			stbtt_FreeBitmap(bitmap, NULL);
		}
		xpos = 2;
		ypos += pixsize-(scale*descent);
	}
}

It’s a bit wasteful in that a new glyph bitmap is created for every single character, even if they repeat, so there’s no caching of those bitmaps going on. If this were a real app, I’d be caching those bitmaps whenever I created them for a particular size.

The primary addition to the graphicc core is the rgba_blend_alphamap() function. It takes the 256 value alpha image generated for a glyph, and copies it to the destination, using the specified color, and the bitmap as the alpha value of the color as it does it. This makes for some nice anti-aliasing, which helps make the thing look smooth.

Well, there you have it. The font support is improving, without dragging in the kitchen sink. It may be possible that graphicc is able to stick to a very slender memory footprint and offer serious enough features.


LAPHLibs gets a makeover

Quite a while ago (it looks like about 3 years), I create the LAPHLibs repository.  It was an outgrowth of various projects I was doing, and an experiment in open licensing.  The repo is full of routines varying from hash functions to bit banging.  Not a ‘library’ as such, but just a curation of things that are all pure LuaJIT code based.

Well, after I spun it out, I didn’t show it much love.  I made a couple of updates here and there as I found fixes while using the routines in other projects.  Recently though, I found that this is the most linked to project of all my github based projects.  As such, I thought it might be useful to give it a makeover because having all that bad code out there doesn’t really speak well of the Lua language, nor my learnings of it over the past few years.

So, I recently spent a few hours cleaning things up.  Most of the changes are documented in the new CHANGELOG.md file.

If you are one of the ten people who so happens to read this blog, and are a user of bits and pieces from that library, you might want to take a look.

One of the biggest sins that I fixed is the fact that in a lot of cases, I was polluting the global namespace with my functions.  It was inconsistent.  Some functions were global, some local, sometimes even in the same file!  Now everything is local, and there are proper exports.

I also got rid of a few things, like the implementation of strtoul().  The native tonumber() function is much more correct, and deals with all cases I had implemented.

There are a few places where I was doing idiomatic classes, and I cleaned those up by adding proper looking ‘constructors’.

Overall, the set of routines stands a little taller than it did before.  I can’t say when’s the next time I’ll do another overhaul.  I did want to play around a bit more with the bit banging stuff, and perhaps I can add a little bit more from projects I’ve done recently, like the schedlua scheduler and the like.

Bottom line, sometimes it’s actually worth revisiting old code, if for no other reason than to recognize the sins of your past and correct them if possible.


Accessing Open vSwitch using LuaJIT – The LJIT2ovs project

I’ve recently had the need to learn Open vSwitch to do some virtualized networking stuff. As part of my learning experience, I’ve created the LJIT2ovs project on GitHub.

Open vSwitch (OVS) is a pretty decent bit of kit.  It’s use is to act as that bit of bridge code on a machine that supports some software defined networking.  Essentially, a network hub/switch/bridge on your machine.  My particular use case is to support interesting networking topologies with VMs, virtual lans and the like in the context of a data center.  Having OVS allows you to do some fairly fancy things in terms of managing VM movements, and virtual lans.

I have certainly done tons of integration with libraries of various types on platforms from the Raspberry pi, to Windows, and Linux in general.  Some code is easier to interop with and some code is harder.  I wrote a series a couple years back on the benefits and pitfalls of different styles of API development.

The OVS code is fairly easy to interop with using LuaJIT.  The publicly exposed stuff is all C99, and not particularly fancy.  There are a few cases where the usage of macros had me scratching my head, but a quick pass with a C compiler typically helps flesh those out.

One example will help illustrate the ease and pitfalls of doing some forms of interop.  I’ll use the UUID features in the library as an example.

First there’s data structures:

ffi.cdef[[
struct uuid {
uint32_t parts[4];
};
]]

The LuaJIT ffi makes pretty short work of that. Now the various routines that need a struct uuid will have access to what they need. The various functions are thus:

ffi.cdef[[
void uuid_init(void);
void uuid_generate(struct uuid *);
void uuid_zero(struct uuid *);
bool uuid_is_zero(const struct uuid *);
int uuid_compare_3way(const struct uuid *, const struct uuid *);
bool uuid_from_string(struct uuid *, const char *);
bool uuid_from_string_prefix(struct uuid *, const char *);
]]

Again, fairly simple. The use of ‘bool’ instead of ‘int’ as a return type from functions makes life a lot easier as the ffi will translate that to lua’s native boolean type. If it were an int, then instead of simpling doing

if not uuid_from_string(someuuidstring) then
  print("FAIL")
end

You’d have to write

if uuid_from_string(someuuidstring) ~= 0 then
  print("FAIL")
end

It seems like such a minor difference, but it’s actually a pretty huge pain in the backside. When the return value is an int, there are myriad things it can mean. One of them is ‘0 == success’, another is ‘0 == failure’, and there can by various others of course, including checking errno to find out what really went wrong or right. the bool is much more straight forward, success is usually indicated with ‘true’ and failure ‘false’.

Where things get a little more tricky and you have to pay special attention, is when you’re doing interop between different parts of the system and you flow through lua along the way.

Part of OVS has this dynamic string thing, which is one of those stretchy buffers, which will grow as necessary as you add stuff to it. One of the things it has is a ‘printf’ style formating thing.

ffi.cdef[[
void ds_put_format(struct ds *, const char *, ...) ;
]]

Yes, the LuaJIT ffi can deal with variadic functions, but you have to take extra care when feeding it the variable arguments. In particular, you’re responsible for dealing with the types of those arguments. This is where the care comes in. With normal functions (non-variadic), the ffi marshaling will just take care of things, and you don’t even realize a “number” is being turned into an ‘int32_t’, or whatever it needs to be. The ffi just figures it out based on the signature, and does the right thing. Not so with variadics. It can’t know what you need to pass.

So, here’s more from the UUID routines, some macros.

#define UUID_LEN 36
#define UUID_FMT "%08x-%04x-%04x-%04x-%04x%08x"
#define UUID_ARGS(UUID)                             \
    ((unsigned int) ((UUID)->parts[0])),            \
    ((unsigned int) ((UUID)->parts[1] >> 16)),      \
    ((unsigned int) ((UUID)->parts[1] & 0xffff)),   \
    ((unsigned int) ((UUID)->parts[2] >> 16)),      \
    ((unsigned int) ((UUID)->parts[2] & 0xffff)),   \
    ((unsigned int) ((UUID)->parts[3]))

In general, macros are of two types. First are the simple #defines which can turn into local variables. In this particular case, the UUID_LEN and UUID_FMT fit the bill. The UUID_LEN is just a numeric constant, so it can just become a regular number. The UUID_FMT doesn’t use any particularly fancy format routines (luckly lua string.format is similar enough in most cases), so that can become a simple string.

local UUID_LEN = 36
local UUID_FMT = "%08x-%04x-%04x-%04x-%04x%08x"

But what about that macro? Can we just turn it into a function, strip out the naughty bits, and call it a day?

local function UUID_ARGS(UUID)                             
    return ( ((UUID).parts[0])),            
    ( rshift((UUID).parts[1], 16)),      
    ( band((UUID).parts[1], 0xffff)),   
    ( rshift((UUID).parts[2], 16)),      
    ( band((UUID).parts[2], 0xffff)),   
    ( ((UUID).parts[3]))
end

Well, I want to write the following:

local success = uuid.uuid_from_string_prefix(id1, "12345678-1234-5678-1234-5678123456781234")
local output = ds.dynamic_string();
ds.ds_put_format(output, uuid.UUID_FMT, uuid.UUID_ARGS(id1))

print("OUTPUT: ", ds.ds_cstr(output));

Basically just round trip a string into a UUID and back out into a stretchy buffer, and print it as a lua string. Does it work? Nope, not quite. In this case, the output would be:

00000000-0000-0000-0000-45e4de9000000000

Not quite what I was expecting. Head scratch. OK, well, obviously I was a bit simplistic in my conversion of that UUID_ARGS macro. Maybe I should have preserved that casting stuff.

local uint32_t = ffi.typeof("uint32_t")

local function UUID_ARGS(UUID)                             
    local p1 = uint32_t(UUID.parts[0])
    local p2 = uint32_t(rshift(UUID.parts[1], 16))
    local p3 = uint32_t(band(UUID.parts[1], 0xffff))
    local p4 = uint32_t(rshift(UUID.parts[2], 16))
    local p5 = uint32_t(band(UUID.parts[2], 0xffff))
    local p6 = uint32_t(UUID.parts[3])

    return p1, p2, p3, p4, p5, p6
end

In addition to the temporary variables (used for debugging), I stripped out a number of parenthesis, because lua lives better with fewer parens. Basically, I used the ‘uint32_t’ type in a way that is similar to a typecast. Of course, it’s actually worse the way I’ve used it here because it creates a temporary number, which needs to be garbage collected, but the result is now:

OUTPUT: 12345678-1234-5678-1234-567812345678

Which is what I was expecting. So, maybe one last pass to clean it up…

local function UUID_ARGS(UUID)                             
    return ffi.cast("unsigned int", (UUID.parts[0])),
        ffi.cast("unsigned int", rshift(UUID.parts[1], 16)),
        ffi.cast("unsigned int", band(UUID.parts[1], 0xffff)),
        ffi.cast("unsigned int", rshift(UUID.parts[2], 16)),
        ffi.cast("unsigned int", band(UUID.parts[2], 0xffff)),
        ffi.cast("unsigned int", UUID.parts[3])
end

So, essentially, a more faithful representation of the original. Note though, although this will work with the UUID_FMT, when using a ‘C’ based ‘printf’ statements, it will NOT work with the lua string.format routine. It will complain that it was expecting a number, but received a ‘cdata’ instead. So, you have to think fairly clearly about how those APIs and macros are going to be used, and therefore how they should be translated.

Here’s that uuid.lua file in its entirety:

local ffi = require("ffi")
local bit = require("bit")

local rshift, band = bit.rshift, bit.band

local function BUILD_ASSERT_DECL(...)
    assert(...);
end

local UUID_BIT = 128;            -- Number of bits in a UUID. */
local UUID_OCTET = (UUID_BIT / 8); -- Number of bytes in a UUID. */

ffi.cdef[[
/* A Universally Unique IDentifier (UUID) compliant with RFC 4122.
 *
 * Each of the parts is stored in host byte order, but the parts themselves are
 * ordered from left to right.  That is, (parts[0] >> 24) is the first 8 bits
 * of the UUID when output in the standard form, and (parts[3] & 0xff) is the
 * final 8 bits. */
struct uuid {
    uint32_t parts[4];
};
]]

BUILD_ASSERT_DECL(ffi.sizeof("struct uuid") == UUID_OCTET);


local UUID_LEN = 36;
local UUID_FMT = "%08x-%04x-%04x-%04x-%04x%08x";

local function UUID_ARGS(UUID)                             
    return ffi.cast("unsigned int", (UUID.parts[0])),
        ffi.cast("unsigned int", rshift(UUID.parts[1], 16)),
        ffi.cast("unsigned int", band(UUID.parts[1], 0xffff)),
        ffi.cast("unsigned int", rshift(UUID.parts[2], 16)),
        ffi.cast("unsigned int", band(UUID.parts[2], 0xffff)),
        ffi.cast("unsigned int", UUID.parts[3])
end


--[[
/* Returns a hash value for 'uuid'.  This hash value is the same regardless of
 * whether we are running on a 32-bit or 64-bit or big-endian or little-endian
 * architecture. */
--]]
local function uuid_hash(uuid)
    return uuid.parts[0];
end

-- Returns true if 'a == b', false otherwise. */
local function uuid_equals(a, b)

    return (a.parts[0] == b.parts[0]
            and a.parts[1] == b.parts[1]
            and a.parts[2] == b.parts[2]
            and a.parts[3] == b.parts[3]);
end


  
ffi.cdef[[
void uuid_init(void);
void uuid_generate(struct uuid *);
void uuid_zero(struct uuid *);
bool uuid_is_zero(const struct uuid *);
int uuid_compare_3way(const struct uuid *, const struct uuid *);
bool uuid_from_string(struct uuid *, const char *);
bool uuid_from_string_prefix(struct uuid *, const char *);
]]

local Lib_uuid = ffi.load("openvswitch")

-- initialize uuid routines
Lib_uuid.uuid_init();

local exports = {
    Lib_uuid = Lib_uuid;

    UUID_BIT = UUID_BIT;
    UUID_OCTET = UUID_OCTET;
    UUID_FMT = UUID_FMT;
    UUID_ARGS = UUID_ARGS;

    -- types
    uuid = ffi.typeof("struct uuid");

    -- inline routines
    uuid_hash = uuid_hash;
    uuid_equals = uuid_equals;

    -- library functions
    uuid_init = Lib_uuid.uuid_init;
    uuid_generate = Lib_uuid.uuid_generate;
    uuid_zero = Lib_uuid.uuid_zero;
    uuid_is_zero = Lib_uuid.uuid_is_zero;
    uuid_compare_3way = Lib_uuid.uuid_compare_3way;
    uuid_from_string = Lib_uuid.uuid_from_string;
    uuid_from_string_prefix = Lib_uuid.uuid_from_string_prefix;
}

return exports

Just a couple more notes on style. When I’m using such a binding, I’ll typically do the following:

local uuid = require("uuid")

From there, I will want to utilize a simple notation to get at things.

local id = uuid.uuid(); 
uuid.uuid_generate(id);

local output = ds.dynamic_string();
ds.ds_put_format(output, uuid.UUID_FMT, uuid.UUID_ARGS(id))

 

I want to minimize the amount of times I have to use the ffi routines directly because they’re fairly low level, and can lead to some very subtle errors if you’re not careful. So, a properly constructed module will wrap up as much as possible.

In some cases I’ll create a ‘class’, which is a table to encapsulate the various bits and related bytes. In this case I did not, but I might still do it later if I find it useful.

I will typically do the ffi.load to get a handle on the library from which the routines are called, and stick that reference into a table, so the library reference won’t be garbage collected, and therefore possibly unloaded from the running system. And lastly, the function names have this construct:

    uuid_generate = Lib_uuid.uuid_generate;

It’s a simple aliasing so that I don’t have to remember which library the function is located in, I can just call uuid.uuid_generate, and the right thing will happen. This isn’t the most performant way to do it as the compiler can’t optimize the function call as much, so it will be slower. These routines are not typically called in a performant loop, so it’s an ok thing. If you were being really hard core, and needed the extra cycles, you’d make the library call directly instead of going through the table indirections.

The last trick you can do is to import that exports table into the global namespace:

for k,v in pairs(uuid) do
    _G[k] = v;
end

If you do this, then your code can drop the table indirection and become simply:

local id = uuid(); 
uuid_generate(id);

local output = dynamic_string();
ds_put_format(output, UUID_FMT, UUID_ARGS(id))

This is looking pretty close to the original ‘C’ code. Fewer pointers to deal with, no memory allocation problems, no funky difference between ‘.’ and ‘->’ indirection. In this way, Lua is just a nicer version of ‘C’ :-) My coworkers will get a kick out of that sentiment.

And so, there you have it. A fairly small example of beginning to interop with OVS. Open vSwitch is a fairly large codebase. It’s split between simple base library routines, reporting, core packet handling, and higher level apps, daemons and utilities. Starting from the top, the utilities are fairly easy to replace, once the interop at the lowest levels is dealt with. In LJIT2ovs, the lower levels are fairly well covered, and coverage increases as needs dictate. At the higher level, I am beginning with replacing simple routines, such as querying the database. At the end of the day, these higher level utilities are much easier to write, maintain, expand, and incorporate when they are written in script, rather than C.

Next time around I’ll examine some of these routines, and how they can be rewritten, and what benefits might acrue from that exercise.


Gems on the interwebs – luakernel, stb, and me

Since the day of their inception, the interwebs have been a tremendous ‘resource’ to mankind.  Sharing academic papers… cat videos!!  Endless amounts of shopping, and who can ever do without pop-up ads!

Well, I have actually found very interesting things on the interwebs of late, particularly related to coding.  There’s almost nothing under the sun which has not been coded to date, or can’t be used as a building block to code up the next big thing.

Interesting people and tidbits I’ve looked at in the past year:

Karl Malbrain – Tons of interesting stuff related to data structures in particular

Sean Barrett – Tons of stuff from truetype font rendering to image reading/writing

Bob Jenkins – Hashes, random, math, physics, just all around good stuff

There are plenty more interesting libraries out there in the world.  These few stand out because they’re not libraries.  They’re not a part of some big grand platform.  They just plain standalone pieces of code which tend to be useful as learning tools.  The other thing that stands out is their liberal licensing.  There’s nothing worse than that monkey hand in the gourd moment when you’ve found some great piece of code to borrow from, and then you realize it’s a GPL’d thing, which is enough to curdle the strongest of milks.

I favor public domain, MIT, Apache, WTF, licensing myself because if you’re going to share, then share…  Otherwise, keep it to yourself.

I the Lua world, I’ve run across some gems lately that have been very interesting.

Reuben Thomaslua-stdlib : Perhaps the most useful thing out of this library is the std.optparse routines.  They allow you to easily specify, and then parse the arguments you pass to your program.  That takes care of an amazingly annoying set of routines (getopt, optind, optarg, argc, argv…) which are extremely buggy for me, and error prone.  There are other gems and techniques in this library which make it a real treasure trove to the Lua developer.

Eric R. Schulzluakernel Wait, what?  A kernel for x64 which boots into a Lua runtime?  Well, ok, embedded Lua did that a long time ago, but this seems like something I can easily approach.  I’ve got to add my graphic library to this thing so that I can boot into a Lua kernel, and party on with my killer graphics.  So, Lua kernels from tiny devices such as the ESP8266, to higher end x64 machines.  Not bad for the little language that could.

I’m just constantly amazed at how, after slaving away at something for a while, I’ll stumble across a gem on the internet which solves the exact problem I was after, or comes close enough to make me feel like I need to leverage that work, rather than start from scratch.

The best thing to do as a developer, is to get in the habit of looking out, leveraging, building your own treasure chest of tools, so that you can create that next big thing even quicker and more robustly than before.

Now I’m going to have to make a project of getting luakernel on one of my tired old boxes, so that I can host my graphics library, and develop the UI code (including full trutype support) even further.


schedlua – async io

And so, finally we come to the point. Thus far, we looked at the simple scheduler, the core signaling, and the predicate and alarm functions built atop that. That’s all great stuff for fairly straight forward apps that need some amount of concurrency. The best part though is when you can do concurrent networking stuff.

Here’s the setup; I want to issue 20 concurrent GET requests to 20 different sites, and get results back. I want the program to halt once all the tasks have been completed.

--test_linux_net.lua
package.path = package.path..";../?.lua"

local ffi = require("ffi")

local Kernel = require("kernel"){exportglobal = true}
local predicate = require("predicate")(Kernel, true)
local AsyncSocket = require("AsyncSocket")

local sites = require("sites");

-- list of tasks
local taskList = {}


local function httpRequest(s, sitename)
	local request = string.format("GET / HTTP/1.1\r\nUser-Agent: schedlua (linux-gnu)\r\nAccept: */*\r\nHost: %s\r\nConnection: close\r\n\r\n", sitename);
	return s:write(request, #request);
end

local function httpResponse(s)
	local BUFSIZ = 512;
	local buffer = ffi.new("char[512+1]");
	local bytesRead = 0
	local err = nil;
	local cumulative = 0

	repeat
		bytesRead, err = s:read(buffer, BUFSIZ);

		if bytesRead then
			cumulative = cumulative + bytesRead;
		else
			print("read, error: ", err)
			break;
		end
	until bytesRead < 1

	return cumulative;
end


local function siteGET(sitename)
	print("siteGET, BEGIN: ", sitename);

	local s = AsyncSocket();

	local success, err = s:connect(sitename, 80);  

	if success then
		httpRequest(s, sitename);
		httpResponse(s);
	else
		print("connect, error: ", err, sitename);
	end

	s:close();

	print("siteGET, FINISHED: ", sitename)
end


local function allProbesFinished()
	for idx, t in ipairs(taskList) do
		if t:getStatus() ~= "dead" then
			return false;
		end
	end

	return true;
end

local function main()
	for count=1,20 do
		table.insert(taskList, Kernel:spawn(siteGET, sites[math.random(#sites)]))
		Kernel:yield();
	end

	when(allProbesFinished, halt);
end

run(main)

Step by step. The httpRequest() function takes a socket, and does the most bare mimimal HTTP GET request, assuming the socket is already connected to the site.

Similarly, the httpResponse() function gets a response back from the server, and reads as much as it can until the socket is closed (because the Connection: close header was sent).

That’s about the most basic of HTTP request/response pairs you can have, ignoring doing any parsing of the returned data.

Alright, so let’s wrap those two up into a function called siteGET(). siteGET(sitename) takes the name of a site, creates a socket, connects it to the site, and then issues the httpRequest(), and then the httpResponse(). Very simple. What I like about this is that the httpRequest(); httpResponse() sequence is executed in serial as far as I’m concerned. I don’t have to be worried about the httpResponse() being issued before the request completes. Furthermore, if I didn’t use a spawn(), I could simply execute the code directly and be none the wiser.

I want to execute these siteGET()s concurrently though, so within main(), I start up 20 of these tasks, and let them go. Then comes the waiting part:

local function allProbesFinished()
	for idx, t in ipairs(taskList) do
		if t:getStatus() ~= "dead" then
			return false;
		end
	end

	return true;
end

	when(allProbesFinished, halt);

Going back to our knowledge of predicates, we know that the ‘when’ function takes a predicate (function that returns true/false), and will execute the second function when the predicate returns true.

OK, so we just need to come up with a predicate which tells us that all the tasks have completed. Easy enough as a list of the tasks is generated when they are spawned. So, we just go through that list and see if any of them are still running. If there is a single one that is still running, the predicate will return false, and ‘halt()’ will not be called. As soon as the last task finished, the predicate will return true, and the halt() function will be called.

Of course, most things in schedlua are convenient compositions of deeper things (with signals being at the core).

Instead of using the ‘when’ function, you could write the code more directly like this:

	while true
		if allProbesFinished() then
			halt();
			break;
		end
		yield();
	end

That doesn’t quite look as nice as just using the when() function I think. Also, you’re sitting in the main() function, which is no big deal as there’s nothing else trying to execute after this, but it just doesn’t seem as clean. Furthermore, the ‘when’ function might have some magic in its implementation, such as a better understanding of the state of tasks, or special knowledge of the scheduler, or who knows what. At any rate, either way essentially implements a barrier, and the technique can be used anywhere you want to perform an action after some set of tasks has completed. The allProbesFinished() function can be generalized to wait on any list of tasks, maybe call it “waitForTasks()” or some such thing.

At any rate, that completes the primitives that are baked into the core schedlua package. Everything from signals, to predicates, alarms, and finally async io. Of course this is Linux, so async io works with any file descriptor, not just network sockets, so file management or device communications in general can be thrown into the mix.

Now that the basics work, it’s a matter of cleaning up, writing more test cases, fixing bugs, reorganizing, and optimizing resource usage a bit. In general though, the constructs are there, and it’s ready to be used for real applications.


Follow

Get every new post delivered to your Inbox.

Join 51 other followers