Kinect to Lua

Good News everyone!  At least that’s how one of my favorite electronics supply shops AndaHammer.com starts every post.   I finally got the Kinect to spit out some color information, and I can consume it using LuaJIT!

What a long and arduous road it has been!  One of the things I can say from my experience is, if you’re going to use the Microsoft SDK 1.0 for Kinect, first uninstall all other drivers you might have been fooling around with, and reboot.  This caught me for the longest time.  I was getting sporadic behavior, errors with invalid pointers, and all manner of frustration.

After Uninstalling the CLNui stuff, and rebooting, the proper device driver for the Kinect installed itself, and things improved.

The next challenge had to do with the format of the data coming off the Kinect.  The color data is a somewhat funky format.  It is RGB-32.  Not RGB-24 (8 bits per component, 24 bits per pixel).  RGB-32 is 8 bits per component, but it is actually 32 bits per pixel.  The remaining 8 bits are ignored.  They are not alpha.  If they were, the value would be set to 255.  But, since they are “ignored”, the final 8-bits are set to ‘0’.  Well, that’s a bit of a bother.  In my ideal world, I’d simply represent the data as a RGBA pixel, and be done with it.  But, I can’t.  What I need is a RGB-32, which will have an OpenGL signature of RGB.  Well, not the hardest thing in the world, but it will take some small changes to my frameworks to make it happen.

In the meanwhile, I just change my GLTexture object to have an internal format of RGB (ignoring alpha), and that fixed the problem.

What you see in the picture is an upside down representation of the view out my apartment window.  Yep, another one of those little things.  The picture is bottom up, or top down, depending on your perspective.  It’s probably top down coming off the camera, and my OpenGL view of it is flipping it.  But, the data is there.

A little bit about the code.  First of all, it’s all in GitHub.  You can find the Nui low level interop stuff in the BanateCoreWin32 repository under the Kinect directory.  In the directory, there is the Kinect.lua file, which is the simplified wrapper on top of the core Nui calls.

How to use it?  First you initialize a sensor, based on an index value:

require "Kinect"
local sensor0 = Kinect.GetSensorByIndex(0, NUI_INITIALIZE_FLAG_USES_COLOR)

In this case, I am initializing the first sensor in my system (you can have multiples), and telling it I’ll only be grabbing color information, no skeleton tracking or depth information.

Then, when I want to grab a frame and copy bits, I do this:

local success = sensor0:GetCurrentColorFrame()
local colorAccessor = Array2DAccessor({
    TypeName = "Ppixel_BGRA_b",
    Width = captureWidth,
    Height = captureHeight,
    Data = sensor0.LockedRect.pBits,
    BytesPerElement= 4,
    Alignment = 1,
})

screenTexture:CopyPixelBuffer(colorAccessor)

sensor0:ReleaseCurrentColorFrame()

At the moment, you MUST make the call to ReleaseCurrentColorFrame, or you’ll find you’re not getting any frames soon enough.

This is the “polling” style of getting frames from the device.  The preferred way is to actually use an eventing model where the device will alert you when frames are available.  My first run through though, I just wanted to get at the pixels as soon as possible, so I polled.

This device is very finicky.  It doesn’t like to be polled too much, and there are tons of error cases that you need to be aware of and deal with properly.  At this point though, I know I am talking to it successfully.  I even managed to make the tilt go up and down, which is nice.

Now that I have a code path that is basically working, I can move on to making it much nicer to use.  For example, it would be nice to represent the camera as an iterator, where I can just essentially say GetNext(), and get the next instance of the color frame, whithout having to worry about releasing things.

At any rate, there it is, ready for other people to hack on and improve.

---