Taming the Leap Motion

I have been playing with the Leap Motion input device of late.  If you haven’t seen one of these, it’s worth taking a look at the video demonstrations.  The Leap joins the likes of the Kinect in terms of offering an alternative form of input than the standard mouse and keyboard.  But, that’s about where the similarities end.

Whereas the Kinect is good at large body movements of multiple players at a distance, it’s not so great at fine hand tracking at a close distance.  This is the specialty of the Leap Motion.  You can point at your screen, move your hands around, tap your fingers and the like.

It’s kind of like turning every monitor into a touch screen, except it provides a larger volume than the flat surface of the screen.

The Leap Motion comes with a SDK for developers to write whatever it is they’re going to write with this new input device.  The core of the SDK is a library written in C++.  That’s great for languages such as Python, Java, C# and the like which can use SWIG to turn the header file into something they can interop to.  This approach might work for Lua as well, but C++ is just a pain to deal with from these dynamic languages.

I want to play with the Leap Motion, but I don’t want to do it through the supplied SDK as C++ just doesn’t work for me.  Fortunately, the Leap Motion provides a different mechanism.  The Leap Motion offers up a WebSocket server interface.  So, you can ‘talk’ to it by simply opening up a websocket and start streaming the results.

Alrighty then, how to get started?

Well, First of all the url for the device is: “ws://127.0.0.1:6437/”

I have a hacked together WebSocket object that’s good enough to establish a connection to this service endpoint.  So, I do that, and encapsulate the connection in a LeapInterface_t object.  This interface serves out a constant stream of JSON formatted data.  To make things simple for Lua consumption, I put an iterator interface on this:

LeapInterface_t.RawFrames = function(self)
  local closure = function()
    local frame, err = self.SocketStream:ReadFrame();
    if not frame then
      return nil, err
    end

    return frame;
  end

  return closure;
end

With this alone, you could do something like:

local leap = LeapInterface()
for rawframe in leap:RawFrames() do
  print(rawframe);
end

That’s pretty much all you need to see the stream of frames coming out of the Leap Motion. But, that’s not particularly useful. There’s a ton of information you can get out of this device. You can get hand tracking, finger and ‘tool’ tracking, and even ‘gestures’. It’s almost too much information to make sense of. So, a little bit of organization is in order.

First, I create an object that will be the central focus of all things Leap Motion. This is the LeapScape object. I chose this name because I conceive of the area the Leap Motion can watch and report on as the LeapScape. There can be hands, pointables, gestures, and the like. I need something to control and coordinate all these things. It looks like this:

local Collections = require("Collections");
local LeapInterface = require("LeapInterface");
local JSON = require("dkjson");

local LeapScape_t = {}
local LeapScape_mt = {
	__index = LeapScape_t,
}

local LeapScape = function()
	local interface, err = LeapInterface({enableGestures=true});

	if not interface then
		return nil, err;
	end

	local obj = {
		Interface = interface;

		Hands = {};
		Pointables = {};
		Frames = {};

		FrameQueue = Collections.Queue.new();
		ContinueRunning = false;
	};

	setmetatable(obj, LeapScape_mt);

	return obj;
end

LeapScape_t.Start = function(self)
	self.ContinueRunning = true;

	Runtime.Scheduler:Spawn(LeapScape_t.ProcessFrames, self)
	Runtime.Scheduler:Spawn(LeapScape_t.ProcessRawFrames, self);
end

LeapScape_t.ProcessRawFrames = function(self)

	for rawframe in self.Interface:RawFrames() do
		local frame = JSON.decode(ffi.string(rawframe.Data, rawframe.DataLength));
		self.FrameQueue:Enqueue(frame);

		coroutine.yield();
	end
end

LeapScape_t.ProcessFrames = function(self, frame)
	while self.ContinueRunning do
		-- get a frame off the queue
		local frame = self.FrameQueue:Dequeue()
		if frame then
			if frame.gestures then
				for _, gesture in ipairs(frame.gestures) do
					self:OnGesture(gesture);
				end
			end
		end
		coroutine.yield();
	end
end

LeapScape_t.OnGesture = function(self, gesture)
  if self.GestureHandler then
    self.GestureHandler:OnGesture(gesture);
  end
end

Another piece that focuses on gesture handling specifically is the “GestureHandler” class. This is fairly straight forward as well. It will take a “gesture”, which is just a decoded JSON string, and perform various actions based on what it reads there. One of the things it does is provide some order to the input. For example, it’s totally possible that while you’re processing a ‘swipe’, you get some intervening ‘screenTap’ and ‘keyTap’ gestures, which may not be desirable. So, the GestureHandler will create some order by only allowing sweep update and ‘end’ gestures to filter through, until the sweep is completed.

GestureHandler_t = {}
GestureHandler_mt = {
	__index = GestureHandler_t;
}

GestureHandler = function()
	local obj = {
		CurrentGesture = "none";
	}

	setmetatable(obj, GestureHandler_mt);

	return obj;
end

GestureHandler_t.OnGesture = function(self, gesture)
	--print("==== GESTURE ====")
	--print("type: ", gesture.type, gesture.state);

	if gesture.type == "screenTap" then
		self:HandleScreenTap(gesture)
	elseif gesture.type == "keyTap" then
		self:HandleKeyTap(gesture)
	elseif gesture.type == "swipe" then
		self:HandleSwipe(gesture);
	elseif gesture.type == "circle" then
		self:HandleCircle(gesture);
	end
end

GestureHandler_t.HandleScreenTap = function(self, gesture)
	if self.CurrentGesture == "none" then
		if self.OnScreenTap then
			self.OnScreenTap(gesture);
		end
	end
end

GestureHandler_t.HandleKeyTap = function(self, gesture)
	if self.CurrentGesture == "none" then
		if self.OnKeyTap then
			self.OnKeyTap(gesture);
		end
	end
end

GestureHandler_t.HandleCircle = function(self, gesture)
	if not (self.OnCircleBegin or self.OnCircling or self.OnCircleEnd) then
		return
	end

	if self.CurrentGesture == "circle" then
		if gesture.state == "stop" then
			if self.OnCircleEnd then
				self.OnCircleEnd(gesture);
			end
			self.CurrentGesture = "none";
		elseif gesture.state == "update" then
			if self.OnCircling then
				self.OnCircling(gesture)
			end
		end
	elseif self.CurrentGesture == "none" then
		self.CurrentGesture = "circle";
		if self.OnCircleBegin then
			self.OnCircleBegin(gesture)
		end
	end
end

GestureHandler_t.HandleSwipe = function(self, gesture)
	if not (self.OnSwipeBegin or self.OnSwiping or self.OnSwipeEnd) then
		return
	end

	if self.CurrentGesture == "swipe" then
		if gesture.state == "stop" then
			if self.OnSwipeEnd then
				self.OnSwipeEnd(gesture);
			end
			self.CurrentGesture = "none";
		elseif gesture.state == "update" then
			if self.OnSwiping then
				self.OnSwiping(gesture)
			end
		end
	elseif self.CurrentGesture == "none" then
		self.CurrentGesture = "swipe";
		if self.OnSwipeBegin then
			self.OnSwipeBegin(gesture)
		end
	end
end

return GestureHandler

OK. So, there’s the lowest level WebSocket, the subsequent LeapInterface, the enclosing LeapScape, and finally the Gesture Handler. Pulling it all together in a little demo program, you get:

local Runtime = require("Runtime");
local LeapScape = require ("LeapScape");
local GestureHandler = require("GestureHandler");

local printDict = function(dict)
	for k,v in pairs(dict) do
		print(k,v)
	end
end

local OnSwipeBegin = function(gesture)
	local p = gesture.position;
	print("============")
	print("SWIPE BEGIN: ", p[1], p[2], p[3])
end

local OnSwipeEnd = function(gesture)
	local p = gesture.position;
	local d = gesture.direction;

	print("SWIPE END: ", p[1], p[2], p[3]);
	print("Direction: ", d[1], d[2], d[3]);
	print("Speed: ", gesture.speed);
end

local OnCircleBegin = function(gesture)
	local p = gesture.position;
	print("============")
	print("CIRCLE BEGIN: ")
end

local OnCircling = function(gesture)
	local n = gesture.normal;
	local direction = "ccw";
	if n[1] <0 and n[3] < 0 then
		direction = "cw"
	end

	print(string.format("CIRCLING: %f %s", gesture.progress, direction));
	--printDict(gesture);
end

local OnCircleEnd = function(gesture)
  local c = gesture.center;
  print(string.format("CIRCLE END: %f [%f, %f, %f]", gesture.radius, c[1], c[2], c[3]));
end

local OnScreenTap = function(gesture)
  local p = gesture.position;
  print("SCREEN TAP: ", p[1], p[2], p[3]);
end

local OnKeyTap = function(gesture)
  local p = gesture.position;
  print("KEY TAP: ", p[1], p[2], p[3]);
end

local main = function()
  local scape = LeapScape();
  local ghandler = GestureHandler();

  -- Swipes
  --ghandler.OnSwipeEnd = OnSwipeEnd;
  --ghandler.OnSwipeBegin = OnSwipeBegin;

  -- Circles
  ghandler.OnCircleBegin = OnCircleBegin;
  ghandler.OnCircling = OnCircling;
  ghandler.OnCircleEnd = OnCircleEnd;

  -- Taps
  --ghandler.OnKeyTap = OnKeyTap;
  --ghandler.OnScreenTap = OnScreenTap;

  scape.GestureHandler = ghandler;

  spawn(scape.Start, scape)
end

run(main);

In this particular case, I’m just tracking the ‘circle’ gesture. If you were doing a real application, you would probably tie this circle gesture to something meaningful in the application, like turning a knob perhaps, or rotating something one way or the other. Notice that within ‘main’, at the very end, the scape.Start routine is spawned. That means that it will run in parallel to any other spawned fibers running in the application, including things like network communications and UI.

The Leap Motion is a nifty device. It presents quite a lot of information to the programmer. Having a fairly simple websocket based interface available makes it relatively easy to do programming with it. I did not require a ‘C’ interface in the end, but just did whatever I wanted with the raw information provided.

I’m sure there will be some very interesting applications created with this device. Using TINN, this becomes a snap as the code here is a TINNSnip located here: leaper