LJIT2libevdev – input device tracking on Linux

Once you start going down into the rabbit hole that is UI on Linux, there seems to be no end. I was wanting to get to the bottom of the stack as it were, because I just want to get raw keyboard and mouse events, and do stuff with them. There is a library that helps you do that call libevdev. Here is the luajit binding to it:

LJIT2libevdev

As it turns out, getting keyboard and mouse activity is highly dependent on what environment you’re in of course. Are you sitting at a Terminal, in which cases ncurses or similar might be your best choice. If you’re looking at a graphics display, then something related to X, or the desktop manager might be appropriate. At the very bottom of it all though is the kernel, and it’s ability to read the keyboard and mouse, and report what it finds up the chain to interested parties. Down there at the very bottom is a userspace library libevdev, which takes care of making the ioctl calls into the kernel to get the raw values. Great! Only caveat is that you need to be setup with the proper permissions to do it because you’re getting ALL of the keyboard and mouse events on the system. Great for key loggers…

Alright, so what does this mean in the context of Lua? Well, libevdev is a straight up C interface to which is a very thin veneer atop the ioctl calls. It would not be that hard to actually replace the ioctl calls with ioctl calls from luajit directly, but the maintainers of libevdev seem to have it covered quite nicely, so ffi calls to the library are sufficient. The library provides some conveniences like tables of strings to convert from the integer values of things to their string name equivalents. These could probably be replaced with the same within lua land, and save the round trips and string conversions. As a low level interface, it does not provide managedment of the various input devices. You can not ask it “give me the logitech mouse”. You have to know which device is the mouse in the first place before you can start asking for input. Similarly, it’s giving you a ton of raw data that you may not be interested in. Things like the sync signals, indicating the end of an event train. Or the skipped data events, so you can catch up if you prefer not to lose any. How to manage it all?

Let’s start at the beginning.

I have found it challenging to find appropriate discussions relating to UI on Linux. Linux has such a long history, and for most of it, the UI subsystems have been evolving and changing in fundamental ways. So, as soon as you find that juicy article dated from 2002, it’s been replaced by something in 2006, and then again in 2012. It also depends on whether you’re talking about X11, Wayland, Qt, Gnome, SDL, terminal, or some other context.

Recently, I was trying to track down the following scenario: I want to start reading input from the attached logitech mouse on my laptop. Not the track pad under my thumbs, and not the little red nubby stick in the middle of the keyboard, but that mouse specifically. How do I do that?

libevdev is the right library to use, but in order to use it, you need a file descriptor for the specific device. The interwebs tell me you simply open up the appropriate /dev/input/eventxxx file and start reading from it? Right. And how do I know which is the correct ‘eventxxx’ file I should be reading from? You can simply do:

$ cat /proc/bus/input

Look at the output, find the device you’re interested in, look at which event it indicates it’s attached to, then go open up that event…

And how do I do that programatically, and consistently such that it will be the same when I move the mouse to a different system? Ah yes, there’s a library for that, and why don’t you just use Python, and…

Or, how about this:

local EVContext = require("EVContext")

local function isLogitech(dev)
    return dev:name():lower():find("logitech") ~= nil
end

local dev = EVContext:getMouse(isLogitech);

assert(dev, "no mouse found")

print(string.format("Input device name: \"%s\"", dev:name()));
print(string.format("Input device ID: bus %#x vendor %#x product %#x\n",
        dev:busType(),
        dev:vendorId(),
        dev:productId()));

-- print out a constant stream of events
for _, ev in dev:events() do
    print(string.format("Event: %s %s %d",
        ev:typeName(),
        ev:codeName(),
        ev:value()));
end

How can I get to this state? First, how about that EVContext thing, and the ‘getMouse()’ call?

EVContext is a convenience class which wraps up all the things in libevdev which aren’t related to a specific instance of a device. So, doing things like iterating over devices, setting the logging level, getting a specific device, etc. Device iteration is a core piece of the puzzle. So, here it is.

function EVContext.devices(self)
    local function dev_iter(param, idx)
        local devname = "/dev/input/event"..tostring(idx);
        local dev, err = EVDevice(devname)

        if not dev then
            return nil; 
        end

	return idx+1, dev
    end

    return dev_iter, self, 0
end

That’s a quick and dirty iterator that will get the job done. Basically, just construct a string of the form ‘/dev/input/eventxxx’, and vary the ‘xxx’ with numbers until you can no longer open up devices. For each one, create a EVDevice object from that name. A bit wasteful, but highly beneficial. Once we can iterate all the input devices, we can leverage this for greater mischief.

Looking back at our code, there was this bit to get the keyboard:

local function isLogitech(dev)
    return dev:name():lower():find("logitech") ~= nil
end

local dev = EVContext:getMouse(isLogitech);

It looks like we could just call the ‘EVContext:getMouse()’ function and be done with it. What’s with the extra ‘isLogitech()’ part? Well, on its own, getMouse() will simply return the first device which reportedly is like a mouse. That code looks like this:

function EVDevice.isLikeMouse(self)
	if (self:hasEventType(EV_REL) and
    	self:hasEventCode(EV_REL, REL_X) and
    	self:hasEventCode(EV_REL, REL_Y) and
    	self:hasEventCode(EV_KEY, BTN_LEFT)) then
    	
    	return true;
    end

    return false;
end

It’s basically saying, a ‘mouse’ is something that has relative movements, at least an x and y axis, and a ‘left’ button. On my laptop, the little mouse nub on the keyboard qualifies, and since it has a lower /dev/input/event number (3), it will be reported before any other mouse on my laptop. So, I need a way to filter on anything that reports to be a mouse, as well as having “logitech” in its name. The code for that is the following from EVContext:

function EVContext.getDevice(self, predicate)
	for _, dev in self:devices() do
		if predicate then
			if predicate(dev) then
				return dev
			end
		else
			return dev
		end
	end

	return nil;
end

function EVContext.getMouse(self, predicate)
	local function isMouse(dev)
		if dev:isLikeMouse() then
			if predicate then
				return predicate(dev);
			end
			
			return true;
		end

		return false;
	end

	return self:getDevice(isMouse);
end

As you can see, ‘EVContext:getDevice()’ takes a predicate (a function that returns true or false). It will iterate through all the devices, applying the predicate to each device in turn. When it finds a device matching the predicate, it will return that device. Of course, you could easily change this to return ALL the devices that match the predicate, but that’s a different story.

The ‘predicate’ in this case is the internal ‘isMouse’ function within ‘getMouse()’, which in turn applies two filters. The first is calling the ‘isLikeMouse()’ function on the device. If that’s satisfied, then it will call the predicate that was passed in, which in this case would be our ‘isLogitech()’ function. If that is satisfied, then the device is returned.

In the end, here’s some output:

Input device name: "Logitech USB Optical Mouse"
Input device ID: bus 0x3 vendor 0x46d product 0xc018

Event: EV_REL REL_Y -1
Event: EV_SYN SYN_REPORT 0
Event: EV_REL REL_Y -1
Event: EV_SYN SYN_REPORT 0
Event: EV_REL REL_X -1
Event: EV_REL REL_Y -2
Event: EV_SYN SYN_REPORT 0
Event: EV_REL REL_Y -1
Event: EV_SYN SYN_REPORT 0
Event: EV_MSC MSC_SCAN 589827
Event: EV_KEY BTN_MIDDLE 1
Event: EV_SYN SYN_REPORT 0
Event: EV_MSC MSC_SCAN 589827
Event: EV_KEY BTN_MIDDLE 0
Event: EV_SYN SYN_REPORT 0
Event: EV_REL REL_Y -2
Event: EV_SYN SYN_REPORT 0
Event: EV_REL REL_X 1
Event: EV_SYN SYN_REPORT 0
Event: EV_REL REL_Y -1

Some relative movements, a middle button press/release, some more movement.

The libevdev library represents some pretty low level stuff, and for the moment it seems to be the ‘correct’ way to deal with system level input device event handling. The LJIT2libevdev binding provide both the fundamental access to the library as well as the higher level device access which is sorely needed in this environment. I’m sure over time it will be beneficial to pull some of the conveniences that libevdev provides directly into the binding, further shrinking the required size of the library. For now though, I am simply happy that I can get my keyboard and mouse events into my application without too much fuss.


Handy Mouse – Using the Leap as a mouse controller

By using the Leap Motion, I have finally taken the step of getting into “Gesturing”.  By that I just mean using my hands and fingers in ways other than keyboard and mouse.  Of course the Leap Motion allows you to point, with a stick, your fingers, and what have you.

As a basic input device, it spews a stream of data to the developer, who must then make sense of it.  This is true of all input devices, even going down to the lowly mouse.  The difference with more traditional devices is their input stream has been well understood and mapped to models that work for existing applications.  Mouse spew: x,y,wheel, button and that’s about it.  Keyboard: keycode, up/down, led state.

The Leap Motion, and other such input devices are a bit different.  Their data streams, and how they map to applications are different.  What does waving your hand wildy do?  What does swiping in the form of an “S” curve do?  Unknown.  Furthermore, how do these motions map to a traditional application?  Should they?

Well, one mapping that I have chosen to take a look at is mouse movement. Seems simple enough.  Basically, pick up a pencil, or chopstick, and use it to point at the screen, move around, and ‘click’ by doing quick dips of the tip.  Seems natural enough.  Here’s a bit of code to achieve that:

 

-- pointertrack.lua
package.path = package.path.."../?.lua"

local LeapScape = require ("LeapScape");
local MouseBehavior = require("MouseBehavior");
local UIOSimulator = require("UIOSimulator");

local scape, err = LeapScape();

if not scape then 
  print("No LeapScape: ", err)
  return false
end

local OnMouseMove = function(param, x, y)
  UIOSimulator.MouseMove(x, y);
end

local mousetrap = MouseBehavior(scape, UIOSimulator.ScreenWidth, UIOSimulator.ScreenHeight);
mousetrap:AddListener("mouseMove", OnMouseMove, nil);

scape:Start();

run();

Of course this uses the TINN runtime, and can be found in the Leap TINNSnip project.

What’s going on here? First of all, create the LeapScape object. Then create a function which will turn x,y, coordinates into simulated mouse movements. Lastly, create an instance of the ‘MouseBehavior’ object, and add the OnMouseMove() function as a listener of the “mouseMove” event. Start the LeapScape, and run() the program.

You do this from a command line. Once the program is running, you can point the stick around the screen and see that the cursor will follow it.

But first, you have to create a calibration that allows you to constrain the movement. With the Leap Motion, you get a roughly 2’x2’x2′ area in which it can detect various movements. What you need to do is create a mapping from a porting of that space into your screen coordinates. So, first you run the mousetrain.lua program. Which looks like this:

--package.path = package.path.."../../?.lua"
package.path = package.path.."../?.lua"

local LeapScape = require ("LeapScape");
local FrameObserver = require("FrameObserver");
local UIOSimulator = require("UIOSimulator");
local StopWatch = require("StopWatch");
local GDI32 = require ("GDI32");
local FileStream = require("FileStream");


--[[
	Map a value from one range to another
--]]
local mapit = function(x, minx, maxx, rangemin, rangemax)
  return rangemin + (((x - minx)/(maxx - minx)) * (rangemax - rangemin))
end

--[[
	Clamp a value to a range
--]]
local clampit = function(x, minx, maxx)
  if x < minx then return minx end
  if x > maxx then return maxx end

  return x
end

local main = function()
  local scape, err = LeapScape();

  if not scape then 
    print("No LeapScape: ", err)
    return false
  end

  local fo = FrameObserver(scape);

  local sensemin = {math.huge, math.huge, math.huge}
  local sensemax = {-math.huge, -math.huge, -math.huge}

  -- We'll use this to do some drawing on the screen
  local hdcScreen = GDI32.CreateDCForDefaultDisplay();


  local busywait = function(millis)
    sw = StopWatch.new();

    while true do
      if sw:Milliseconds() > millis then
        break
      end
      coroutine.yield();
    end
  end

  local drawTarget = function(originx, originy, width, height)
    local brushColor = RGB(255,0,0);

    x = originx - width/2;
    y = originy - height/2;

    x = clampit(x, 0, UIOSimulator.ScreenWidth-1 - width);
    y = clampit(y, 0, UIOSimulator.ScreenHeight-1 - height);

    local right = x + width
    local bottom = y + height
--print(x,y,width,height)
    hdcScreen:SetDCBrushColor(brushColor)
    hdcScreen:RoundRect(x, y, right, bottom, 4, 4)
  end

  local observerange = function(param, event)
    local newvalue = false
    local tp = event.tipPosition;

    sensemin[1] = math.min(tp[1], sensemin[1])
    sensemin[2] = math.min(tp[2], sensemin[2])
    sensemin[3] = math.min(tp[3], sensemin[3])

    sensemax[1] = math.max(tp[1], sensemax[1])
    sensemax[2] = math.max(tp[2], sensemax[2])
    sensemax[3] = math.max(tp[3], sensemax[3])
  end

  local dwellAtPosition = function(x, y)
    drawTarget(x,y, 32,32);
    busywait(500);
    fo:AddPointerObserver(observerange, nil)
    busywait(1000);
    fo:RemovePointerObserver(observerange, nil)
  end

  local matchTargets = function()
    dwellAtPosition(0, UIOSimulator.ScreenHeight-1);
    dwellAtPosition(0, 0);
    dwellAtPosition(UIOSimulator.ScreenWidth-1, 0);
    dwellAtPosition(UIOSimulator.ScreenWidth-1, UIOSimulator.ScreenHeight-1);
  end

  local writeConfig = function()
    fs = FileStream.Open("sensor.cfg");

    local output = {
      string.format("do return {");
      string.format("sensemin = {%3.2f, %3.2f, %3.2f};", sensemin[1], sensemin[2], sensemin[3]);
      string.format("sensemax={%3.2f, %3.2f, %3.2f};", sensemax[1], sensemax[2], sensemax[3]);
      string.format("} end");
    }
    output = table.concat(output,"\n");
    fs:WriteString(output);
    fs:Close();
  end

  scape:Start();
  matchTargets();
  writeConfig();
  stop();
end

run(main);

This snippet will place a target in each of the 4 corners of the screen. You point at it for a bit, and then move on to the next one. Easy enough. Once you’ve done that, it will write this range information out to a configuration file. Then you can use the mouse pointtracker to your heart’s content.

There is a ‘MouseBehavior’ object, which is a work in progress. It basically filters through the stream of events coming off the Leap device and determines what is ‘move’, what is ‘click’ and the like, and fires off the appropriate event to whomever may be observing. Oh yes, the IObservable/IEnumerable thing comes home to roost.

The Behaviors are an interesting place to be. It really makes you think. What is a “mouse click”? Is it a dip of the pointing device? By how much, and how long? Is there a ‘right click’? How can I sumulate the wheel? Is it perhaps a circular motion? This is where being able to annotate a gesture, and then subsequently search for that pattern in the data stream becomes really interesting.

For now, I’m happy enough to get basic mouse movement. Soon enough though, chording gestures should be close at hand.