All the pretty little asyncs…

I have gone on about various forms of async for quite some time now. So could there possibly be more? Well, yes of course!

Here’s the scenario I want to enable. I want to keep track of my file system activity, sending the various operations to a distant storage facility. I want to do this while a UI displays what’s going on, and I want to be able to configure things while its happening, like which events I really care to shadow, and the like.

I don’t want to use multiple OS level threads if I can at all avoid them as they will complicate my programming tremendously. So, what to do.

Well, first I’ll start with the file tracking business. I have talked about change journals in the past. This is a basic mechanism that Windows has to track changes to the file system. Every single open, close, delete, write, etc, has an entry in this journal. If you’re writing a backup program, you’ll be using change journals.

The essence of the change journal functionality is usage of the DeviceIoControl() function. Most of us are very familiar with the likes of CreateFile(), ReadFile(), WriteFile(), CloseHandle(), when it comes to dealing with files. But, for everything else, there is this DeviceIOControl() function.

What is a device? Well, you’d be surprised to learn that most things in the Windows OS are represented by ‘devices’ just like they are in UNIX systems. For example, ‘C:’, is a device. But, also “DISPLAY1” is also a device, as are “LCD” and “PhysicalDisk0”. When it comes to controlling devices, the Win32 level API calls will ultimately make DeviceIoControl() calls with various parameters. That’s great to know as it allows you to create whatever API you want, as long as you know the nuances of the device driver you’re trying to control.

But, I digress. The key point here is that I can open up a device, and I can make a DeviceIoControl() call, and true to form, I can use OVERLAPPED structures, and IO Completion Ports. That makes these calls “async”, or with TINN, cooperative.

To wrap it up in a tidy little bow, here is a Device class which does the grunt work for me:

--[[
References
http://msdn.microsoft.com/en-us/magazine/cc163415.aspx
--]]
local ffi = require("ffi")
local bit = require("bit")
local bor = bit.bor;

local core_file = require("core_file_l1_2_0");
local core_io = require("core_io_l1_1_1");
local Application = require("Application")
local IOOps = require("IOOps")
local FsHandles = require("FsHandles");
local errorhandling = require("core_errorhandling_l1_1_1");
local WinBase = require("WinBase");


local Device = {}
setmetatable(Device, {
	__call = function(self, ...)
		return self:open(...)
	end,
})
local Device_mt = {
	__index = Device,
}

function Device.init(self, rawhandle)
	local obj = {
		Handle = FsHandles.FsHandle(rawhandle)
	}
	setmetatable(obj, Device_mt)
	
	Application:watchForIO(rawhandle, rawhandle)

	return obj;
end


function Device.open(self, devicename, dwDesiredAccess, dwShareMode)
	local lpFileName = string.format("\\\\.\\%s", devicename);
	dwDesiredAccess = dwDesiredAccess or bor(ffi.C.GENERIC_READ, ffi.C.GENERIC_WRITE);
	dwShareMode = bor(FILE_SHARE_READ, FILE_SHARE_WRITE);
	local lpSecurityAttributes = nil;
	local dwCreationDisposition = OPEN_EXISTING;
	local dwFlagsAndAttributes = FILE_FLAG_OVERLAPPED;
	local hTemplateFile = nil;

	local handle = core_file.CreateFileA(
        lpFileName,
        dwDesiredAccess,
        dwShareMode,
     	lpSecurityAttributes,
        dwCreationDisposition,
        dwFlagsAndAttributes,
     	hTemplateFile);


	if handle == INVALID_HANDLE_VALUE then
		return nil, errorhandling.GetLastError();
	end

	return self:init(handle)
end

function Device.getNativeHandle(self)
	return self.Handle.Handle;
end

function Device.createOverlapped(self, buff, bufflen)
	local obj = ffi.new("FileOverlapped");
	
	obj.file = self:getNativeHandle();
	obj.OVL.Buffer = buff;
	obj.OVL.BufferLength = bufflen;

	return obj;
end

function Device.control(self, dwIoControlCode, lpInBuffer, nInBufferSize, lpOutBuffer, nOutBufferSize)
	local lpBytesReturned = nil;
	local lpOverlapped = self:createOverlapped(ffi.cast("void *", lpInBuffer), nInBufferSize);


	local status = core_io.DeviceIoControl(self:getNativeHandle(), 
          dwIoControlCode, 
          ffi.cast("void *", lpInBuffer),
          nInBufferSize,
          lpOutBuffer,
          nOutBufferSize,
          lpBytesReturned,
          ffi.cast("OVERLAPPED *",lpOverlapped));

	local err = errorhandling.GetLastError();

	-- Error conditions
	-- status == 1, err == WAIT_TIMEOUT (258)
	-- status == 0, err == ERROR_IO_PENDING (997)
	-- status == 0, err == something else

	if status == 0 then
		if err ~= ERROR_IO_PENDING then
			return false, err
		end
	end

    local key, bytes, ovl = Application:waitForIO(self, lpOverlapped);

    return bytes;
end

return Device

I’ve shown this kind of construct before with the NativeFile object. That object contains Read, and Write functions as well, but lacks the control() function. Of course the two could be combined for maximum benefit.

How to use this thing?

dev = Device("c:")
dev:control(...)

OK, that’s out of the way. Now, what about this change journal thing? Very simple now that the device is handled.
A change journal can look like this:

-- USNJournal.lua
-- References
-- http://msdn.microsoft.com/en-us/library/windows/desktop/aa364563(v=vs.85).aspx
-- http://www.microsoft.com/msj/0999/journal/journal.aspx
-- http://www.microsoft.com/msj/1099/journal2/journal2.aspx
-- 

local ffi = require("ffi");
local bit = require("bit");
local bor = bit.bor;
local band = bit.band;

local core_io = require("core_io_l1_1_1");
local core_file = require("core_file_l1_2_0");
local WinIoCtl = require("WinIoCtl");
local WinBase = require("WinBase");
local errorhandling = require("core_errorhandling_l1_1_1");
local FsHandles = require("FsHandles");
local Device = require("Device")

--[[
	ChangeJournal

	An abstraction for NTFS Change journal management
--]]
local ChangeJournal = {}
setmetatable(ChangeJournal, {
	__call = function(self, ...)
		return self:open(...);
	end,
});

local ChangeJournal_mt = {
	__index = ChangeJournal;
}

ChangeJournal.init = function(self, device)
	local obj = {
		Device = device;
	}
	setmetatable(obj, ChangeJournal_mt);

	local jinfo, err = obj:getJournalInfo();

	print("ChangeJournal.init, jinfo: ", jinfo, err)

	if jinfo then
		obj.JournalID = jinfo.UsnJournalID;
		obj.LowestUsn = jinfo.LowestValidUsn;
		obj.FirstUsn = jinfo.FirstUsn;
		obj.MaxSize = jinfo.MaximumSize;
		obj.MaxUsn = jinfo.MaxUsn;
		obj.AllocationSize = jinfo.AllocationDelta;
	end

	return obj;
end


ChangeJournal.open = function(self, driveLetter)
	local device, err = Device(driveLetter)

	if not device then
		print("ChangeJournal.open, ERROR: ", err)
		return nil, err
	end

	return self:init(device);
end


ChangeJournal.getNextUsn = function(self)
	local jinfo, err = self:getJournalInfo();

	if not jinfo then
		return false, err;
	end

	return jinfo.NextUsn;
end



ChangeJournal.getJournalInfo = function(self)
	local dwIoControlCode = FSCTL_QUERY_USN_JOURNAL;
	local lpInBuffer = nil;
	local nInBufferSize = 0;
	local lpOutBuffer = ffi.new("USN_JOURNAL_DATA");
	local nOutBufferSize = ffi.sizeof(lpOutBuffer);

	local success, err = self.Device:control(dwIoControlCode, 
          lpInBuffer,
          nInBufferSize,
          lpOutBuffer,
          nOutBufferSize);

	if not success then
		return false, errorhandling.GetLastError();
	end

	return lpOutBuffer;
end

function ChangeJournal.waitForNextEntry(self, usn, ReasonMask) 
 	usn = usn or self:getNextUsn();
 	local ReasonMask = ReasonMask or 0xFFFFFFFF;
 	local ReturnOnlyOnClose = false;
 	local Timeout = 0;
 	local BytesToWaitFor = 1;

    local ReadData = ffi.new("READ_USN_JOURNAL_DATA", {usn, ReasonMask, ReturnOnlyOnClose, Timeout, BytesToWaitFor, self.JournalID});

    local pusn = ffi.new("USN");
    
    -- This function does not return until the USN
    -- record exits
	local BUF_LEN = ffi.C.USN_PAGE_SIZE;
	local Buffer = ffi.new("uint8_t[?]", BUF_LEN);
    local dwBytes = ffi.new("DWORD[1]");

	local success, err = self.Device:control(FSCTL_READ_USN_JOURNAL, 
        ReadData,
        ffi.sizeof(ReadData),
        Buffer,
        BUF_LEN);

	if not success then 
		return false, err
	end

	local UsnRecord = ffi.cast("PUSN_RECORD", ffi.cast("PUCHAR",Buffer) + ffi.sizeof("USN")); 

    return UsnRecord;
end

return ChangeJournal;

This very much looks like the change journal I created a few months back. The primary difference is the device control stuff is abstracted out into the Device object, so it does not need to be repeated here.

When we want to track the changes to the device, we make repeated calls to ‘waitForNextEntry’.

local function test_waitForNextEntry(journal)
    local entry = journal:waitForNextEntry();

    while entry do
        printJournalEntry(entry);
        entry = journal:waitForNextEntry();
    end
end

This is your typical serially written code. There’s nothing that look special about it, no hint of async processing. Behind the covers though, way back in the Device:control() function, the actual sending of a command to the device happens using IO Completion Port, so if you’re running with TINN, this particular task will ‘waitForIO’, and other tasks can continue.

So, using it in context looks like this:

local function main()
    local journal, err = ChangeJournal("c:")

    spawn(test_waitForNextEntry, journal);
    periodic(function() print("tick") end, 1000)
end

run(main)

In this case, I spawn the journal waiting/printing thing as its own task. Then I setup a periodic timer to simply print “tick” every second, to show there is some activity.

Since the journaling is cooperative (mostly waiting for io control to complete), the periodic timer, or UI processing, or what have you, is free to run, without any hindrance.

Combine this with the already cooperative UI stuff, and you can imagine how easy it could be to construct the scenario I set out to construct. Since all networking and file system operations in TINN are automatically async, it would be easy to log these values, or send them across a network to be stored for later analysis or what have you.

And there you have it. Async everywhere makes for some very nice scenarios. Being able to do async on any device, whether with standard read/write operations, or io control, makes for very flexible programming.

Next time around, I’m going to show how to do async DNS queries for fun and profit.


Interface Obscura – DXGI, DXVA, MMDevice…

If you have programmed with Windows over the years, you have been subjected to a plethora of languages, paradigms, frameworks, APIs, tectonic shifts, and the like. Why is this so? Well, because Windows is fairly old, and stuff that worked 30 years ago still works today, mostly. Hardware has evolved, gained and changed capabilities, and along with these changes, so too have the frameworks that support it.

One interesting trend that has been occurring with Windows over the past few OS releases, is that things are becoming more modular, and more properly layered. For example, the API sets that now exist in the core of Windows properly layer and segregate APIs such that you don’t have to pull in the entire world just to do a conversion from an Ansi string to a multi-byte string.

There are some parts of the system that offer up more obscure interfaces that most programmers won’t experience directly. Recently I’ve been wanting to get at the core of audio and visual stuff. In Windows, you could be using the Media Foundation framework, or you would be using the older ‘Wave’ media framework. Both of these at their core are supported by the MMDevice library. That’s an obscure library which gives you things like the names of your audio endpoints, and various attributes about them. Ultimately you can open up an audio stream on a device and get bytes out of it.

Another one, a little less obscure is the Direct X Graphics Interface (DXGI). This one gives you access to low level information about the graphics adapters and capabilities of your machine. How is it that the control panel knows about your different monitors for example? It ultimately calls down to DXGI I would bet.

Yet another one is DXVA. This one, closely related to DXGI, gives you access to graphics ‘surfaces’. This is how you can actually do high speed rendering directly to the screen for example. You can also make a high speed screen capture program, without the use of mirror drivers, because you can get a handle on the actual graphics surface that’s being used to render the screen. This is how the application and desktop thumbnails are able to be rendered.

In most cases, these interfaces are steeped in the Component Object Model (COM). COM is a fairly mature part of the system which was created to ease the task of joining disparate programming languages and system components using a common runtime infrastructure. It is the dynamic type system of Windows if you will. COM served a world where there was no universal type system, and you had to be able to marshal data across disparate libraries and application domains. If I were to do it from scratch today, I’d probably use something like LuaJIT’s ffi and just write the interfaces in standard ‘C’ headers, but this came from a different time.

Recently I wanted to get a handle on the audio endpoints of my machine, and just print out the various attributes of each endpoint. Here’s the code:

local MMDevice = require("MMDevice")

for device in MMDevice:AudioEndpoints() do
  print("Device ID: ", device:getID())

  for property in device:properties() do
    print("  Property: ", property.vt, tostring(property))
  end
end

Audio endpoints are things like ‘microphone’ and ‘speaker’. They either render sound (speaker), or capture sound (microphone). The MMDevice:AudioEndpoints() function is a typical Lua iterator on the IMMDevice interfaces that represent the various endpoints in the system. Since this is an iterator, it can be used with the Lua Fun functional programming library to do all sorts of filtering, reductions, and the like. But, in this particular case, I just want to print out the various properties associated with the endpoint.

Each device has an iterator that will feed out the properties. That is the ‘properties()’ function. Again, a standard Lua iterator which will just spew out the properties one by one. In this case, the properties are represented by a very COM like VARIANT structure. When you look it up, the VARIANT, is nothing more than a giant union of all the base types that the COM system understands. In a modern language such as Lua, I can actually return the base type instead of the boxed up union, but for now it’s good enough.

Here, the VARIANT object has a simple __tostring() implementation, to turn some of the more common types into their string equivalents, it looks something like this:

local PROPVARIANT = ffi.typeof("PROPVARIANT")
local PROPVARIANT_mt = {
    __tostring = function(self)
        if self.vt == ffi.C.VT_BLOB then
            return tostring(self.blob)
        elseif self.vt == ffi.C.VT_BOOL then
            return not (self.boolVal == 0);
        elseif self.vt == ffi.C.VT_CLSID then
            return tostring(ffi.cast("CLSID *", self.puuid))
        elseif self.vt == ffi.C.VT_LPWSTR then
            return core_string.toAnsi(self.pwszVal)
        elseif self.vt == ffi.C.VT_UI4 then
            return tonumber(self.uintVal)
        end

        return tostring(self.pcVal)
    end,
}
ffi.metatype(PROPVARIANT, PROPVARIANT_mt)

There are various levels of “COM Support” within the APIs of the Windows system. Some are fairly basic, providing nothing more than a fairly thin veneer over an otherwise standard C interface. The Kinect interface is like this, as is the DXGI interface. There may not be any usage of things like VARIANT, and there is not much ‘inheritance’ of interfaces other than from ‘IUnknown’. Then, there are other interfaces that pull you further down into the COM rabbit hole. MMDevice is one such interface. You need VARIANT, and IDispatch (at least in a cursory way), IStorage, IPropertyStorage, and quite a few GUIDs along the way. Everything is done with GUIDs, and you have to worry about using “Release()” on your interfaces, CoMemoryFree() and a whole bunch of other stuff which, if not done properly, will lead to leaks, bugs, and worse.

And this is why the various frameworks exist. This knowledge is too obscure and esoteric for most developers to have to concern themselves with it. Better someone who really knows and understands the intricacies to deal with it, and provide a higher level abstraction that smooths over some of the rough spots.

Why not just make the lower level stuff more approachable in the first place? Well, yah, it’s that legacy thing isn’t it? In some cases you can make a complete break and just code something up from scratch. The Kinect API does this. I can get frames out of my Kinect without having to worry about any multimedia frameworks, and without pulling in half the COM world.

It took me a good couple of days of pulling on the COM thread to bring in just enough support to make these interfaces work. Now that the basics of COM are in the system though, it’s much easier to pull in more interfaces. Mostly it’s a tedious job of looking up all the interfaces, putting them into a ffi header file, coding up the stubs to make the viable calls, making sure the __gc method is implemented to call “Release()”, etc. At the end of it, you gain access to what is likely a fairly useful interface without all the fuss of dealing with the low level COM stuff.

I implemented the MMDevice interface, not just to gain access to the audio endpoints, but really as a practice run for implementing the DXVA interfaces. In Windows 8, the DXVA interfaces contain the desktop duplication API, which replaces the old mirror driver API for screen copying. Using this low level API, I will redo the plumbing on my screen capture application, and dramatically improve the performance browser based screen sharing. The basic screen capture portion should remain fairly small, at a couple hundred lines of code, but the capabilities will be dramatically improved, while reducing the bandwidth requirements, which will mean even faster frame rates. It’s all goodness.

And so it goes. Windows has a rich history of APIs, and with that history comes a lot of baggage that most developers probably don’t want to deal with. There are new, more modern APIs introduced all the time. LuaJIT forms a pretty easy bridge to some of the less trod parts of the system, and gives rapid scriptable access to the new parts. Once that bridge has been formed, you can do some pretty nice things within the Lua environment on Windows.


Multitasking single threaded UI – Gaming meets networking

There are two worlds that I want to collide. There is the active UI gaming world, then there’s the more passive networking server world. The difference between these two worlds is the kind of control loop and scheduler that they both typically use.

The scheduler for the networking server is roughly:

while true do
  waitFor(networkingEvent)
  
  doNetworkingStuff()
end

This is great, and works quite well to limit the amount of resources required to run the server because most of the time it’s sitting around idle. This allows you to serve up web pages with a much smaller machine than if you were running flat out, with say a “polling” api.

The game loop is a bit different. On Windows it looks something like this:

while true do
  ffi.fill(msg, ffi.sizeof("MSG"))
  local peeked = User32.PeekMessageA(msg, nil, 0, 0, User32.PM_REMOVE);
			
  if peeked ~= 0 then
    -- do regular Windows message processing
    local res = User32.TranslateMessage(msg)		
    User32.DispatchMessageA(msg)
  end

  doOtherStuffLikeRunningGamePhysics();
end

So, when I want to run a networking game, where I take some input from the internet, and incorporate that into the gameplay, I’ve got a basic problem. Who’s on top? Which loop is going to be THE loop?

The answer lies in the fact that the TINN scheduler is a basic infinite loop, and you can modify what occurs within that loop. Last time around, I showed how the waitFor() function can be used to insert a ‘predicate’ into the scheduler’s primary loop. So, perhaps I can recast the gaming loop as a predicate and insert it into the scheduler?

I have a ‘GameWindow’ class that takes care of creating a window, showing it on the screen and dealing with drawing. This window has a run() function which has the typical gaming infinite loop. I have modified this code to recast things in such a way that I can use the waitFor() as the primary looping mechanism. The modification make it look like this:

local appFinish = function(win)

  win.IsRunning = true
  local msg = ffi.new("MSG")

  local closure = function()
    ffi.fill(msg, ffi.sizeof("MSG"))
    local peeked = User32.PeekMessageA(msg, nil, 0, 0, User32.PM_REMOVE);
			
    if peeked ~= 0 then
      local res = User32.TranslateMessage(msg)
      User32.DispatchMessageA(msg)

      if msg.message == User32.WM_QUIT then
        return win:OnQuit()
      end
    end

    if not win.IsRunning then
      return true;
    end
  end

  return closure;
end

local runWindow = function(self)
	
  self:show()
  self:update()

  -- Start the FrameTimer
  local period = 1000/self.FrameRate;
  self.FrameTimer = Timer({Delay=period, Period=period, OnTime =self:handleFrameTick()})

  -- wait here until the application window is closed
  waitFor(appFinish(self))

  -- cancel the frame timer
  self.FrameTimer:cancel();
end

GameWindow.run = function(self)
  -- spawn the fiber that will wait
  -- for messages to finish
  Task:spawn(runWindow, self);

  -- set quanta to 0 so we don't waste time
  -- in i/o processing if there's nothing there
  Task:setMessageQuanta(0);
	
  Task:start()
end

Starting from the run() function. Only three things need to occur. First, spawn ‘runWindow’ in its own fiber. I want this routine to run in a fiber so that it is cooperative with other parts of the system that might be running.

Second, call ‘setMessageQuanta(0)’. This is a critical piece to get a ‘gaming loop’. This quanta is the amount of time the IO processing part of the scheduler will spend waiting for an IO event to occur. This time will be spent every time through the primary scheduler’s loop. If the value is set to 0, then effectively we have a nice runaway infinite loop for the scheduler. IO events will still be processed, but we won’t spend any time waiting for them to occur.

This has the effect of providing maximum CPU timeslice to various other waiting fibers. If this value is anything other than 0, let’s say ‘5’ for example, then the inner loop of the scheduler will slow to a crawl, providing no better than 60 iterations of the loop per second. Not enough time slice for a game. Setting it to 0 allows more like 3000 iteractions of the loop per second, which gives more time to other fibers.

That’s the trick of this integration right there. Just set the messageQuanta to 0, and away you go. To finish this out, take a look at the runWindow() function. Here just a couple of things are set in place. First, a timer is created. This timer will ultimately end up calling a ‘tick()’ function that the user can specify.

The other thing of note is the use of the waitFor(appCheck(self)). This fiber will block here until the “appCheck()” predicate returns true.

So, finally, the appFinish() predicate, what does that do?

Well, I’ll be darned if it isn’t the essence of the typical game window loop, at least the Windows message handling part of it. Remember that a predicate that is injected to the scheduler using “waitFor()” is executed every time through the scheduler’s loop, so the scheduler’s loop is essentially the same as the outer loop of a typical game.

With all this in place, you can finally do the following:


local GameWindow = require "GameWindow"
local StopWatch = require "StopWatch"

local sw = StopWatch();

-- The routine that gets called for any
-- mouse activity messages
function mouseinteraction(msg, wparam, lparam)
	print(string.format("Mouse: 0x%x", msg))
end

function keyboardinteraction(msg, wparam, lparam)
	print(string.format("Keyboard: 0x%x", msg))
end


function randomColor()
		local r = math.random(0,255)
		local g = math.random(0,255)
		local b = math.random(0,255)
		local color = RGB(r,g,b)

	return color
end

function randomline(win)
	local x1 = math.random() * win.Width
	local y1 = 40 + (math.random() * (win.Height - 40))
	local x2 = math.random() * win.Width
	local y2 = 40 + (math.random() * (win.Height - 40))

	local color = randomColor()
	local ctxt = win.GDIContext;

	ctxt:SetDCPenColor(color)

	ctxt:MoveTo(x1, y1)
	ctxt:LineTo(x2, y2)
end

function randomrect(win)
	local width = math.random(2,40)
	local height = math.random(2,40)
	local x = math.random(0,win.Width-1-width)
	local y = math.random(0, win.Height-1-height)
	local right = x + width
	local bottom = y + height
	local brushColor = randomColor()

	local ctxt = win.GDIContext;

	ctxt:SetDCBrushColor(brushColor)
	--ctxt:RoundRect(x, y, right, bottom, 0, 0)
	ctxt:Rectangle(x, y, right, bottom)
end


function ontick(win, tickCount)

	for i=1,30 do
		randomrect(win)
		randomline(win)
	end

	local stats = string.format("Seconds: %f  Frame: %d  FPS: %f", sw:Seconds(), tickCount, tickCount/sw:Seconds())
	win.GDIContext:Text(stats)
end



local appwin = GameWindow({
		Title = "Game Window",
		KeyboardInteractor = keyboardinteraction,
		MouseInteractor = mouseinteraction,
		FrameRate = 24,
		OnTickDelegate = ontick,
		Extent = {1024,768},
		})


appwin:run()

This will put a window on the screen, and draw some lines and rectangles at a frame rate of roughly 24 frames per second.

And thus, the two worlds have been melded together. I’m not doing any networking in this particular case, but adding it is no different than doing it for any other networking application. The same scheduler is still at play, and everything else will work as expected.

The cool thing about this is the integration of these two worlds does not require the introduction of multiple threads of execution. Everything is still within the same single threaded context that TINN normally runs with. If real threads are required, they can easily be added and communicated with using the computicles that are within TINN.


My Head In The Cloud – putting my code where my keyboard is

I have written a lot about the greatness of LuaJIT, coding for the internet, async programming, and the wonders of Windows. Now, I have finally reached a point where it’s time to put the code to the test.

I am running a service in Azure: http://nanotechstyles.cloudapp.net

This site is totally fueled by the work I have done with TINN. It is a static web page server with a couple of twists.

First of all, you can access the site through a pretty name:
http://www.nanotechstyles.com

If you just hit the site directly, you will get the static front page which has nothing more than an “about” link on it.

If you want to load up a threed model viewing thing, hit this:
http://www.nanotechstyles.com/threed.html

If you want to see what your browser is actually sending to the server, then hit this:
http://www.nanotechstyles.com/echo

I find the echo thing to be interesting, and I try hitting the site using different browsers to see what they produce.  This kind of feedback makes it relatively easy to do rapid turnarounds on the webpage content, challenging my assumptions and filling in the blanks.

The code for this web server is not very complex.  It’s the same standard ‘main’ that I’ve used in the past:

local resourceMap = require("ResourceMap");
local ResourceMapper = require("ResourceMapper");
local HttpServer = require("HttpServer")

local port = arg[1] or 8080

local Mapper = ResourceMapper(resourceMap);

local obj = {}

local OnRequest = function(param, request, response)
	local handler, err = Mapper:getHandler(request)


	-- recycle the socket, unless the handler explictly says
	-- it will do it, by returning 'true'
	if handler then
		if not handler(request, response) then
			param.Server:HandleRequestFinished(request);
		end
	else
		print("NO HANDLER: ", request.Url.path);
		-- send back content not found
		response:writeHead(404);
		response:writeEnd();

		-- recylce the request in case the socket
		-- is still open
		param.Server:HandleRequestFinished(request);
	end

end

obj.Server = HttpServer(port, OnRequest, obj);
obj.Server:run()

In this case, I’m dealing with the OnRequest() directly, rather than using the WebApp object.  I’m doing this because I want to do some more interactions at this level that the standard WebApp may not support.

Of course the ‘handlers’ are where all the fun is. I guess it makes sense to host the content of the site up on the site for all to see and poke fun at.

My little experiment here is to give my code real world exposure, with the intention of hardening it, and gaining practical experience on what a typical web server is likely to see out in the wild.

So, if you read this blog, go hit those links. Soon enough, perhaps I will be able to serve up my own blog using my own software. That’s got a certain circular reference to it.


ReadFile – The Good, the bad, and the async

If you use various frameworks on any platform, you’re probably an arm’s length away from the nasty little quirks of the underlying operating system.  If you are the creator of such frameworks, the nasty quirks are what you live with on a daily basis.

In TINN, I want to be async from soup to nuts.  All tcp/udp, socket stuff is already that way.  Recently I’ve been adding async support for “file handles”, and let me tell you, you have to be very careful around these things.

In the core windows APIs, in order to read from a file, you do two things.  You first open a file using the CreateFile(), function.  This may be a bit confusing, because why would you use “create” to ‘open’ an existing file?  Well, you have to think of it like a kernel developer might.  From that perspective, what you’re doing is ‘create a file handle’.  While you’re doing this, you can tell the function whether to actually create the file if it doesn’t exist already, open only if it exists, open read-only, etc.

The basic function signature for CreateFile() looks like this:

HANDLE WINAPI CreateFile(
  _In_      LPCTSTR lpFileName,
  _In_      DWORD dwDesiredAccess,
  _In_      DWORD dwShareMode,
  _In_opt_  LPSECURITY_ATTRIBUTES lpSecurityAttributes,
  _In_      DWORD dwCreationDisposition,
  _In_      DWORD dwFlagsAndAttributes,
  _In_opt_  HANDLE hTemplateFile
);

Well, that’s a mouthful, just to get a file handle. But hay, it’s not much more than you’d do in Linux, except it has some extra flags and attributes that you might want to take care of. Here’s where the history of Windows gets in the way. There is a much simpler function “OpenFile()”, which on the surface might do what you want, but beware, it’s a lot less capable, a leftover from the MSDOS days. The documentation is pretty clear about this point “don’t use this, use CreateFile instead…”, but still, you’d have to wade through some documentation to reach this conclusion.

Then, the ReadFile() function has this signature:

BOOL WINAPI ReadFile(
  _In_         HANDLE hFile,
  _Out_        LPVOID lpBuffer,
  _In_         DWORD nNumberOfBytesToRead,
  _Out_opt_    LPDWORD lpNumberOfBytesRead,
  _Inout_opt_  LPOVERLAPPED lpOverlapped
);

Don’t be confused by another function, ReadFileEx(). That one sounds even more modern, but in fact, it does not support the async file reading that I want.

Seems simple enough. Take the handle you got from CreateFile(), and pass it to this function, including a buffer, and you’re done? Well yah, this is where things get really interesting.
Windows supports two forms of IO processing. Async, and synchronous. The Synchronous case is easy. You just make your call, and your thread will be blocked until the IO “completes”. That is certainly easy to uderstand, and if you’re a user of the standard C library, or most other frameworks, this is exactly the behaviour you can expect. Lua, by default, using the standard io library will do exactly this.

The other case is when you want to do async io. That is, you want to initiate the ReadFile() and get an immediate return, and handle the processing of the result later, perhaps with an alert on an io completion port.

Here’s the nasty bit. This same function can be used in both cases, but has very different behavior. It’s a subtle thing. If you doing synchronous, then the kernel will track the fileposition, and automatically update it for you. So, you can do consecutive ReadFile() calls, and read the file contents from beginning to end.

But… When you do things async, the kernel will not track your file pointer. Instead, you must do this on your own! When you do async, you pass in a instance of a OVERLAPPED structure, wich contains things like a pointer to the buffer to be filled, as well as the size of the buffer. This structure also contains things like the offset within the file to read from. By default, the offset is ‘0’, which will have you reading from the beginning of the file every single time.

typedef struct _OVERLAPPED {
    ULONG_PTR Internal;
    ULONG_PTR InternalHigh;
    union {
        struct {
            DWORD Offset;
            DWORD OffsetHigh;
        };

        PVOID Pointer;
    };

    HANDLE hEvent;
} OVERLAPPED, *LPOVERLAPPED;

You have to be very careful and diligent with using this structure, and the proper calling sequences. In addition, if you’re going to do async, you need to call CreateFile() with the appropriate OVERLAPPED flag. In TINN, I have created the NativeFile object, which pretty much deals with all this subtlety. The NativeFile object presents a basic block device interface to the user, and wraps up all that subtlety such that the interface to files is clean and simple.

-- NativeFile.lua

local ffi = require("ffi")
local bit = require("bit")
local bor = bit.bor;

local core_file = require("core_file_l1_2_0");
local errorhandling = require("core_errorhandling_l1_1_1");
local FsHandles = require("FsHandles")
local WinBase = require("WinBase")
local IOOps = require("IOOps")

ffi.cdef[[
typedef struct {
  IOOverlapped OVL;

  // Our specifics
  HANDLE file;
} FileOverlapped;
]]

-- A win32 file interfaces
-- put the standard async stream interface onto a file
local NativeFile={}
setmetatable(NativeFile, {
  __call = function(self, ...)
    return self:create(...);
  end,
})

local NativeFile_mt = {
  __index = NativeFile;
}

NativeFile.init = function(self, rawHandle)
	local obj = {
		Handle = FsHandles.FsHandle(rawHandle);
		Offset = 0;
	}
	setmetatable(obj, NativeFile_mt)

	if IOProcessor then
		IOProcessor:observeIOEvent(obj:getNativeHandle(), obj:getNativeHandle());
	end

	return obj;
end

NativeFile.create = function(self, lpFileName, dwDesiredAccess, dwCreationDisposition, dwShareMode)
	if not lpFileName then
		return nil;
	end
	dwDesiredAccess = dwDesiredAccess or bor(ffi.C.GENERIC_READ, ffi.C.GENERIC_WRITE)
	dwCreationDisposition = dwCreationDisposition or OPEN_ALWAYS;
	dwShareMode = dwShareMode or bor(FILE_SHARE_READ, FILE_SHARE_WRITE);
	local lpSecurityAttributes = nil;
	local dwFlagsAndAttributes = bor(ffi.C.FILE_ATTRIBUTE_NORMAL, FILE_FLAG_OVERLAPPED);
	local hTemplateFile = nil;

	local rawHandle = core_file.CreateFileA(
        lpFileName,
        dwDesiredAccess,
        dwShareMode,
    	lpSecurityAttributes,
        dwCreationDisposition,
        dwFlagsAndAttributes,
    	hTemplateFile);

	if rawHandle == INVALID_HANDLE_VALUE then
		return nil, errorhandling.GetLastError();
	end

	return self:init(rawHandle)
end

NativeFile.getNativeHandle = function(self)
  return self.Handle.Handle
end

-- Cancel current IO operation
NativeFile.cancel = function(self)
  local res = core_file.CancelIo(self:getNativeHandle());
end

-- Close the file handle
NativeFile.close = function(self)
  self.Handle:free();
  self.Handle = nil;
end

NativeFile.createOverlapped = function(self, buff, bufflen, operation, deviceoffset)
	if not IOProcessor then
		return nil
	end

	fileoffset = fileoffset or 0;

	local obj = ffi.new("FileOverlapped");

	obj.file = self:getNativeHandle();
	obj.OVL.operation = operation;
	obj.OVL.opcounter = IOProcessor:getNextOperationId();
	obj.OVL.Buffer = buff;
	obj.OVL.BufferLength = bufflen;
	obj.OVL.OVL.Offset = deviceoffset;

	return obj, obj.OVL.opcounter;
end

-- Write bytes to the file
NativeFile.writeBytes = function(self, buff, nNumberOfBytesToWrite, offset, deviceoffset)
	fileoffset = fileoffset or 0

	if not self.Handle then
		return nil;
	end

	local lpBuffer = ffi.cast("const char *",buff) + offset or 0
	local lpNumberOfBytesWritten = nil;
	local lpOverlapped = self:createOverlapped(ffi.cast("uint8_t *",buff)+offset,
		nNumberOfBytesToWrite,
		IOOps.WRITE,
		deviceoffset);

	if lpOverlapped == nil then
		lpNumberOfBytesWritten = ffi.new("DWORD[1]")
	end

	local res = core_file.WriteFile(self:getNativeHandle(), lpBuffer, nNumberOfBytesToWrite,
		lpNumberOfBytesWritten,
  		ffi.cast("OVERLAPPED *",lpOverlapped));

	if res == 0 then
		local err = errorhandling.GetLastError();
		if err ~= ERROR_IO_PENDING then
			return false, err
		end
	else
		return lpNumberOfBytesWritten[0];
	end

	if IOProcessor then
    	local key, bytes, ovl = IOProcessor:yieldForIo(self, IOOps.WRITE, lpOverlapped.OVL.opcounter);
--print("key, bytes, ovl: ", key, bytes, ovl)
	    return bytes
	end
end

NativeFile.readBytes = function(self, buff, nNumberOfBytesToRead, offset, deviceoffset)
	offset = offset or 0
	local lpBuffer = ffi.cast("char *",buff) + offset
	local lpNumberOfBytesRead = nil
	local lpOverlapped = self:createOverlapped(ffi.cast("uint8_t *",buff)+offset,
		nNumberOfBytesToRead,
		IOOps.READ,
		deviceoffset);

	if lpOverlapped == nil then
		lpNumberOfBytesRead = ffi.new("DWORD[1]")
	end

	local res = core_file.ReadFile(self:getNativeHandle(), lpBuffer, nNumberOfBytesToRead,
		lpNumberOfBytesRead,
		ffi.cast("OVERLAPPED *",lpOverlapped));

	if res == 0 then
		local err = errorhandling.GetLastError();

--print("NativeFile, readBytes: ", res, err)

		if err ~= ERROR_IO_PENDING then
			return false, err
		end
	else
		return lpNumberOfBytesRead[0];
	end

	if IOProcessor then
    	local key, bytes, ovl = IOProcessor:yieldForIo(self, IOOps.READ, lpOverlapped.OVL.opcounter);

    	local ovlp = ffi.cast("OVERLAPPED *", ovl)
    	print("overlap offset: ", ovlp.Offset)

--print("key, bytes, ovl: ", key, bytes, ovl)
	    return bytes
	end

end

return NativeFile;

This is enough of a start. If you want to simply open a file:

local NativeFile = require("NativeFile")
local fd = NativeFile("sample.txt");

From there you can use readBytes(), and writeBytes(). If you want to do streaming, you can feed this into the new and improved Stream class like this:

local NativeFile = require("NativeFile") 
local Stream = require("stream") 
local IOProcessor = require("IOProcessor")

local function main()

  local filedev, err = NativeFile("./sample.txt", nil, OPEN_EXISTING, FILE_SHARE_READ)

  -- wrap the file block device with a stream
  local filestrm = Stream(filedev)

  local line1, err = filestrm:readLine();  
  local line2, err = filestrm:readLine();  
  local line3, err = filestrm:readLine()

  print("line1: ", line1, err)  
  print("line2: ", line2, err)  
  print("line3: ", line3, err) 
end

run(main)

The Stream class looks for readBytes() and writeBytes(), and can provide the higher level readLine(), writeLine(), read/writeString(), and a few others. This is great because it can be fed by anything that purports to be a block device, which could be anything from an async file, to a chunk of memory.

And that’s about it for now. There are subtleties when dealing with async file access in windows. Having a nice abstraction on top of it gives you all the benefits of async without all the headaches.

 


Its About Time – TINN Timers

The last time I wrote about time: Hurry Up and Wait –  TINN Timing the focus was on introducing the wait() function and explaining how that fits into the TINN scheduler in general.

Having a wait() function is a great low level primitive.  It allows you to pause execution for an amount of time, in a synchronous manner.  That will work well when you’re doing something serially and need to take break at specified intervals.  What if your usage pattern is more complex though?  More likely than not, if you’re writing a web server of some sort, you’ll be doing this sort of sequence:
executeasynccodethattakestime()

if codeNotDoneAfter sometime then

cancel code

end

do other stuff while waiting

Basically, I need to do something like send off a web request.  If the request has not been satisfied within a specified amount of time, I want to cancel the operation.  I need to be able to do this asynchronously because I may have many requests in flight.  So, what to do?  I obviously need a Timer of some sort that will deal with this for me.

local Task = require("IOProcessor")

local Timer = {}
setmetatable(Timer, {
  __call = function(self, ...)
    return self:create(...)
  end,
});

local Timer_mt = {
  __index = Timer;
}

Timer.init = function(self,params)
  local obj = {
    Delay = params.Delay;
    Period = params.Period;
    Running = false;
    TimerFunc = params.TimerFunc;
  }
  setmetatable(obj, Timer_mt);

  if self.Running then
    obj:start()
  end

  return obj;
end

Timer.create = function(self, ...)
  return self:init(...);
end

Timer.start = function(self)
  self.Running = true;

  local function closure()
    if self.Delay then
      wait(self.Delay);
      self.TimerFunc(self);
    end

    if not self.Period then
      return
    end

    while self.Running do
      wait(self.Period)
      self.TimerFunc(self);
    end
  end

  spawn(closure);
end

Timer.stop = function(self)
  self.Running = false;
end

return Timer

To dissect, basically an object that provides easy wrapper convenience for the wait() function. You specify an initial delay, and a period and call the start() function. Start will spawn the actual function that is involved in doing the waiting and executing of the specified function.

Here is a simple use case:

-- test_Timer.lua

local Timer = require("Timer")

local function printThis(timer)
  print("this")
end

local function main()
  local timer = Timer {Delay = 1*1000, Period = 300, TimerFunc = printThis}

  timer:start();

  -- wait a few seconds, then stop the time
  print("wait 4 seconds...")
  wait(4*1000)
  print("stop timer")
  timer:stop(1000*4);

  -- Wait a few more seconds then exit
  print("wait 2 seconds...")
  wait(1000*2);
end

run(main)

In this case, I create a timer that has an initial 1 second delay, and a period of 300 milliseconds. So, after the initial delay, the printThis() function will be called. Then every 300 milliseconds after that, it will be called again.

In the sample, the timer is started, which causes it to run independently of the main task. Within the main task, wait 4 seconds, then call the stop() function on the time. Wait two more seconds, and finally exit altogether. This shows that a timer can run independently. The function that is called can be anything. If you want the function to have some parameters, it is itself probably a closure (function within a function). Additionally, since the function is passed the timer as the only parameter, it can cause the timer to stop. Here’s another example where a timer is running, and will stop after a certain number has been reached.

local count = 0;
local function counter(timer)
  count = count + 1;
  if count >= 5 then
    timer:stop();
  end
  print("counter: ",count)
end

local testCounter = function()
  local timer = Timer {Period = 200, TimerFunc = counter, Running=true}
end

run(testCounter)

The counter function is simple. Basically, increment a counter. Once it reaches 5, stop the time. Starting the counter in the first place is also fairly easy. Just set a period (forget the initial delay), tell it which function is to be executed every period, and set it to run automatically (without requiring an explicit ‘start()’).

This will call the function once every 200 milliseconds, and then quit. Nice to have.

With this little component in hand, you can probably imagine how I/O might be accomplished with timeouts. Basically, any I/O operation would look like:

function cancel(ioperator)
  local function closure(timer)
    timer:stop();
    ioperator:cancel();
  end
  return closure;
end

op = startioop(someparams)
timer=Timer{Delay=1000, TimerFunc=cancel(op), Running=true}

Here, a timer is created with a delay of one second. After one second, the timer fires and the cancel.closure() function is called. The operation is cancelled, if it’s not already done, and that’s the end of things. This assumes that there is no harm in canceling an already finished transaction.

Well, that’s just great. I/O with timeouts fully implemented in user space, asynchronous, and all that. This is a great thing for timers, io, and TINN code in general.

The only thing missing is the ability to do async on more than just sockets. So, next up is async ‘file’ I/O in general, of which sockets are a specialized subset.


Exposing Yourself to the interwebs – Deciding OS configurations

Although I work for a “high tech” company, I am by no means a system administrator.  I do not consider myself to be a seasoned IT pro.  I am a coder/architect/hacker, raised in the finest traditions of getting things done by flinging code at problems.  So, here I am contemplating how best to properly configure my various machines on hand to achieve a variety of goals.  What to do?

Some goals I have.  I want to run a directory service at home.  For my personal domain (nanotechstyles.com), I want to have a local Active Directory, and within my house I want to ‘domain join’ my various machines.  I started out by building my Shuttle PC, and putting the Hyper-V Server 2012 on it.  The plan here was to then install a version of Windows Server 2012 R2.  Here’s a sidetrack though…

Did you know that Hyper-V Server is essentially a free OS?  Whah?  Yah, it is the core hypervisor, plus an instance of server core.  That’s just enough to get you going so that you can install stuff into VMs like Windows 8, Windows Server, and the like.  But, it’s actually enough to run the likes of TINN.  So, I can totally run a whole set of networking services with nothing more than the Hyper-V Server on the box.  Given that Linux is usually playing the role of “free bare bones OS, just SSH into the box”, I find it a bit funny that Microsoft has essentially the same thing, but it’s not advertised as such.

At any rate, I first installed Hyper-V Server, which has no UI to speak of.  You can configure this thing using PowerShell.  If you’re IT saavy, you can probably whip together the various scripts required to get this setup going with all sorts of VMs and the like.  I set this box up a week ago, and played around from the command line for a while.  I browsed the interwebs looking for scripts and whatnot to help me get the OS instances installed.  Now I’m reconsidering my choice.  All I’m going to do is install an instance of Windows Server 2012 R2 on the machine, so why bother with this bare bones configuration in the first place?  I must admit my CLI foo is weak.  I craves me a GUI to fondle and fiddle with.  So, I think I’m going to repave the machine, and just go ahead and install Server 2012 R2 on it as the primary thing.  I still get Hyper-V anyway, so I can still put other OS images on the box, but I can cut the nonsense with the CLI and just get on with my business.

At the same time, I ‘refurbished’ the tower that the Shuttle PC replaced.  I’ve determined that although this box is 4 years old, it’s still quite serviceable.  I took out the primary hard disk, and stuck it into one of those aluminum disk cases so now it’s serving as an external USB 3.0 disk drive.  I stuffed a Samsung 256GB SSD drive into the tower, and repaved with the just released Windows 8.1.  Although I already had some graphics cards in the machine previously, I added another dual monitor card so I could have 3 displays connected to the thing.

What’s this machine to be used for?  Well, I didn’t upgrade any other parts of the configuration, so, it still only has 8Gb of RAM.  I figure since I’m not a gamer, this will make a sweet setup as my “command central”.  I can program on this machine.  With the three monitors, it gives me: 1 (portrait mode) Sublime Text, 2 (landscape) web browsing, etc, 2 (landscape) Netflix.  Now I don’t have to watch Netflix on the Mac while programming on the PC.  I can just have that sitting off to the side on one of the three monitors.

Using 8.1 is interesting.  I have it on the Surface RT, and there it’s only used for playing a few games and browsing the internet.  For a tablet, it’s fine.  It gets the job done.  On my work laptop, I have 8.0, and like most people introduced to something new, it was kind of annoying at first.  The laptop has a touch screen, and I tend to touch it every once in a while to do a swipe or two.  For some reason though, Lenovo just can’t get the touch pad right, so it’s caused more annoyance than help, so I’ve turned off the touchpad swiping, which is kind of a bummer.

Now, on the big rig, having 8.1 on a triple monitor display, where touch is not a factor, what’s the experience?  Well, the key is to get used to the lucky charms.  Those magical parts of the screen where the mouse tracking will bring up one set of ‘charms’ or another.  I’ve just now learned that to close apps, I can move the mouse to the upper left corener, and right-click on the presented thumbnail to close the app.  Eh.  Nice to know, not changing my life significantly, but convenient at times.

It’s also true that you can bring up the “start” screen on any of the three monitors.  This is actually kind of convenient as I can leave one screen with the start screen up, and use that to launch whatever apps I want with a quick click on an icon.

So, this setup is turning out to be workable.  It’s not a question of “is it better than the mac” for me.  It’s about using whatever tools are available to do the job.  To run my home domain, Windows Server is a good choice.  As my primary development desktop, although I’ll use the Mac to do Android, iOS, MacOS, and Linux, my tower does great for Windows development, and I actually prefer sitting in front of these three monitors to get a lot of other stuff done as well.

There you have it then.  Lack of Command Line Interface skills leads me to install a UI enabled Windows Server 2012 as the domain controller et al.  My desktop, is Windows 8.1.

 


The Innevitable March of Technology – Surface 2

This morning I was reading the live blog stream of the Surface 2 launch by Microsoft. It should be a well known fact that I work for Microsoft. I am obviously a supporter of the company, but I don’t consider myself to be an apologist.

At any rate, as I was watching the launch, reading the specs, thinking about what it all means, I had the following experience.

The have an instance of the Surface RT. Just to give you some perspective as to how pragmatic I am, the only real complaint I have about it is the kick stand does not stay in place so that you can reverse the angle and read it while it’s sitting on the counter top, and you’re standing up looking down at it. Other than that, whether it has 10, 10,000, or 100,000 apps available doesn’t really matter to me. I use it to browse the internet, and to play that bubble breaking game. I occasionally use it to take pictures around the house, so it’s sync’d with SkyDrive, so that I can easily get the pictures from my desktop machine.

In our house, having just moved recently, there are literally 10s of devices which have varying forms of compute capabilities. We have everything from simple microcontrollers, to high end laptops, and “server” machines. What I’ve come to realize is that they all serve some purpose, and we no longer live in a world of scarce compute resources. You can literally purchase single purpose devices for almost all of your electronics needs.

My phone gets the most usage. Email/Calendar is the number one ‘app’, with web browsing in a relatively distant 3rd, unless we’re out shopping. Texting would be a very distant 4th, and bringing up the rear would be actually making phone calls. Photographing falls in there somewhere depending on whether we’re on vacation.

The Surface RT gets used for web browsing, particularly when I’m idling around the living room not using the TV.

The wife has an iPad, and it’s #1 usage is youtube video watching. An extremely distant second would be taking some pictures around the house. It never travels with us.

We have an appleTV (for airPlay from the iPad), and a roku 3 attached to the living room “tv”. 99.8% of our video watching comes through the roku box. Recently I added a digital antenna so my mom could watch football, so that’s the other 0.2%. We only have cable to get the internet. We do not have any standard cable channels.

There is a wifi client access point connected to the tv because it has an ethernet port on the back, but I prefer not to string ethernet cables. So, that’s another dedicated device. The access point is essentially the same as a tp-link mini router.

My home office has the “big iron”. Lenovo X1 Carbon. This is my Microsoft work machine. I no longer use a desktop machine at work because laptops are good enough in terms of compiling power, and it’s way more portable.

Then there’s this MacBook Pro. I write iOS, MacOS, Linux, and Windows software, so this one laptop gives me all three OSs in one tidy little box.

The noisy beast under the desk is a machine I built about 3 years ago. Intel quad core something or other. I actually rarely turn it on these days. Any storage purpose it was serving is now taken up by the Synology 1512+ sitting quietly on the shelf.

One interesting thread over the past couple of years has been following the tracks of hardkernel and their Odroid machines. I started with the Odroid-X, then X2, U2, and most recently the XU. The XU is really cool with quad A15/A7 chip. Great for trying out Android code, and for fiddling with various Linux distros as well.

So, with all this, what about the new Surface 2? I want to get one of those, and am willing to pay my own money for it. Why? My wife has the iPad, and that suits her youTube needs. I have need for a fairly robust and transportable compute platform that I can use in my garage shop. I don’t want a ‘laptop’, because given their combination of permanently attached keyboard, and relatively limited mounting/viewing options, they’re not practical. Neither is a fixed ‘desktop’ machine.

I’ll use it in the shop for a couple of things. In the past, I’ve had a laptop attached to my various 3D printers and automated router. Now that I have USB capabilities, because I use tinyG for my automation controller, I can use the Surface 2 as the controlling device. I can’t use the iPad, because it doesn’t do USB.

They’re not quite cheap enough to have one per automated device, but it is cheap enough to have a couple of them.

What else? I can see attaching a couple of them to the wall in the shop to do simple controls like powering on sections of the shop, the filtration system, etc. And lastly, of course I’ll be mounting a projector in there, and it will likely be powered by a roku box. But, I’ll use the Surface 2 as the controller of that AV system. It will be much better than having a remote because it will present big buttons on a grime cleanable screen, which is perfect for a shop.

As for the new surface pro, I honestly don’t know how I would use that particular device, and that’s fine by me. I’ve already got the work laptop, so I would not be using it to compile for work. I could use it as my dedicated home “PC” because for the moment I use the desktop machine to run TurboTax, and it would be nice to get that app off that particular machine.

Bottom line for me, there’s some interesting things to be had from these new devices, so I’m going to invest a bit to get a couple. We no longer live in a time where this is a super huge life changing “we better save up for years and justify it for the whole family” sort of decision. These devices will serve a purpose in the short term, and in the future, something else will be developed which will serve another purpose.

One thing I will note though. On the Surface 2, probably the most impressive feature to me as a tech head is the multi-sensor keyboard. It has 1,100 sensors on it! It’s essentially a giant touch pad, that has some tactile keys on its surface. I think that might have some interesting possibilities. Also, the low light camera seems really useful. I’m not one for video calling, and I’d prefer the back camera had this ability, but it is one feature I look for, even for a regular camera.

And so it goes.


How About that Web Server Again?

Has it really been more than a month?

Well, you see, I bought a house, packed, moved, took daughter to college, and…

The last time I wrote about making a simple WebServer, I left the job half finished. I showed that through layering, from a basic socket server, you could then build a simple http handler, which had not much more than an ‘OnRequest’ function.

That works great, and gives you the right level of access if you’re going to do things with http in general, that are beyond simple static file serving. But what if your interactions are more complex than you care to deal with using simple ‘if-then-else’ logic.

This time around, I’m going to use a new and improved form of the WebApp which first of all deals with the new and improved IOCP based sockets, and leverages the HttpServer object of yore.

Here is the basic web server:

-- main.lua
local WebApp = require("WebApp")
local resourceMap = require("ResourceMap");

local port = arg[1] or 8080

local app = WebApp(resourceMap, port);
app:run();

And you simply invoke it with:

tinn main.lua 8080

Well, that seems easy enough. If you wanted to get real crazy, and win an obfuscated code competition, you could do it in one line:

-- main.lua
require("WebApp")(require("ResourceMap"), arg[1] or 8080):run()

Or, if you’re a real Lua head, you could do it all on the command line, without even using the ‘main.lua’ file:

tinn -e "require('WebApp')(require('ResourceMap'), arg[1] or 8080):run()"

That’s all fun and games, and it’s cool to see that you can implement a web server in a single command line. But what’s in that magical ResourceMap file?

local URL = require("url");
local StaticService = require("StaticService");

local function replacedotdot(s)
  return string.gsub(s, "%.%.", '%.')
end

local HandleFileGET = function(request, response)
  local absolutePath = replacedotdot(URL.unescape(request.Url.path));
  local filename = './wwwroot'..absolutePath;

  StaticService.SendFile(filename, response)
  return false;
end

local ResourceMap = {
  ["/"] = {name="/",
    GET = HandleFileGET,
  };
}

return ResourceMap;

The idea here is that we want to route any ‘GET’ methods to the HandleFileGET() function. There is a ResourceMapper object within tinn that utilizes a structurce such as the one in ResourceMap. The general layout is {[pattern] = {name=””, METHOD = function [,METHOD = function]},}

Using this simple mechanism, you can easily route the handling of a particular resource request to a particular function.

This table can have entries that are much more interesting. For example, if you want to handle the retrieval of ‘favicon.ico’ differently than other files, you can just add a specific mapping for that.

local ResourceMap = {
  ["/"] = {name="/",
    GET = HandleFileGET,
  };

  ["/favicon.ico"] = {name="/favicon.ico",
    GET = HandleFaviconGET,
  };
}

You could have multiple methods per resource as well:

local ResourceMap = {
  ["/"] = {name="/",
    GET = HandleFileGET,
    POST = HandleFilePOST,
  };

  ["/favicon.ico"] = {name="/favicon.ico",
    GET = HandleFaviconGET,
  };
}

In general, the longest prefix match algorithm is applied to whatever is supplied within the resource field of the request. If you want to deal with all methods of a particular resource, without having to specify them explicitly, then you can use the magic method ‘_DEFAULT’.

local ResourceMap = {
  ["/"] = {name="/",
    GET = HandleFileGET,
    POST = HandleFilePOST,
    _DEFAULT = HandleFileDEFAULT,
  };

  ["/favicon.ico"] = {name="/favicon.ico",
    GET = HandleFaviconGET,
  };
}

In this way, if there is a request for a resource, and a method that we don’t know about at the time of creating the map, the HandleFileDEFAULT() function will be called to deal with it. That might be handy in the case where you’d like to handle these unknown method requests in a generic way, or you might want to change it over time without having to change your mapping.

Another case is when the resource isn’t actually a resource. Take for example a ‘CONNECT’ request:

CONNECT localhost:9000

In this case, the ‘resource’ does not start with a ‘/’, so the longest prefix match won’t land it on anything in our map. I need to deal with these with a different pattern. Well, the pattern part of the map is nothing more than a standard pattern in the Lua sense of the word, so a ‘.’ will l match any character. The following map will do the trick.

local ResourceMap = {
  ["."] = {name=".",
    CONNECT = HandleCONNECT,
  };

  ["/"] = {name="/",
    GET = HandleFileGET,
    POST = HandleFilePOST,
  };

  ["/favicon.ico"] = {name="/favicon.ico",
    GET = HandleFaviconGET,
  };
}

In this way, we’ll deal with the CONNECT method if it shows up.

This is an affirmative list. If there is a match to one of the patterns, then the mapped function is executed. If no pattern is found, either because the resource itself does not match, or because the resource does not have a function to handle the specified method, then a general 404 error is returned.

That’s about all there is to it. Create a mapping between URLs and functions, and you’re all done. Of course there’s the function itself:

local URL = require("url");
local StaticService = require("StaticService");

local function replacedotdot(s)
  return string.gsub(s, "%.%.", '%.')
end

local HandleFileGET = function(request, response)
  local absolutePath = replacedotdot(URL.unescape(request.Url.path));
  local filename = './wwwroot'..absolutePath;
	
  StaticService.SendFile(filename, response)

  return false;
end

Not a lot of magic in this particular one. First of all, there’s that simplistic ‘replacedotdot()’ function. That’s just a casual attempt to restrict the file access to the directory of our choosing. In HandleFileGET, the first thing to do is urldecode the path specified, then feed that to the replacedotdot() function. Then, take whatever comes out of that, and prepend it with ‘./wwwroot’, and finally feed that to the StaticService.SendFile() function, which is a standard part of TINN.

The return value of this function has meaning. If you return false, or ‘nil’, then the socket representing this particular request will be recycled, assuming there is potentially another request coming on the same socket.

If you instead return ‘true’, then the system assumes you are handling any subsequent recycling, or closing, and it will not take any further action with the underlying socket.

This is a nice feature in that it allows you to construct much more interesting interactions than the standard request/response, without much fuss. For example, you could easily open up websockets, upgrade connections in general, or do whatever you want.

At any rate, with TINN, you can create a simple web server in a single command line. I find that to be a fairly useful thing.


Hurry Up and Wait – TINN Timing

Moving right along. First I needed to do basic networking. Starting at the lowest level of socket interaction, advancing up the stack through specific TCP and HTTP uses. Then back down to UDP.

With basic async networking covered, the next thing that comes up is timing. The general socket IO is covered. You can basically build an entire service using nothing more than asynchronous socket calls. But, most servers are more interesting than that. There are situations where you’ll want to cancel out an async operation if it’s taking too long, or you might want to perform some operation over time, repeatedly. So, clearly the TINN system needs some concept of time management.

Here’s the kind of code I would like to write in order to do something every 500 milliseconds:

require ("IOProcessor");

local test_wait = function(interval)
  while true do
    wait(interval);
    print(string.format("interval: %d", IOProcessor.Clock:Milliseconds()));
  end
end

run(test_wait)

Basically, there is a new ‘wait()’ function. You give it a number of milliseconds, and it will suspend the coroutine you’re currently in for the given amount of time. This capability comes courtesy of some changes to the base scheduler. The changes are the following:

wait = function(millis)
  local nextTime = IOProcessor.Clock:Milliseconds() + millis;
  return IOProcessor:yieldUntilTime(nextTime);
end

IOProcessor.yieldUntilTime = function(self, atime)
  if self.CurrentFiber ~= nil then
    self.CurrentFiber.DueTime = atime;
    tabutils.binsert(self.FibersAwaitingTime, self.CurrentFiber, compareTaskDueTime);

    return self:yield();
  end
  return false;
end

The yieldUntilTime() function will take the currently running fiber (coroutine) and put it into the list of FibersAwaitingTime. This is simply a table which is maintained in sorted order from lowest to highest. Once a fiber is placed on this list, it is no longer in the list of currently active fibers. it will sit on this list until it’s DueTime has passed.

The main scheduling loop will step through the fibers that are sitting in the AwaitingTime list using the following:

IOProcessor.stepTimeEvents = function(self)
  local currentTime = self.Clock:Milliseconds();

  -- traverse through the fibers that are waiting
  -- on time
  local nAwaiting = #self.FibersAwaitingTime;
  for i=1,nAwaiting do

    local fiber = self.FibersAwaitingTime[1];
    if fiber.DueTime <= currentTime then
      -- put it back into circulation
      fiber.DueTime = 0;
      self:scheduleFiber(fiber);

      -- Remove the fiber from the list of fibers that are
      -- waiting on time
      table.remove(self.FibersAwaitingTime, 1);
    end
  end
end

Basically, step through the list of fibers that are waiting for their wait time to expire. For all those that qualify, put them back into the list of active fibers by calling the ‘shcheduleFiber()’ function.

This begins to get very interesting I think. Of course once you create a timer, or even async i/o, you probably also want the ability to cancel such operations. But, that’s another matter.

Doing this little exercise, the scheduler begins to take on some more of the attributes of what you might find in the core OS. The big difference is that everything is done in the Lua space, and does not rely on OS primitives, so it is actually somewhat portable to other architectures. That’s a nice idea to have such a nice scheduler available across multiple LuaJIT environments.

While going through this exercise, I also started looking at other features of schedulers, thinking about priorities, and other factors that might go into a really good scheduler. So, at least one thing becomes apparent to me. Having this scheduler readily available and transformable in the Lua space makes it relatively easy to try out different scheduling techniques that will match the particular situation at hand. There is even the possibility of changing the scheduling algorithm dynamically based on attributes of the running system.

Exciting times.