How About that Web Server Again?

Has it really been more than a month?

Well, you see, I bought a house, packed, moved, took daughter to college, and…

The last time I wrote about making a simple WebServer, I left the job half finished. I showed that through layering, from a basic socket server, you could then build a simple http handler, which had not much more than an ‘OnRequest’ function.

That works great, and gives you the right level of access if you’re going to do things with http in general, that are beyond simple static file serving. But what if your interactions are more complex than you care to deal with using simple ‘if-then-else’ logic.

This time around, I’m going to use a new and improved form of the WebApp which first of all deals with the new and improved IOCP based sockets, and leverages the HttpServer object of yore.

Here is the basic web server:

-- main.lua
local WebApp = require("WebApp")
local resourceMap = require("ResourceMap");

local port = arg[1] or 8080

local app = WebApp(resourceMap, port);
app:run();

And you simply invoke it with:

tinn main.lua 8080

Well, that seems easy enough. If you wanted to get real crazy, and win an obfuscated code competition, you could do it in one line:

-- main.lua
require("WebApp")(require("ResourceMap"), arg[1] or 8080):run()

Or, if you’re a real Lua head, you could do it all on the command line, without even using the ‘main.lua’ file:

tinn -e "require('WebApp')(require('ResourceMap'), arg[1] or 8080):run()"

That’s all fun and games, and it’s cool to see that you can implement a web server in a single command line. But what’s in that magical ResourceMap file?

local URL = require("url");
local StaticService = require("StaticService");

local function replacedotdot(s)
  return string.gsub(s, "%.%.", '%.')
end

local HandleFileGET = function(request, response)
  local absolutePath = replacedotdot(URL.unescape(request.Url.path));
  local filename = './wwwroot'..absolutePath;

  StaticService.SendFile(filename, response)
  return false;
end

local ResourceMap = {
  ["/"] = {name="/",
    GET = HandleFileGET,
  };
}

return ResourceMap;

The idea here is that we want to route any ‘GET’ methods to the HandleFileGET() function. There is a ResourceMapper object within tinn that utilizes a structurce such as the one in ResourceMap. The general layout is {[pattern] = {name=””, METHOD = function [,METHOD = function]},}

Using this simple mechanism, you can easily route the handling of a particular resource request to a particular function.

This table can have entries that are much more interesting. For example, if you want to handle the retrieval of ‘favicon.ico’ differently than other files, you can just add a specific mapping for that.

local ResourceMap = {
  ["/"] = {name="/",
    GET = HandleFileGET,
  };

  ["/favicon.ico"] = {name="/favicon.ico",
    GET = HandleFaviconGET,
  };
}

You could have multiple methods per resource as well:

local ResourceMap = {
  ["/"] = {name="/",
    GET = HandleFileGET,
    POST = HandleFilePOST,
  };

  ["/favicon.ico"] = {name="/favicon.ico",
    GET = HandleFaviconGET,
  };
}

In general, the longest prefix match algorithm is applied to whatever is supplied within the resource field of the request. If you want to deal with all methods of a particular resource, without having to specify them explicitly, then you can use the magic method ‘_DEFAULT’.

local ResourceMap = {
  ["/"] = {name="/",
    GET = HandleFileGET,
    POST = HandleFilePOST,
    _DEFAULT = HandleFileDEFAULT,
  };

  ["/favicon.ico"] = {name="/favicon.ico",
    GET = HandleFaviconGET,
  };
}

In this way, if there is a request for a resource, and a method that we don’t know about at the time of creating the map, the HandleFileDEFAULT() function will be called to deal with it. That might be handy in the case where you’d like to handle these unknown method requests in a generic way, or you might want to change it over time without having to change your mapping.

Another case is when the resource isn’t actually a resource. Take for example a ‘CONNECT’ request:

CONNECT localhost:9000

In this case, the ‘resource’ does not start with a ‘/’, so the longest prefix match won’t land it on anything in our map. I need to deal with these with a different pattern. Well, the pattern part of the map is nothing more than a standard pattern in the Lua sense of the word, so a ‘.’ will l match any character. The following map will do the trick.

local ResourceMap = {
  ["."] = {name=".",
    CONNECT = HandleCONNECT,
  };

  ["/"] = {name="/",
    GET = HandleFileGET,
    POST = HandleFilePOST,
  };

  ["/favicon.ico"] = {name="/favicon.ico",
    GET = HandleFaviconGET,
  };
}

In this way, we’ll deal with the CONNECT method if it shows up.

This is an affirmative list. If there is a match to one of the patterns, then the mapped function is executed. If no pattern is found, either because the resource itself does not match, or because the resource does not have a function to handle the specified method, then a general 404 error is returned.

That’s about all there is to it. Create a mapping between URLs and functions, and you’re all done. Of course there’s the function itself:

local URL = require("url");
local StaticService = require("StaticService");

local function replacedotdot(s)
  return string.gsub(s, "%.%.", '%.')
end

local HandleFileGET = function(request, response)
  local absolutePath = replacedotdot(URL.unescape(request.Url.path));
  local filename = './wwwroot'..absolutePath;
	
  StaticService.SendFile(filename, response)

  return false;
end

Not a lot of magic in this particular one. First of all, there’s that simplistic ‘replacedotdot()’ function. That’s just a casual attempt to restrict the file access to the directory of our choosing. In HandleFileGET, the first thing to do is urldecode the path specified, then feed that to the replacedotdot() function. Then, take whatever comes out of that, and prepend it with ‘./wwwroot’, and finally feed that to the StaticService.SendFile() function, which is a standard part of TINN.

The return value of this function has meaning. If you return false, or ‘nil’, then the socket representing this particular request will be recycled, assuming there is potentially another request coming on the same socket.

If you instead return ‘true’, then the system assumes you are handling any subsequent recycling, or closing, and it will not take any further action with the underlying socket.

This is a nice feature in that it allows you to construct much more interesting interactions than the standard request/response, without much fuss. For example, you could easily open up websockets, upgrade connections in general, or do whatever you want.

At any rate, with TINN, you can create a simple web server in a single command line. I find that to be a fairly useful thing.


How About that Web Server

Although echo servers are the “Hello, World” of network programs, http servers are much more useful.

local HttpServer = require("HttpServer");
local StaticService = require("StaticService");

-- a simple ping template
local pingtemplate = [[
<html>
  <head>
    <title>HTTP Server</title>
  </head>
  <body>ping</body>
</html>
]]

-- Called every time there is a new http request
local OnRequest = function(param, request, response)
  if request.Url.path == "/ping" then
    response:writeHead("200")
    response:writeEnd(pingtemplate);
  else
    local filename = './wwwroot'..request.Url.path;
    StaticService.SendFile(filename, response);
  end
end

local server = HttpServer(8080, OnRequest);
server:run();

This looks a little bit like the echo service which was based on the raw socket server. Instead of “OnAccept”, this one implements “OnRequest”. The ‘request’ is an instance of a ‘WebRequest’ object, which contains all the stuff you’d expect in a WebRequest (resource, headers…). The routine is also handed a ‘WebResponse’ object as well. This is a simple convenience because it just wraps the netstream that is associated with the request object.

The HttpServer code itself looks like this:

local SocketServer = require("SocketServer")

local IOCPSocket = require("IOCPSocket")
local IOCPNetStream = require("IOCPNetStream");
local WebRequest = require("WebRequest");
local WebResponse = require("WebResponse");
local URL = require("url");

HttpServer = {}
setmetatable(HttpServer, {
  __call = function(self, ...)
    return self:create(...);
  end,
});

HttpServer_mt = {
  __index = HttpServer;
}

HttpServer.init = function(self, port, onRequest, onRequestParam)
  local obj = {
    OnRequest = onRequest;
    OnRequestParam = onRequestParam;
  };
  setmetatable(obj, HttpServer_mt);
	
  obj.SocketServer = SocketServer(port, HttpServer.OnAccept, obj);

  return obj;
end

HttpServer.create = function(self, port, onRequest, onRequestParam)
  return self:init(port, onRequest, onRequestParam);
end


HttpServer.OnAccept = function(self, sock)
  local socket = IOCPSocket:init(sock, IOProcessor);
  local stream, err = IOCPNetStream:init(socket);

  if self.OnRequest then
    local request, err  = WebRequest:Parse(stream);

    if request then
      request.Url = URL.parse(request.Resource);
      local response = WebResponse:OpenResponse(request.DataStream)
      self.OnRequest(self.OnRequestParam, request, response);
    else
      print("HandleSingleRequest, Dump stream: ", err)
    end
  else
    -- do nothing and let the socket close
  end
end

HttpServer.run = function(self)
  return self.SocketServer:run();
end

return HttpServer;

The ‘OnAccept()’ function this time around takes the unadorned socket, wraps it into a nicer socket object (so the io completion stuff can happen), and then uses the HttpRequest object to parse what’s on the stream. If the request is found the be intact, a response object is created and the two are handed off to the ‘OnRequest’ function if it exists.

This construct allows you to compose a webserver to meet your needs. You can spawn wherever you want, to run whichever part you want to run in parallel. At the top end, the consumer of this object won’t know the different, and can thus just handle the individual requests.

So, what’s so good about all this?
Well, first of all the TINN runtime, all up, is about 3Mb.
What you get for that is access to pretty much all the interesting stuff that Windows APIs have to offer. Whether it be network, OS, graphics, crypto, or multi-thread related, it’s all available right there in the little package.

This is good when you want to start creating simple REST based web services for this and that. For example, if you want to expose a webcam feed from your PC, or your PC acts as a hub for various wireless “internet of things” devices around your home, or whatever, you just write some lua code, without worrying about interop libraries, compiling, or anything else more interesting. Just a little script and away you go.


When Is Software Engineering – Hitting the Interwebs

So far, I’ve tamed an OS API that gives me the list of services running on my machine. I’ve been able to slap an iterator on it so that I could more easily deal with the information within my scripting language. Surely this is engineering?

My daughter, who knows Python, is not impressed. Next step, surely if I can make this information readily available through a web interface, it will be “Engineering”…

local WebApp = require("WebApp")

local HttpRequest = require "HttpRequest"
local HttpResponse = require "HttpResponse"
local URL = require("url");
local StaticService = require("StaticService");
local SCManager = require("SCManager");
local JSON = require("dkjson");

local HandleSingleRequest = function(stream, pendingqueue)
  local request, err  = HttpRequest.Parse(stream);

  if not request then
    print("HandleSingleRequest, Dump stream: ", err)
    return
  end

  local urlparts = URL.parse(request.Resource)

  if urlparts.path == "/system/services" then
    local mgr, err = SCManager();
    local res = {}

    for service in mgr:services() do
      if service.Status.State == "RUNNING" then
        table.insert(res, service);
      end
    end

    local jsonstr = JSON.encode(res, {indent=true});

    local response = HttpResponse.Open(stream)
    response:writeHead("200")
    response:writeEnd(jsonstr);
  else
    local filename = './wwwroot'..urlparts.path;
    local response = HttpResponse.Open(stream);
    StaticService.SendFile(filename, response)
  end

  -- recycle the stream in case a new request comes
  -- in on it.
  return pendingqueue:Enqueue(stream)
end

Runtime = WebApp({port = 8080, backlog=100})
Runtime:Run(HandleSingleRequest);

Surely this is engineering!! The business end of this code is right there with the familiar mgr:services() iterator. In this case, I take each of the records returned and stuff them into a table. Then, after all are returned, I turn that into a JSON string, and return it as the web result.

RunningServices

I only want to return the services that are in a “RUNNING” state, so I do that check before I actually stuff the record into the table. Well, there you have it. I’ve now gone from simply being able to make a simple system call, to being able to display the results of that call in a webpage accessible on my machine.  Is this “Software Engineering” yet?

Does it become engineering because I wrote the code that consumes a lower level framework?  Does it become engineering if I actually wrote the lower level framework?  Is it only engineering if I wrote the OS that supports that lower level framework?

As Doctor Evil would say; “Somebody throw me a frickin bone…”

I’ll try to impress my daughter with this code, then I’ll try it out on the cocktail circuit.  Perhaps someone will see this as software engineering…

But wait, there’s more!  That ‘query’ where I filtered for only the “RUNNING” services looked a bit enemic, and what about changing the state of those services?  Surely there’s room to do some engineering in there…