NetNet

It seems like a law of the universe.  The littlest things matter the most.  There’s the seed of an Oak tree, DNA, and of course sockopt in network programming.

I’ve been doing some network programming of late, and it’s very interesting how much work goes into creating a model that actually works.  Basically, I want the following:

• The ability to act as a ‘server’
• Host Thousands of concurrent connections
• Run in environments that don’t support multi-threading
• Be adaptable to multiple protocols

Of course Lua is the answer, so what’s the question?  There are already tons of frameworks that probably do exactly what I want, but since I’m an engineer, I simply must execute from scratch no?  Actually, in terms of Lua, there are some options for creating simple http servers, but they all have various constraints.

The most robust effort borrows a page, and even some core code, from the Node.js world.  Luvit is one such project, that endeavors to replicate the simplicity and robustness of the Node environment.  Another one is LuaNode, which does a similar thing, with a different codebase.  One of the key differences between these two efforts is the underylying networking library that each of them relies upon.  Luvit utilizes the libuv library, which is the low level networking code abstracted from Node.js.  Must be good stuff then.  LuaNode utilizes the Boost.Asio library for the guts of its asynchronous networking power.  These are both very powerful libraries, written by and supported by people who obviously really know what they’re doing.  They work across multiple platforms, and do amazing things for humanity.

OK.  That’s it, work’s finished, I can go home now…

But then again, I want to learn something here.  I don’t want to just use the libraries as they stand.  Each of them is pretty big, by Lua standards, and they do a lot more than I really need at this point.  I’m not a big C++ fan, so I want to stay within the confines of Lua.  So, I want to understand, and implement some networking layer on my own, so that when I finally do succumb, and buy into one of the existing libraries, I’ll really understand the choices that I’m making and why.  Why not use ZeroMQ for example, vs either of these two libraries?

So, I begin at the beginning, and for the moment, I’m fairly confined to what I can do on Windows, ignoring Linux and MacOS.

The first constraint I chose for myself was the ‘single threaded’ model.  Ultimately I/O completion ports are the most efficient way to go on Windows, but that’s not going to work on other platforms, and it’s actually quite a pain to program.  So, I want a single threaded environment.  In order to do that, I have chosen to use non-blocking sockets as one of my core primitives.  The way a non-blocking socket works is that when you go to use it for a send or recv, you’ll get back an error, indicating that the call would block, if the data you’re expecting isn’t already available.  OK.  That’s nice.  So, in my simplest view of the world, I would do something like:

function Socket:Accept()
  local success,err = self.Socket:Accept()  -- call non-blocking accept

  -- err with be either nil upon success
  -- or 'wouldbock'
  -- or some other error
  return success, err
end

function main(...)
  local listener = Socket.new(...)

  while (true) do
    local accepted = listener:AcceptNewSocket()

    if accepted then
      -- do some work with the accepted socket
    end

    -- do some work with existing sockets
  end
end

The basic model here is, if something is available, that will be returned immediately. If nothing is available (in this case a new accepted connection), then nil is returned, along with the reason the resource was not available.

If you wanted to do a blocking call, you could simply do this:

repeat
  accepted, err = listener:Accept()
  if not accepted and err ~= "wouldblock" then
    return err
  end
until accepted

That will basically run until a socket connection is accepted, or there is an error other than ‘wouldblock’.

This points towards a general pattern where the higher application layer is taking on the responsibility for how the socket is dealt with. Whether it be a blocking call, or timeouts, they can be handled in the same way. A timeout could do the same thing as above, but use a stopwatch to count down the amount of time it spends looping, and break out when the specified time has expired.

Well, that’s a good start. But, how do you even compose the Socket object in the first place? I’ll be darned if I’ll be able to remember the calls and proper sequencing of things, and setsockopt, etc. So, backing up a bit, I first define a convenient “Socket” object.

ffi.cdef[[
typedef struct {
	SOCKADDR_STORAGE	*Address;
	SOCKET			Socket;
	bool			Listening;
} TCP_SOCKET;
]]

This is very Windows specific, but it’s generic enough to be represented on any platform. All this structure does is represent a socket, and the address the socket might be attached to, either as a client, server, or peer.

Now for a nice metatype, which will make dealing with this thing a little easier:

TCP_SOCKET = nil
TCP_SOCKET_mt = {
	__index = {
		--[[
			Setting various options
		--]]
		SetOption = function(self, optlevel, optname, optval, optlen)
			local err
			err = wsock.setsockopt(self.Socket, optlevel, optname, ffi.cast("const char *", optval), optlen);

			if err ~= 0 then
				err = wsock.WSAGetLastError()
			end

			return err
		end,

		SetReuseAddress = function(self, reuse)
			local err
			if reuse then
				local optval = ffi.new("uint32_t[1]", 1)
				err = self:SetOption(SOL_SOCKET, SO_REUSEADDR, optval, ffi.sizeof("uint32_t"))
			else
				local optval = ffi.new("uint32_t[1]", 0)
				err = self:SetOption(SOL_SOCKET, SO_REUSEADDR, optval, ffi.sizeof("uint32_t"))
			end

			return err
		end,

		SetNonBlocking = function(self, nonblocking)
			local err
			if nonblocking == false then
				local lpiMode = ffi.new("uint32_t[1]", 0);
				err = wsock.ioctlsocket(self.Socket, FIONBIO, lpiMode);
			else
				local lpiMode = ffi.new("uint32_t[1]", 1);
				err = wsock.ioctlsocket(self.Socket, FIONBIO, lpiMode);
			end

			return err
		end,


		GetBytesPendingReceive = function(self)
			local lpPending = ffi.new("uint32_t[1]", 0);
			local err = wsock.ioctlsocket(self.Socket, FIONREAD, lpPending);
			if err == 0 then
				local pending = lpPending[0]
				return pending
			end

			return nil, wsock.WSAGetLastError()
		end,


		--[[
			Primary functions
		--]]
		Accept = function(self)
			local sock =  wsock.accept(self.Socket, nil, nil);
			--print("TcpSocket:Accept - ", tonumber(sock));
			if sock ~= SOCKET_ERROR then
				local socket = TCP_SOCKET(nil, sock, false)
				return socket
			end

			return nil
		end,

		BindToPort = function(self, port, family)
			family = family or AF_INET
			local addr = CreateIPV4WildcardAddress(family, port)
			local addrlen = ffi.sizeof("struct sockaddr_in")
			local err = wsock.bind(self.Socket, ffi.cast("struct sockaddr *",addr), addrlen)

			return err
		end,

		Close = function(self)
			local err = wsock.closesocket(self.Socket)
			return err
		end,

		CloseDown = function(self)
			local how = SD_BOTH
			local err = wsock.shutdown(self.Socket, how)
			if err == 0 then
				err = self:Close();
			end

			if err < 0 then
				err = wsock.WSAGetLastError()
			end

			return err
		end,

		Connect = function(self)
			local addr = ffi.cast("struct sockaddr *", self.Address)
			local err = wsock.connect(self.Socket, addr, self.Address:Size());
			return err
		end,

		StartListening = function(self, backlog)
			backlog = backlog or 5

			wsock.listen(self.Socket, backlog)
			self.Listening = true
		end,



		Receive = function(self, buff, bufflen)
			--print("TcpSocket:Receive(), Pending: ", self:GetBytesPendingReceive());

			local result = wsock.recv(self.Socket, buff, bufflen, 0);
			local err

			if result == -1 then
				err = wsock.WSAGetLastError();
				result = nil
			end

			return result, err
		end,

		Send = function(self, buff, bufflen)
			bufflen = tonumber(bufflen);
			local result = wsock.send(self.Socket, buff, bufflen, 0);
			local err

			if result == -1 then
				err = wsock.WSAGetLastError();
				result = nil
			end

			return result, err
		end,


	},
}
TCP_SOCKET = ffi.metatype("TCP_SOCKET", TCP_SOCKET_mt)

There are a couple of nice things about having this structure. When I want to set a socket to non blocking, for example, I can simply do the following:

socket:SetNonBlocking(true)

Couldn’t be any easier, and I don’t have to remember the exact socket option needed to do it. One function that’s in here, which is really convenient, is the “CloseDown()” function. Yes, you can simply call “Close” on a socket, but that’s the moral equivalent of the “Kill” command. The more proper way to close down communications on a TCP socket, is to call shutdown, then closesocket. That way, both parties receive the proper closedown sequence of packets, and buffers in flight get a chance of being cleaned up properly.

There’s more conveniences I need though. I can never remember how to properly create those address structures, and with the ever emergent IPv6 vs IPv4, who can ever get things straight. So, I have some conveniences:

function CreateTcpSocket()
	local socket = wsock.WSASocketA(AF_INET, SOCK_STREAM, 0, nil, 0, WSA_FLAG_OVERLAPPED);
	--local socket = wsock.WSASocketA(AF_INET, SOCK_STREAM, IPPROTO_TCP , nil, 0, 0);
	if socket == INVALID_SOCKET then
		return nil, wsock.WSAGetLastError()
	end

	local sockstruct = TCP_SOCKET(nil, socket, false)

	return sockstruct
end

function CreateTcpServerSocket(port, backlog)
	backlog = backlog or 5

	local sock = CreateTcpSocket()
	if not sock then return nil end

	sock:SetReuseAddress(true);
	sock:BindToPort(port)
	sock:StartListening(backlog)

	return sock
end

The function CreateTcpSocket(), does exactly that. It will create a socket for the tcp protocol. At the point of creation, the socket is an “active” one, ready to be used to connect to something. The CreateTcpServerSocket() function will create a ‘server’ socket, which is ready to listen for incoming connection requests. I could go one step further, and give it a specific interface to bind to, but in this case, it will just bind to whichever interface it chooses on the machine.

Of particular note is the sequence of options that are set once the generic socket is created. For various server scenarios, it’s important to call SetReuseAddress(true) for a server side socket. This enables various things, including the case where multiple IP addresses resolve to the same machine. At any rate, it’s just one of those little things that should be done for more robust servers. Then there’s the classic Bind, and Listen that must occur. I never remember the order in which things should happen, and whatnot, so having this convenience function makes things very… convenient.

These are just the basics. I could use LuaSocket, or I could use one of those libraries. Going down the path of creating the same from scratch helps me to evaluate those other libraries for their robustness of implementation. Or, in the end, I might decide that for my purposes, what little I have is sufficient for my needs, and I can just move on to the more interesting aspects of my little service.

---