Simplified API Development – Part 2, First Steps

The series so far:

Simplified API Development – Part I, Ground Rules

To recap the ground rules:

1) All functions will have a return value

2) The first value is either true/false, or the return value of the function

So, I’ll take a step back.  What functions are we talking about exactly?  Well, let’s take a look at networking APIs just for kicks.  Of course I’m going to use the LuaJIT FFI mechanism to demonstrate, but realistically you could use any modern programming language that has multiple return value capabilities.

First up is the FFI layer.  In LuaJIT, the FFI layer is the most basic interop to whatever C library functions you are trying to call.  In order to create the interop layer, you need to identify a few key components.  First of all, which library (.dll) are your functions located in?  In the case of winsock, the primary .dll is ws2_32.dll.  This library is located on every modern day Windows machine.  If you wanted to, you could do a ‘dumpbin /EXPORTS ws2_32.dll’ and see all the functions that are exported from that library.

Next, you have to identify the header files that contain the function prototypes that you’re going to be binding to.  In the case of WinSock, the header files are numerous, and include: WinTypes.h, WinBase.h, mstcpip.h, inaddr.h, in6addr.h, ws2tcpip.h, ws2def.h, winsock2.h… There’s probably a couple more, but these contain the bulk of what you’ll actually use.

These various headers contain all the enums, constants, typedefs, and function prototypes that are expressed within the library.  For interop with through FFI, these headers need to be massaged a bit before they can be used.  Here’s a very simple case, the various functions utilize some specific data type aliases, so you either need to change the function prototypes, or you need to express those aliases.  If you choose to express the aliases, then you’ll have code like this:

 

local ffi = require "ffi"

ffi.cdef[[
typedef uint8_t		u_char;
typedef uint16_t 	u_short;
typedef uint32_t        u_int;
typedef unsigned long   u_long;
typedef uint64_t 	u_int64;
typedef uintptr_t	SOCKET;
typedef uint16_t 	ADDRESS_FAMILY;
typedef unsigned int    GROUP;
]]

Then there’s the various constants. In typical ‘C’ fashion, the header files are littered with #define statements such as:

#define SOCK_STREAM     1    // stream socket
#define SOCK_DGRAM      2    // datagram socket
#define SOCK_RAW        3    // raw-protocol interface

That’s great for C programming, as it’s just lexical sugar. With LuaJIT, you might want to put a little bit more structure around it to get optimal performance:

-- Socket Types
ffi.cdef[[
struct SocketType {
static const int SOCK_STREAM     = 1;    // stream socket
static const int SOCK_DGRAM      = 2;    // datagram socket
static const int SOCK_RAW        = 3;    // raw-protocol interface
static const int SOCK_RDM        = 4;    // reliably-delivered message
static const int SOCK_SEQPACKET  = 5;    // sequenced packet stream
};
]]

I’ll come back to this a bit later.

Next, there’s the interfaces to the functions themselves. In the case of networking, I’ll just focus on the traditional Berkeley socket calls:

-- Berkeley Sockets calls
ffi.cdef[[
u_long	htonl(u_long hostlong);
u_short htons(u_short hostshort);
u_short ntohs(u_short netshort);
u_long	ntohl(u_long netlong);

unsigned long inet_addr(const char* cp);
char* inet_ntoa(struct   in_addr in);

int inet_pton(int Family, const char * szAddrString, const void * pAddrBuf);
const char * inet_ntop(int Family, const void *pAddr, intptr_t strptr, size_t len);

SOCKET socket(int af, int type, int protocol);

SOCKET accept(SOCKET s,struct sockaddr* addr,int* addrlen);

int bind(SOCKET s, const struct sockaddr* name, int namelen);

int closesocket(SOCKET s);

int connect(SOCKET s, const struct sockaddr * name, int namelen);

int getsockname(SOCKET s, struct sockaddr* name, int* namelen);

int getsockopt(SOCKET s, int level, int optname, char* optval,int* optlen);

int ioctlsocket(SOCKET s, long cmd, u_long* argp);

int listen(SOCKET s, int backlog);

int recv(SOCKET s, char* buf, int len, int flags);

int recvfrom(SOCKET s, char* buf, int len, int flags, struct sockaddr* from, int* fromlen);

int select(int nfds, fd_set* readfds, fd_set* writefds, fd_set* exceptfds, const struct timeval* timeout);

int send(SOCKET s, const char* buf, int len, int flags);

int sendto(SOCKET s, const char* buf, int len, int flags, const struct sockaddr* to, int tolen);

int setsockopt(SOCKET s, int level, int optname, const char* optval, int optlen);

int shutdown(SOCKET s, int how);



int gethostname(char* name, int namelen);

struct hostent* gethostbyaddr(const char* addr,int len,int type);
struct hostent* gethostbyname(const char* name);

int GetNameInfoA(const struct sockaddr * sa, DWORD salen, char * host, DWORD hostlen, char * serv,DWORD servlen,int flags);
int getaddrinfo(const char* nodename,const char* servname,const struct addrinfo* hints,PADDRINFOA * res);
void freeaddrinfo(PADDRINFOA pAddrInfo);
]]

At this point, you have all you need to interop with the socket interfaces of whatever platform you’re running on, in this case Windows.

local ffi = require "ffi"
local lib = ffi.load("ws2_32")
local SocketType = ffi.new("struct SocketType");
local Family = ffi.new("struct FamilyType");
local Protocol = ffi.new("struct Protocol");

local sock = lib.socket(Family.AF_INET, SocketType.SOCK_STREAM, Protocol.IPPROTO_TCP);

That’s a good start, and if you don’t want to get any more hand holding from the Lua environment, you’re done.

Of particular note, there is no “C” side to this. The LuaJIT environment dynamically loads the winsock library into the running address space, and makes the calls to the appropriate functions magically. There’s not a ton of marshalling going on, because the FFI layer represents data structures directly in their ‘C’ native form. If you have more complex structures on the script side, you’ll have to do some marshalling. There’s no separate “interop library” that needs to be compiled and shipped with your executable, nothing to get in the way, you’re just done. This is quite different from the native Lua language, and JavaScript, which require you to write some C code to perform interop with these native libraries. Same goes for languages such as C#, where you have to create the data structures for marshalling, and create the P/Invoke signatures for each and every function. A very time consuming and error prone exercise to be sure.

At any rate, this is what’s required for basic interop. If the API is as simple as the Berkeley sockets API, then you might not want to go any further, but, as we’ll see, going just a little bit further will make life a lot easier when you start composing your application.

Until next time…
 

---