When Is Software Engineering – Of Iterators and Closures

Last time around, I flexed my “software engineering” muscles by tackling an obscure Windows API using a scripting language. This works out well for when I’m at a cocktail party. I can keep people enthralled with my skills an an engineer by throwing out words like “SC Handle”, and “Enum Process Info”. The less my audience knows, the more intelligent I appear to be, and they can believe that my “engineer” title is well deserved.

Well, I haven’t been getting the intended response out of the cocktail scene of late, so I decided to master a couple new words. “Iterator” is a nice word because it doesn’t mean much more than enumerate, or list, or count, but it’s not in common usage, other than by the geeky programming crowd. So, it’s a sure win for the cocktail circuit. The other word is much more powerful because it sounds like prison, or a roadblock, or something “closure”.

Building off of my last bit of code, where I can make a single call to get a list of services running on my machine, it would be nice to get an ‘iterator’ on that set of services so I can then do more interesting things.

Ultimately, I want to be able to write the following code:

local mgr, err = SCManager();
for service in mgr:services() do
  print(service.ServiceName, service.DisplayName);
end

Which might result in the following kind of output:

KeyIso        CNG Key Isolation
Netlogon      Netlogon
SamSs         Security Accounts Manager
VaultSvc      Credential Manager
...

Actually, if I just queried for all services on my Windows machine, I’d get a few hundred of them listed, but you should get the general idea.

The trick here is in the ‘services()’ function of the SCManager object. That is what is known as an iterator, and it’s what gets me phone numbers at the parties. The code looks like this:

SCManager.services = function(self, dwServiceType)
  local InfoLevel = ffi.C.SC_ENUM_PROCESS_INFO;
  local dwServiceType = dwServiceType or ffi.C.SERVICE_TYPE_ALL;
  local dwServiceState = dwServiceState or ffi.C.SERVICE_STATE_ALL;
  local lpServices = nil;
  local cbBufSize = 0;
  local pcbBytesNeeded = ffi.new("DWORD[1]");
  local lpServicesReturned = ffi.new("DWORD[1]");
  local lpResumeHandle = ffi.new("DWORD[1]");
  local pszGroupName = nil;

  local status = service_core.EnumServicesStatusExA(
    self.Handle,
    InfoLevel,
    dwServiceType,
    dwServiceState,
    lpServices,
    cbBufSize,
    pcbBytesNeeded,
    lpServicesReturned,
    lpResumeHandle,
    pszGroupName);

  if status == 0 then
    local err = error_handling.GetLastError();

    if err ~= ERROR_MORE_DATA then
      return false, err;
    end
  end

  -- we now know how much data needs to be allocated
  -- so allocate it and make the call again
  cbBufSize = pcbBytesNeeded[0];
  lpServices = ffi.new("uint8_t[?]", cbBufSize);

  local status = service_core.EnumServicesStatusExA(
    self.Handle,
    InfoLevel,
    dwServiceType,
    dwServiceState,
    lpServices,
    cbBufSize,
    pcbBytesNeeded,
    lpServicesReturned,
    lpResumeHandle,
    pszGroupName);

    if status == 0 then
      local err = error_handling.GetLastError();
      return false, err;
    end
	
    local nServices = lpServicesReturned[0];

    local idx = -1;

    local function closure()
      idx = idx + 1;
      if idx >= nServices then
        return nil;
      end

      local res = {};

      local services = ffi.cast("ENUM_SERVICE_STATUS_PROCESSA *", lpServices);

      if services[idx].lpServiceName ~= nil then
        res.ServiceName = ffi.string(services[idx].lpServiceName);
      else
        return nil;
      end

      if services[idx].lpDisplayName ~= nil then
        res.DisplayName = ffi.string(services[idx].lpDisplayName);
      end

      local procStatus = {
        State = serviceState[services[idx].ServiceStatusProcess.dwCurrentState] or "UNKNOWN",
        ServiceType = getServiceType(services[idx].ServiceStatusProcess.dwServiceType),
        ProcessId = services[idx].ServiceStatusProcess.dwProcessId,
        ServiceFlags = services[idx].ServiceStatusProcess.dwServiceFlags,
	}
      res.Status = procStatus;

      return res;
    end

    return closure;
end

It’s not as silly as it might at first seem. There are two calls to the EnumServicesStatusExA() function. This is in classic Windows API style. The first call tells you how big of a buffer you need to allocate if you want to get the full results back at once. The second call actually retrieves all the data into a single chunk of memory. At this point you actually have all the data (there’s an edge case where you might have more services than will fit into the maximum 64k buffer, but I’m ignoring that for the moment).

The ‘closure()’ function is what actually gets returned when you call ‘mgr:services()’. This is the only little bit of magic that requires some explaining. This is particular to the way Lua works with iterators. that is, a function can return another function, and if the context is a ‘for’ loop, the returned function will be called repeatedly until it returns the ‘nil’ value.

I so happened to name the function ‘closure’, but that’s not necessary, it could be named anything. The easiest way to think about the ‘closure’ is; it’s a function which always has access to those variables that were enclosed in the wrapper function around it. So, for example, when I could do this:

local nextRecord = mgr:services();
while true do
  local record = nextRecord();
  if record == nil then
    break;
  end
  print(record.ServiceName, record.DisplayName);
end

The ‘for record in mgr:services() do’ is just a short hand for the while loop form that I’ve written above. Each time the ‘nextRecord()’ call is made, the ‘closure’ function is called, and it still has access to the variables that were in place from the last time around. Thus, the ‘idx’ value can be incremented each time through, and it won’t be reset. That’s the magic of closures, and this is how ‘iterators’ are implemented.

If my coding foo were more substantial, I might be able to tie this in to monads, but then people might think I’m talking about biology, nematodes, advertising, or math, so I’ll leave it at that.

With this newfound vacabulary in hand, I’m sure my cocktail party prospects will be on the rise once again.

This is pretty satisfying. I have tamed the system call, and managed to coerce the results into a form that is much more native to the scripting environment. Now all I have to do is get it into a form that is suitable for publishing on the web. Along the way, I’m going to have to introduce some stream based query capability, as well as JSON, but that’s another story.

---