Spelunking Windows – NTFS Change Journals

Since Windows 2000, the NTFS file system has had this capability to journal most file system actions. You know, things like open file, close file, delete, move, extend, add/remove attributes, etc. Originally, this capability was known as USN Journal, based on the sequence number that is part of the implementation. I name it NTFS Change Journals just because that’s a good enough explanation. The code for this example can be found here: https://github.com/Wiladams/TINN/blob/master/tests/test_USNJournal.lua

 

What can you do with this? Well, I want to get a list of all the changes that have occured on my c: since the beginning of time…

local test_getJournalEntries = function(StartUsn, ReasonMask, printRoutine)
  StartUsn = StartUsn or 0
  ReasonMask = ReasonMask or 0xFFFFFFFF;
  printRoutine = printRoutine or printJournalEntry;

  -- print all the USNs
  for entry in journal:entries(nil, ReasonMask) do
    printRoutine(entry);
    collectgarbage();
  end
end

Well, that’s what I’d like to do anyway. First, I need a convenient wrapper for the Change Journal. The API for the Change Journal is wrapped up in some DeviceIoControl() function calls. This is more akin to writing against a low level driver than the more typical FindFile/Next type of API that is the core of the Windows API. For that reason, it’s useful to wrap it up. Here’s some excerpt.

ChangeJournal.open = function(self, driveLetter)
  local handle, err = self:getVolumeHandle(driveLetter);
  if not handle then
    return false, err;
  end

  return self:init(handle);
end

ChangeJournal.entries = function(self, StartUsn, ReasonMask)
  StartUsn = StartUsn or 0;
  ReasonMask = ReasonMask or 0xFFFFFFFF;
  local ReturnOnlyOnClose = false;
  local Timeout = 0;
  local BytesToWaitFor = 0;

  local BUF_LEN = ffi.C.USN_PAGE_SIZE;
  local Buffer = ffi.new("uint8_t[?]", BUF_LEN);
  local ReadData = ffi.new("READ_USN_JOURNAL_DATA",
    {StartUsn, ReasonMask, ReturnOnlyOnClose, Timeout, BytesToWaitFor, self.JournalID});

  local dwBytes = ffi.new("DWORD[1]");
  local bytesReturned = 0;
  local dwRetBytes = 0;
  local UsnRecord = nil;
  local nextBuffUsn = StartUsn;

  local closure = function()

    if dwRetBytes == 0 then
      ReadData.StartUsn = nextBuffUsn;

      local status = core_io.DeviceIoControl( self:getNativeHandle(),
        FSCTL_READ_USN_JOURNAL,
        ReadData,
        ffi.sizeof(ReadData),
        Buffer,
        BUF_LEN,
        dwBytes,
        nil);

      if status == 0 then
        local err = errorhandling.GetLastError();
        return nil;
      end

      bytesReturned = dwBytes[0];

      -- skip past the initial USN
      nextBuffUsn = ffi.cast("USN *", Buffer)[0];
      dwRetBytes = bytesReturned - ffi.sizeof("USN");

      if dwRetBytes == 0 then
        -- reached end of records
        return nil;
      end

      -- Find the first record
      UsnRecord = ffi.cast("PUSN_RECORD", ffi.cast("PUCHAR",Buffer) + ffi.sizeof("USN"));
      dwRetBytes = dwRetBytes - UsnRecord.RecordLength;

      return UsnRecord;
    end

    -- Return the next record
    UsnRecord = ffi.cast("PUSN_RECORD",(ffi.cast("PCHAR",UsnRecord) + UsnRecord.RecordLength));
    dwRetBytes = dwRetBytes - UsnRecord.RecordLength;

    return UsnRecord;
  end

  return closure;
end

Great, with this minimal amount of functions in hand, I can now iterate the journal using the first bit of code.

Of course, what you can actually do with this journal information is another story. You could use it for backup/restore, virus checking, file browsing, whatever. Another thing that starts to emerge from playing around with this low level stuff is that you can really dive more deeply into the file system, at its very core, exploring MFTs and the like.

Another fun thing to do is just look at the latest changes that are occuring to the file system from this moment forward. This can be done fairly easily by simply looking for any new entries that show up.

ChangeJournal.waitForNextEntry = function(self, usn, ReasonMask)
  usn = usn or self:getNextUsn();
  local ReasonMask = ReasonMask or 0xFFFFFFFF;
  local ReturnOnlyOnClose = false;
  local Timeout = 0;
  local BytesToWaitFor = 1;

  local ReadData = ffi.new("READ_USN_JOURNAL_DATA", {usn, ReasonMask, ReturnOnlyOnClose, Timeout, BytesToWaitFor, self.JournalID});

  local pusn = ffi.new("USN");

  -- This function does not return until the USN
  -- record exits
  local BUF_LEN = ffi.C.USN_PAGE_SIZE;
  local Buffer = ffi.new("uint8_t[?]", BUF_LEN);
  local dwBytes = ffi.new("DWORD[1]");

  local status = core_io.DeviceIoControl( self:getNativeHandle(),
        FSCTL_READ_USN_JOURNAL,
        ReadData,
        ffi.sizeof(ReadData),
        Buffer,
        BUF_LEN,
        dwBytes,
        nil);

  if status == 0 then
    return false, errorhandling.GetLastError();
  end

  local UsnRecord = ffi.cast("PUSN_RECORD", ffi.cast("PUCHAR",Buffer) + ffi.sizeof("USN"));

  return UsnRecord;
end

local test_waitForNextEntry = function()
  local entry = journal:waitForNextEntry();

  while entry do
    printJournalEntry(entry);
    entry = journal:waitForNextEntry();
  end
end

I find it informative to run this little loop while I perform various activities on my machine. For example, when I launch a web browser, there’s a flury of activity related to opening, creating, closing, deleting various files. When I bring up a text editor, same thing. There’s the file I’m editing, which gets opened, then there’s invariably a temporary file that gets created, and possibly some addon files and the like.

It might be useful if you’re watching for viruses to see the activities of various files being opened which you don’t expect. With this tool in hand, being able to do that becomes a little bit easier.

At any rate, the Change Journal is a nice little bit of kit that has existed in the system for quite some time. There have been numerous tools built around it over the years. This little bit of code just makes it that much more approachable, at least for the common man programmer such as myself.

Advertisements


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s