The Stream’s The Thing

So, again with that streaming thing…

The nice thing about streams is that they give you an easily composable component to construct more interesting things.  The .net frameworks have a whole host of streaming interfaces that deal with everything from files, to networking, to compression, encoding, encryption, you name it, and they all have the same style of interface.  This makes it relatively easy to compose things like an encrypted stream of compressed bytes being sent over a network in LSB order.

Why not Lua then?

How to do it?  Well, first of all, you start with the basic Stream interface.  At the lowest level, you need to be able to read and write single and multiple bytes.  Here’s a simple interface for that:

function ReadByte()
function ReadBytes(buffer, count, offset)

function WriteByte(value)
function WriteBytes(buffer, count offset)

And that’s that, wash your hands, everything is done…

Just kidding. Now it’s time to build upon this. But first. The single byte versions should be obvious. Just write or read a single byte at a time. This could be good enough to work on any stream, but won’t give you the best performance, or maybe it would. So, the multi-byte versions exist. The intention here is to allow the consumer to determine when and how memory buffers are allocated. This is particularly useful in the ReadBytes case.

local buff = ffi.new("uint8_t[1024]")
local numWritten = ReadBytes(buff, 1024, 0)

The buffer is a chunk of memory, not a Lua string object or table. The function must return the number of bytes written. This is typical of the types of interfaces you find for networking and files. It allows you to make multiple calls to the function until all the bytes are written. Similarly, it takes an offset (default to 0) so that you can reuse the same buffer multiple times.

WriteBytes() is similar.

Now, what if what you want is to construct a stream where you can read integers, floats, and the like. Not only that, but you want to be able to deal with byte order. Sometimes the stream is in Little-Endian, and sometimes it might be Big-Endian. So, it would be nice to have a stream that can deal with that automatically. Here’s a partial interface at this next level:

BinaryStream
  ReadByte()
  ReadInt16()
  ReadInt32()
  ReadSingle()
  ReadDouble()

  WriteByte(value)
  WriteInt16(value)
  WriteInt32(value)
  WriteInt64(value)
  WriteSingle(value)
  WriteDouble(value)

That’s good. Now putting it together, you could construct a stream, and then wrap it with a BinaryStream, and do the following:

local BinaryStream = require "BinaryStream"
local MemoryStream = require "MemoryStream"

function test_IntValues()
	local mstream = MemoryStream.new(1024);
	local bstream = BinaryStream.new(mstream);

	bstream:WriteInt16(16)
	bstream:WriteInt16(24)
	bstream:WriteInt16(-24);
	bstream:WriteInt32(32)
	bstream:WriteInt32(40)
	bstream:WriteInt32(-40)

	-- rewind the stream
	mstream:Seek(0)

	-- Verify values
	assert(bstream:ReadInt16() == 16);
	assert(bstream:ReadInt16() == 24);
	assert(bstream:ReadInt16() == -24);
	assert(bstream:ReadInt32() == 32);
	assert(bstream:ReadInt32() == 40);
	assert(bstream:ReadInt32() == -40);

	print("PASS");
end

In this case, first a MemoryStream is created, with a size of 1024 bytes. That stream is then encased within a BinaryStream so that simple numeric values can be written and read from the stream. This is nice and handy when you need to serialize some object into a stream, and you don’t really care whether that stream is in memory, a file on disk, or a network connection. The BinaryStream will work, as long as the underlying object implements those four basic stream functions.

One benefit of Lua being a dynamic language is that no class hierarchy needs to be implemented or enforced. There is no “Stream Object” per se. It’s just functions that exist on a table. Any table can contain these functions. That’s nice because any table is free to implement these, as well as other functions, without having to imply any particular object orientation, like single or multiple inheritance and the like. That’s actually quite a relief and quite powerful.

So, there it is, easy streaming. The LAPHLibs implement this, so that streaming is just that much easier. Streaming combined with knowledge of the data types available in the system provides the basis for a very powerful data serialization system.

---