The Stream’s the thing

Way back in the day, along with the dawning of man, came the UNIX pipe.  Pipes are great for easily funneling the output of one process to the next:

cat file | more

Much later, C# continued the theme with streams.  Not quite at the process level, but within programs themselves.  Streams are great because they provide a simple common interface to just about anything:

stream.WriteByte(abyte); stream.Write(somebytes); stream.WriteTo(stream)

Well, this is great.  A nice abstraction for packaging things up and sending them somewhere else.  Although Lua has a simple io library, which has a nice enough interface, it’s missing a couple of things.  At the very lowest level, I want a stream on a chunk of memory.  In the .net frameworks, that’s provided by the MemoryStream class.  You tell it how much memory, and then you can do Read(), Write() to that memory as if it were any other stream.

This is a pattern that shows up a lot in network programming as well.  I’ve got data, and I need to encapsulate it into a chunk of memory, and then send it out over a connection of some sort.  It sure would be nice to have that MemoryStream in Lua.

So, I created a MemoryStream object.  But, fefore I could create that, I had to first create the ByteArray object.  It’s nothing more than a length, and ‘unsigned char *’, for the actual data.  You can allocate one like this:

ba = BufferArray():new(256)

This is possible with LuaJit.  You can do it in regular Lua, but not so easily, and not without writing a bit of C code I believe.  The interface to this thing is simple, you just access the data directly:

ba.data[0] = 23

under the covers, my allocation code is doing this:

self.data = ffi.new(‘unsigned char[?]’, size)

Through the magic of LuaJit, this will allocate exactly 256 bytes.  If I were using a regular Lua table and did this, it would allocate 8 times the amount, as lua numbers are actually doubles (8 bytes each).

Now, armed with this simple ByteArray, I can build the MemoryStream:

ms = MemoryStream(256)

ms:WriteByte(35);

ms:WriteByte(66);

ms:WriteByte(67);

To write ‘A’,’B’,’C’ into the memory buffer.  But, if I’m just going to write characters, that would be of limited usefulness.  I really want to deal with integers, and fractions thereof.  So, I can also do this:

ms:WriteInt32(527)

ms:WriteFloat(37.5)

And likewise for all the other numerical types.  Well, now we’re talking!  The MemoryStream will automatically write the value as the appropriate number of bytes in network byte order (big endian).  That’s useful because now I can write out the attached ByteArray as a single chunk, using whatever socket library I happen to have.

It’s just one of those fundamental building block sorts of things for me.  A ByteArray, a streaming interface, that makes it relatively easy to do something like chaining streams together, just like with UNIX pipes.  Now that I have an easily managed chunk of memory, I can go back and look at the PixelBuffer class I created, and refactor it to just use a ByteArray at its base.  That means eliminating more code!

Meanwhile, back at the graphics ranch, I managed to squeeze 1200 bytes out of my glsl.lua code.  Basically just refactoring the routines to utilize a couple of common base routines.  That’s 1200 bytes I can now allocate to improving the ray tracer, or adding in some other feature.  I’m thinking the ray tracer will be the best place to spend these newfound bites.

I’ve also decided that I don’t need to have the ability to read all file formats known to man for text maps.  I’ll have Targa built in, because I wrote code for that a long time ago, and not only is it an easy format to deal with, but the run length codec it uses can be repurposed, which saves more code.

With that, the only real hard dependency I have, beyond LuaJit itself, is the glfw framework/library, and LuaSocket.  For now, I’ll stick with both of these as they are cross platform, and not too onerous.  With all this work, surely a new release is in the offing!

 

---