Taming Raspeberry Pi Display Manager

Last time around, I was busy slaying the bcm_host interface.  One of the delightful thing that follows on from slaying dragons is that you get to plunder their treasure.  In this particular case, with the bcm_host stuff in hand, you can now do fantastic things like put pixels on the screen.

Where to start?

The display system of the Raspberry Pi was at first very confusing, and perplexing.  In order for me to understand it, I had to first just ignore the X Window system, because that’s a whole other thing.  At the same time, I had to hold back the urge to ‘just give me the frame buffer!’.  You can in fact get your hands on the frame buffer, but if you do it in a nicely controlled way, you’ll get the benefit of some composited ‘windows’ as well.

Some terminology:

VideoCore – The set of libraries that is at the core of the Raspberry Pi hardware for audio and video.  This is not just one library, but a set of different libraries, each of which provides some function.  The sources are not available, but various header files are (located in /opt/vc/*).  There isn’t much documentation, other than what’s in the headers, and this is what makes them difficult to use.

EGL, EGLES, OpenVG, OpenMAX – These are various APIs defined by the Khronos Group.  They cover windowing, open GL for embedded devices, and 2D vector graphics.  These are similarly supplied as opaque libraries, with their header files made available.  Although these libraries are very useful and helpful, they are not strictly required to get something displayed on the screen.

This time around, I’m only going to focus on the parts of the VideoCore, ignoring all the Khronos specific stuff.

The first part of VideoCore is the vc_dispmanx.  This is the display manager service API.  Keep in mind that word “service”.  From a programmer’s perspective, you can consider the display manager to be a ‘service’, meaning, you send commands to it, and they are executed.  I have created a LuaJIT FFI file for vc_dispmanx.lua, which essentially gives me access to all the functions within.  Of course, following my own simple API development guidelines, I created a ‘helper’ file as well; DisplayManX.lua.

The short story is this.  Within DisplayManX, you’ll find the implementation of 4 convenience classes:

DMXDisplay – This is what gives you a ‘handle’ on the display.  This could be an attached HDMI monitor, or a composite screen.  Either way, you can use this handle to get the size, and other characteristics.  This handle is also necessary to perform any other operations on the display, from creating a window, to displaying content.

DMXElement – A representation of a visual element on the DMXDisplay.  I’m trying to avoid the word ‘window’ because it does not have a title bar, close box, resize, etc.  Those are all visual elements to be developed by a higher level thing.  This DMXElement is what you need to carve out a piece of the screen where you intend to do some drawing.  You can give an element a “layer”, so they can be ordered.  The DMXDisplay, acts as a fairly rudimentary element manger, so it does basic front to back ordering of your elements.

DMXResource – Just like I’m trying to avoid the word ‘window’ with respect to DMXElement, I’ll try to avoid the word ‘view’ in describing DMXResource.  A resource is essentially like a bitmap.  You create a resource, with a certain size, and pixel format, fill it in with stuff, and then ultimately display it on the screen by writing it into the DMXElement.  If you were creating a traditional windowing environment, this would be the backing store of a window.

DMXUpdate – This is like a transaction.  As mentioned earlier, DMX is a ‘service’, and you send commands to the service.  In order to send commands, you must bracket them in an ‘update begin’/’update end’ pairing.  You can send a ‘batch’ of commands by just placing several commands between the update begin/end.  This object represents the bracketing transaction.

The good news is, you don’t really need to worry about this low level of detail if you want to use these classes.

So, How about an example?

 

-- A simple demo using dispmanx to display an overlay

local ffi = require "ffi"
local bit = require "bit"
local bnot = bit.bnot
local band = bit.band
local bor = bit.bor
local rshift = bit.rshift
local lshift = bit.lshift

local DMX = require "DisplayManX"


ALIGN_UP = function(x,y)  
    return band((x + y-1), bnot(y-1))
end

-- This is a very simple graphics rendering routine.
-- It will fill in a rectangle, and that's it.
function FillRect( image, imgtype, pitch, aligned_height,  x,  y,  w,  h, val)
    local         row;
    local         col;
    local srcPtr = ffi.cast("int16_t *", image);
    local line = ffi.cast("uint16_t *",srcPtr + y * rshift(pitch,1) + x);

    row = 0;
    while ( row < h ) do
	col = 0; 
        while ( col < w) do
            line[col] = val;
	    col = col + 1;
        end
        line = line + rshift(pitch,1);
	row = row + 1;
    end
end

-- The main function of the example
function Run(width, height)
    width = width or 200
    height = height or 200


    -- Get a connection to the display
    local Display = DMXDisplay();
    Display:SetBackground(5, 65, 65);

    local info = Display:GetInfo();
    
    print(string.format("Display is %d x %d", info.width, info.height) );

    -- Create an image to be displayed
    local imgtype =ffi.C.VC_IMAGE_RGB565;
    local pitch = ALIGN_UP(width*2, 32);
    local aligned_height = ALIGN_UP(height, 16);
    local image = ffi.C.calloc( 1, pitch * height );

    FillRect( image, imgtype,  pitch, aligned_height,  0,  0, width,      height,      0xFFFF );
    FillRect( image, imgtype,  pitch, aligned_height,  0,  0, width,      height,      0xF800 );
    FillRect( image, imgtype,  pitch, aligned_height, 20, 20, width - 40, height - 40, 0x07E0 );
    FillRect( image, imgtype,  pitch, aligned_height, 40, 40, width - 80, height - 80, 0x001F );

    local BackingStore = DMXResource(width, height, imgtype);

	
    local dst_rect = VC_RECT_T(0, 0, width, height);

    -- Copy the image that was created into 
    -- the backing store
    BackingStore:CopyImage(imgtype, pitch, image, dst_rect);

 
    -- Create the view that will actually 
    -- display the resource
    local src_rect = VC_RECT_T( 0, 0, lshift(width, 16), lshift(height, 16) );
    dst_rect = VC_RECT_T( (info.width - width ) / 2, ( info.height - height ) / 2, width, height );
    local alpha = VC_DISPMANX_ALPHA_T( bor(ffi.C.DISPMANX_FLAGS_ALPHA_FROM_SOURCE, ffi.C.DISPMANX_FLAGS_ALPHA_FIXED_ALL_PIXELS), 120, 0 );
    local View = Display:CreateElement(dst_rect, BackingStore, src_rect, 2000, DISPMANX_PROTECTION_NONE, alpha);
 

    -- Sleep for a second so we can see the results
    local seconds = 5
    print( string.format("Sleeping for %d seconds...", seconds ));
    ffi.C.sleep( seconds )

end


Run(400, 200);

This is a sample taken from one of the original hello_pi examples, but made to work with this simplified world that I’ve created. To get started:

local DMX = require "DisplayManX"

This will simply pull in the display management service so we can start using it.

Next up, we see a simple rectangle filling routine:

-- This is a very simple graphics rendering routine.
-- It will fill in a rectangle, and that's it.
function FillRect( image, imgtype, pitch, aligned_height,  x,  y,  w,  h, val)
    local         row;
    local         col;
    local srcPtr = ffi.cast("int16_t *", image);
    local line = ffi.cast("uint16_t *",srcPtr + y * rshift(pitch,1) + x);

    row = 0;
    while ( row < h ) do
	col = 0; 
        while ( col < w) do
            line[col] = val;
	    col = col + 1;
        end
        line = line + rshift(pitch,1);
	row = row + 1;
    end
end

We don’t actually use the imagetype, nor the aligned_height. Basically, we’re assuming an image that has 16-bit pixels, and the ‘pitch’ tells us how many bytes per row. So, go through 16-bit value at a time, and set it to the color value specified.

Next, we come to the main event. We want to create a few semi-transparent rectangles, and display them on the screen. Then wait a few seconds for you to view the results before cleaning the whole thing up.

    local Display = DMXDisplay();
    Display:SetBackground(5, 65, 65);

One of the first actions is to create the display object, and set the background color. The funny thing you’ll notice, if you run this code, is suddenly your monitor seems to have a lot more screen real estate than you thought. Yep, X is taking up a smaller portion of the screen (if you’re running X). Same with the regular terminal. If you were running something like XBMC, you’d be seeing your full display being utilized. This is how they do it. At any rate, there’s an application right there. If you want to set the border color of your screen, just do those two lines of code, and you’re done…

Moving right along. We need a chunk of memory allocated, which will be what actually gets displayed in the window.

    -- Create an image to be displayed
    local imgtype =ffi.C.VC_IMAGE_RGB565;
    local pitch = ALIGN_UP(width*2, 32);
    local aligned_height = ALIGN_UP(height, 16);
    local image = ffi.C.calloc( 1, pitch * height );

For those who are framebuffer obsessed, there’s you’re window’s frame buffer right there. It’s just a chunk of memory of the appropriate size to match the pitch and alignment requirements of the pixel format you’ve selected. There are a fair number of formats to choose from, including RGBA32 if you want to burn up a lot of memory.

This would typically be represented as a “Bitmap” or “PixelMap”, or “PixelBuffer” object in most environments. Next time around, I’ll encapsulate it in one such object, but for now, it’s just a pointer to a chunk of memory ‘image’.

Now that we’ve got our chunk of memory, we fill it with color:

    FillRect( image, imgtype,  pitch, aligned_height,  0,  0, width,      height,      0xFFFF );
    FillRect( image, imgtype,  pitch, aligned_height,  0,  0, width,      height,      0xF800 );
    FillRect( image, imgtype,  pitch, aligned_height, 20, 20, width - 40, height - 40, 0x07E0 );
    FillRect( image, imgtype,  pitch, aligned_height, 40, 40, width - 80, height - 80, 0x001F );

As described earlier, the DMXResource is required to actually display stuff in a display element, so we need to create that:

    local BackingStore = DMXResource(width, height, imgtype);

Don’t get tripped up by the name of the variable. It could be anything, I just used “BackingStore” to emphasize the fact that it’s the backing store of our display element.

Now to copy the image into the backing store:

    local dst_rect = VC_RECT_T(0, 0, width, height);

    -- Copy the image that was created into 
    -- the backing store
    BackingStore:CopyImage(imgtype, pitch, image, dst_rect);

Here, I do call it ‘CopyImage’, because that is in fact what you’re doing. In all generality, the lower level API is simply ‘write_data’, but why be so obtuse when we can be more precise. At this point, we have our ‘bitmap’ written into our backing store, but it’s not displayed on the screen yet!

This last part is the only ‘black magic’ part of the whole thing, but you’ll soon see it’s nothing too exciting. We need to create an element on the display, and that element needs to have our BackingStore as it’s backing.

    -- Create the view that will actually 
    -- display the resource
    local src_rect = VC_RECT_T( 0, 0, lshift(width, 16), lshift(height, 16) );
    dst_rect = VC_RECT_T( (info.width - width ) / 2, ( info.height - height ) / 2, width, height );
    local alpha = VC_DISPMANX_ALPHA_T( bor(ffi.C.DISPMANX_FLAGS_ALPHA_FROM_SOURCE, ffi.C.DISPMANX_FLAGS_ALPHA_FIXED_ALL_PIXELS), 120, 0 );

First we establish the source and destination rectangles. The src_rect indicates what part of our ‘BackingStore’ we want to display. The ‘dst_rect’ says where we’ll want to locate the window on the display. That’s the only challenging part to wrap your head around. src_rect -> related to bitmap image, dst_rect -> related to position of window. In this case, we want to take the whole bitmap, and locate the window such that it is centered on the display.

And finally, we create the element (window) on the screen:

    local View = Display:CreateElement(dst_rect, BackingStore, src_rect, 2000, DISPMANX_PROTECTION_NONE, alpha);

Since the window already has a backing store, it will immediately display the contents, our nicely crafted rectangles.

And that about does it.

That’s about the hardest it gets. It gets easier from here once you start integrating with EGL and the other libraries. These are the bare minimums to get something up on the screen. If you were doing something like a game, you’d probably create a couple of resources, with their attendant PixelBuffers, and variously swap them into the View. This is essentially what EGL does, it manages these low level details for you.

What you’ll notice is that I don’t deal with any mouse or keyboard in this particular example. The input subsystem is a whole other ballgame, and is very Linux specific, not VideoCore specific. I will incorporate that later.

To finish this installment, I’ll leave this teaser…

local display = DMXDisplay()
display:Snapshot()

What’s that about?

---