My Project
Multi-Media Abstraction Layer (MMAL). Draft Version 0.1.
Contents

Introduction

MMAL (Multi-Media Abstraction Layer) is a framework which is used to provide a host-side, simple and relatively low-level interface to multimedia components running on VideoCore. It also provides a component interface so that new components can be easily created and integrated into the framework.

There is no requirement that all the components be running on VideoCore as MMAL doesn't put any restriction on where components live. The current implementation for instance provides some components which can be run on both host-side or VideoCore (e.g. the splitter component).

Features

The MMAL API has been designed to support all the following features:

  • Sufficiently generic to support different kinds of multimedia component.
  • Simple to use from client side (mostly synchronous except where it matters).
  • Straightforward API for designing components (e.g. avoids multiple data paths, as found in RIL).
  • Allows for fully-optimised implementation of components (e.g. zero-copy buffer passing).
  • Portability (API is self-contained).
  • Supports multiple instances (e.g. of VideoCore).
  • Extensible without breaking source or binary backward compatibility.

API concepts

The MMAL API is based on the concept of components, ports and buffer headers. Clients create MMAL components which expose ports for each individual elementary stream of data they support (e.g. audio/video). Components expose input ports to receive data from the client, and expose output ports to return data to the client.

Data sent to or received from the component needs to be attached to a buffer header. Buffer headers are necessary because they contain buffer specific ancillary data which is necessary for the component and client to do their processing (e.g timestamps).

Components

MMAL lets clients create multi-media components (video encoders, video decoders, camera, and so-on) using a common API. Clients exchange data with components using buffer headers. A buffer header has a pointer to the payload data. Buffer headers are sent to and received from ports that are provided by components.

A typical decoder component would have a single input port and a single output port, but the same architecture could also be used for components with different layouts (e.g. a camera with a capture and preview port, or a debugging component with just an input port).

Component Creation

Each component is identified by a unique name. To create a specific component the client needs to call mmal_component_create with the desired component's name as an argument. This call will return a context (MMAL_COMPONENT_T) to the component. This context will expose the input and output ports (MMAL_PORT_T) supported by this specific component.

Note
All VideoCore components have a name starting with the "vc." prefix (this prefix is used to distinguish when a creation request needs to be forwarded to VideoCore).

Component Ports

A port (MMAL_PORT_T) is the entity which exposes an elementary stream (MMAL_ES_FORMAT_T) on a component. It is also the entity to which buffer headers (MMAL_BUFFER_HEADER_T) are sent or from which they are received.

Clients do not need to create ports. They are created automatically by the component when this one is created but the format of a port might need to be set by the client depending on the type of component the client is using.

For example, for a video decoding component, one input port and one output port will be available. The format of the input port must be set by the client (using mmal_port_format_commit) while the format of the output port will be automatically set by the component once the component has enough information to find out what its format should be.

If the input port format contains enough information for the component to determine the format of the output port straight away, then the output port will be set to the proper format when mmal_port_format_commit returns. Otherwise the output format will be set to MMAL_ENCODING_UNKNOWN until the component is fed enough data to determine the format of the output port. When this happens, the client will receive an event on the output port, signalling that its format has changed.

Buffer Headers

Buffer headers (MMAL_BUFFER_HEADER_T) are used to exchange data with components. They do not contain the data directly but instead contain a pointer to the data being transferred.

Separating the buffer headers from the payload means that the memory for the data can be allocated outside of MMAL (e.g. if it is supplied by an external library) while still providing a consistent way to exchange data between clients and components.

Buffer headers are allocated from pools and are reference counted. The refcount will drop when mmal_buffer_header_release is called and the buffer header will be recycled to its pool when it reaches zero. The client can be notified when the buffer header is recycled so that it can recycle the associated payload memory as well.

A pool of buffer headers should be created after committing the format of the port. When the format is changed, the minimum and recommended size and number of buffers may change.

Note
The current number of buffers and their size (MMAL_PORT_T::buffer_num and MMAL_PORT_T::buffer_size) are not modified by MMAL, and must be updated by the client as required after changes to a port's format.

Queues of Buffer Headers

Queues (MMAL_QUEUE_T) are a facility that allows thread-safe processing of buffer headers from the client. Callbacks triggered by a MMAL component when it sends a buffer header to the client can simply put the buffer in a queue and let the main processing thread of the client get its data from the queue.

Pools of Buffer Headers

Pools (MMAL_POOL_T) let clients allocate a fixed number of buffer headers, and a queue (MMAL_QUEUE_T). They are used for buffer header allocation. Optionally a pool can also allocate the payload memory for the client.

Pools can also be resized after creation, for example, if the port format is changed leading to a new number or size of buffers being required.

Port Parameters

Components support setting and getting component specific parameters using mmal_port_parameter_set and mmal_port_parameter_get. Parameters are identified using an integer index; parameter data is binary. See the Pre-defined MMAL parameter IDs page for more information on the pre-defined parameters.

Port Events

Components can generate events on their ports. Events are sent to clients as buffer headers and thus when the client receives a buffer header on one of the component's port it should check if the buffer header is an event and in which case process it and then release it (mmal_buffer_header_release). The reason for transmitting events in-band with the actual data is that it is often very valuable to know exactly when the event happens relative to the the actual data (e.g. with a focus event, from which video frame are we in focus).
Buffer headers used to transmit events are allocated internally by the framework so it is important to release the buffer headers with mmal_buffer_header_release so the buffer headers make it back to their actual owner.

Event buffer headers are allocated when the component is created, based on the minimum number and size of control port buffers set by the component. Component wide events (not port specific) are sent to the control port callback when that port is enabled. Port events are sent to the port callback, the same as data buffers, but the 'cmd' field is non-zero.

Versioning

The API requires that the MMAL core be the same or more recent version as the components and clients. Clients and components can be older and the API will still work both at compile-time and run-time.

Example Code

The following code is a simple example on how to do video decoding using MMAL. Note that the code is intended to be clear and illustrate how to use MMAL at its most fundamental level, not necessarily the most efficient way to achieve the same result. Use of opaque images, tunneling and zero-copy inter-processor buffers can all improve the performance or reduce the load.

The Port Connection Utility functions can also be used to replace much of the common "boilerplate" code, especially when a pipeline of several components needs to be processed.

#include <mmal.h>
...
static void input_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
// The decoder is done with the data, just recycle the buffer header into its pool
}
static void output_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
mmal_queue_put(queue, buffer); // Queue the decoded video frame
}
...
MMAL_COMPONENT_T *decoder = 0;
// Create the video decoder component on VideoCore
status = mmal_component_create("vc.ril.video_decoder", &decoder);
ABORT_IF_ERROR(status);
// Set format of video decoder input port
MMAL_ES_FORMAT_T *format_in = decoder->input[0]->format;
format_in->type = MMAL_ES_TYPE_VIDEO;
format_in->es->video.width = 1280;
format_in->es->video.height = 720;
format_in->es->video.frame_rate.num = 30;
format_in->es->video.frame_rate.den = 1;
format_in->es->video.par.num = 1;
format_in->es->video.par.den = 1;
status = mmal_format_extradata_alloc(format_in, YOUR_H264_CODEC_HEADER_BYTES_SIZE);
ABORT_IF_ERROR(status);
format_in->extradata_size = YOUR_H264_CODEC_HEADER_BYTES_SIZE;
memcpy(format_in->extradata, YOUR_H264_CODEC_HEADER_BYTES, format_in->extradata_size);
status = mmal_port_format_commit(decoder->input[0]);
ABORT_IF_ERROR(status);
// Once the call to mmal_port_format_commit() on the input port returns, the decoder will
// have set the format of the output port.
// If the decoder still doesn t have enough information to determine the format of the
// output port, the encoding will be set to unknown. As soon as the decoder receives
// enough stream data to determine the format of the output port it will send an event
// to the client to signal that the format of the port has changed.
// However, for the sake of simplicity this example assumes that the decoder was given
// all the necessary information right at the start (i.e. video format and codec header bytes)
MMAL_FORMAT_T *format_out = decoder->output[0]->format;
if (format_out->encoding == MMAL_ENCODING_UNKNOWN)
ABORT();
// Now we know the format of both ports and the requirements of the decoder, we can create
// our buffer headers and their associated memory buffers. We use the buffer pool API for this.
decoder->input[0]->buffer_num = decoder->input[0]->buffer_num_min;
decoder->input[0]->buffer_size = decoder->input[0]->buffer_size_min;
MMAL_POOL_T *pool_in = mmal_pool_create(decoder->input[0]->buffer_num,
decoder->input[0]->buffer_size);
decoder->output[0]->buffer_num = decoder->output[0]->buffer_num_min;
decoder->output[0]->buffer_size = decoder->output[0]->buffer_size_min;
MMAL_POOL_T *pool_out = mmal_pool_create(decoder->output[0]->buffer_num,
decoder->output[0]->buffer_size);
// Create a queue to store our decoded video frames. The callback we will get when
// a frame has been decoded will put the frame into this queue.
MMAL_QUEUE_T *queue_decoded_frames = mmal_queue_create();
decoder->output[0]->userdata = (void)queue_decoded_frames;
// Enable all the input port and the output port.
// The callback specified here is the function which will be called when the buffer header
// we sent to the component has been processed.
status = mmal_port_enable(decoder->input[0], input_callback);
ABORT_IF_ERROR(status);
status = mmal_port_enable(decoder->output[0], output_callback);
ABORT_IF_ERROR(status);
// Enable the component. Components will only process data when they are enabled.
status = mmal_component_enable(decoder);
ABORT_IF_ERROR(status);
// Data processing loop
while (1)
{
// The client needs to implement its own blocking code.
// (e.g. a semaphore which is posted when a buffer header is put in one of the queues)
WAIT_FOR_QUEUES_TO_HAVE_BUFFERS();
// Send empty buffers to the output port of the decoder to allow the decoder to start
// producing frames as soon as it gets input data
while ((buffer = mmal_queue_get(pool_out->queue)) != NULL)
{
status = mmal_port_send_buffer(decoder->output[0], buffer);
ABORT_IF_ERROR(status);
}
// Send data to decode to the input port of the video decoder
if ((buffer = mmal_queue_get(pool_in->queue)) != NULL)
{
READ_DATA_INTO_BUFFER(buffer);
status = mmal_port_send_buffer(decoder->input[0], buffer);
ABORT_IF_ERROR(status);
}
// Get our decoded frames. We also need to cope with events
// generated from the component here.
while ((buffer = mmal_queue_get(queue_decoded_frames)) != NULL)
{
if (buffer->cmd)
{
// This is an event. Do something with it and release the buffer.
continue;
}
// We have a frame, do something with it (why not display it for instance?).
// Once we're done with it, we release it. It will magically go back
// to its original pool so it can be reused for a new video frame.
}
}
// Cleanup everything
mmal_queue_destroy(queue_decode_frames);