Is OpenMAX important for Free Software?

Much as OpenGL gives you access to 3D hardware, OpenMAX allows you to take advantage of hardware codecs. This is a brief overview introducing what OpenMAX is, explaining why it is useful for the open source community, and outlining steps for integration with free codecs, and open source multimedia frameworks and applications.

What is OpenMAX?

OpenMAX is a set of C APIs specified by the Khronos Group (who also co-ordinate standards like OpenGL and OpenAL). Whereas media frameworks like GStreamer and DirectShow are quite generic, providing all capabilities from codec integration through to synchronization of playback and recording and network access, OpenMAX more strictly defines three layers of operation:

• OpenMAX IL (Integration Layer) is an interface to multimedia codecs implemented in hardware or software. It does not provide any interfaces for synchronized capture or playback of video and audio.
• OpenMAX DL (Development Layer) APIs "specify audio, video and imaging functions that can be implemented and optimized on new CPUs, hardware engines, and DSPs and then used for a wide range of accelerated codec functionality such as MPEG-4, H.264, MP3, AAC and JPEG."
• OpenMAX AL (Application Layer) provides acceleration of capture and presentation of audio, video, and images.

The significance of this layering is that it allows hardware and software developers to implement conformance to a particular layer, so that device manufacturers can more reliably integrate components from each. This creates a free market for media components as commodities; and of course open source businesses are well suited to operating in such an environment.

OpenMAX is already availabile in generally open source platforms like Maemo and Android. As part of my work with Renesas I've been developing OpenMAX IL components for the video encoding and decoding hardware on the SH-Mobile processor series. (However, this post does not necessarily reflect the views of my employer).

Open Source implementations

OpenMAX components implement a specific C API. All components need to manage their ports and synchronize access to their input and output data buffers, so implementations generally include a shared library for the IL core, as well as some OpenMAX components required to pass Khronos conformance tests. There are (at least) three open source implementations of OpenMAX IL:

• Bellagio, developed mainly by STMicroelectronics and Nokia.
• TI have an implementation of OpenMAX for OMAP.
• OpenCore, the multimedia framework used by the Android platform, includes an open source implementation of OpenMAX IL. [gitweb]

So far I've been working with Bellagio, which has an active open source community. It has a good balance between commercial concerns like manufacturer deadlines and conformance testing, and openness to the community by encouraging and integrating development forks, and having a responsive mailing list and bug tracker.

Xiph OpenMAX

I haven't mentioned specific codecs yet; OpenMAX currently encourages use of non-free codecs like MP3, MPEG-4 and H.264. This in itself is not good for the aims of Free Software, but I think that the API standardization that OpenMAX offers can simplify the productization of hardware implementations of free codecs.

Xiph.org develops free codecs (Ogg Vorbis, Theora, Dirac, Speex, CELT, FLAC). Ogg Vorbis is required by the OpenMAX IL specification, but there are not yet any other OpenMAX IL implementations of the other codecs. Developing software OpenMAX IL components will allow application developers to implement Ogg support ahead of hardware support. It would also give hardware manufacturers a set of specific, well-defined goals for implementing Ogg support, with the understanding that the hardware components, when shipped with these software control APIs, will work in a variety of open source applications with minimal modifications.

There were a few Xiph.org people at FOMS 2009, so I introduced what we'd need to do to implement OpenMAX IL components for Xiph.org codecs:

• Choose an OpenMAX IL framework
• Implement generic Ogg mux/demux components (instead of single Ogg Vorbis component)
• Implement IL components for each codec (Theora, Dirac, Speex, CELT, FLAC)
• Implement GStreamer OpenMAX plugins for each codec

A recent thread, [Flac-dev] FLAC support for Android?, discusses requirements for implementing OpenMAX IL component for the lossless audio codec FLAC.

Free Software application support

In order to make use of OpenMAX components, applications need to either use the OpenMAX APIs directly or use a framework which does. For example, there is already an OpenMAX-GStreamer project which implements GStreamer plugin wrappers for Bellagio OpenMAX IL components. This allows any GStreamer application to take advantage of hardware codecs when they are available, or fall back to software implementations otherwise. This fits well with the GStreamer project's stated aim of of not implementing codecs, but providing routing, discovery and synchronization.

Other applications will need to use OpenMAX directly; good candidates would be applications that target mobile/embedded systems like Gnash, Fennec, WebKit and VoIP clients, as well as server-side transcoding or rendering software that needs high throughput.

Remember this:

• Mobile processors increasingly have hardware units for video encoding and decoding, as well as audio and image processing
• OpenMAX gives you access to hardware codecs (audio/video, image processing etc.)
• Implementing OpenMAX components for free codecs will give manufacturers a clear path to hardware implementation

At some point in the near future it'd be great to get a few open source OpenMAX implementers together at a conference, ideally at a more general multimedia workshop like FOMS to discuss application integration. Perhaps at FOMS 2010, or FOMS Europe? In any case it'd be good to get some more discussion going: do you think OpenMAX is important for Open Source, and for Free Software? What other barriers do you think there are to hardware support for free codecs? And would you be interested in helping out with developing and testing OpenMAX support for your favourite codecs, and in your favourite applications?

A month of Mondays

The last month or so has been fairly busy. I'll write more about each of these activities, but here's a quick summary of what I was up to (from about mid-January to mid-February):

• Attended the Foundations of Open Media Software developer workshop, which was again held during the week before linux.conf.au in Hobart, Tasmania. I had helped organize the workshop (what little I could do from Japan), gave an update on the state of liboggz and an introduction to OpenMAX.
• Participated in the Annodex Association AGM (and I got to announce the results of the election of the 2009 committee!)
• Gave a talk at the LCA OpenMobile miniconf about my work with Renesas: developing software APIs for managing the hardware codecs of the SHMobile processor series, and OpenMAX IL support.
• Chaired the LCA Multimedia miniconf, which I'd organized together with Silvia Pfeiffer
• Gave a presentation at linux.conf.au, Ogg Chopping: Techniques for Programming Correctness and Efficiency, mainly about functional programming.
• Was invited to join the W3C Media Fragments Working Group
• Got confirmation of Mozilla's funding, contracted through Wikimedia, for work on Firefox's Ogg seeking and server negotiation. Michael Dale and I were interviewed by Rodney Gedda of Computerworld: Wikipedia video gets boost with $100,000 Mozilla grant.
• Prepared and held my PhD intermediate thesis defence.

Somewhere in there I also managed to fit in a few days skiing in Hokkaido, and some time in the office in Tokyo. I've spent the last week relaxing back home in Kyoto and taking stock before getting back into things.

Break's over! onwards and upwards.

I love Mondays! Over the last two years I've learned a lot about how to get multiple tasks done in parallel -- a mix of GTD and some other techniques I've been developing. Unfortunately it sometimes means making sacrifices, like getting up in the wee hours of the morning to get work done on a ski trip; but every few months I also need a reset; I tend to go a bit crazy whenever I visit friends back in Sydney ;-)

If you've got any advice or words of encouragement please share them in the comments!