Release: libfishsound 0.9.1

This is a maintenance release, fixing a security vulnerability in Speex header processing as outlined in oCERT 2008-02. When used in a client for web video content, as in the OggPlay Firefox Plugin or the Ogg DirectShow filters, a specially crafted Ogg Speex stream hosted on a server could be used to allow an attacker to execute arbitrary code on the client system. The OggPlay plugin binaries available from www.annodex.net have already been updated.

Details

The Speex header contains a 32-bit modeID field, interpreted by libspeex as a signed int (spx_int32_t) The normal way to use this is to index into a global mode list to retrieve a SpeexMode *:

mode = (SpeexMode *)speex_mode_list[modeID];

and then use that to set up a decoder:

st = speex_decoder_init(mode);

This calls speex_decoder_init() in libspeex, which looks like:

void *speex_decoder_init(const SpeexMode *mode)
{
   return mode->dec_init(mode);
}

So if you don't check that the modeID given in the stream header is within the bounds of speex_mode_list[], arbitrary code can be executed. libfishsound was checking the upper bound (modeID < SPEEX_NB_MODES) but was not checking against negative values.

Discussion

This header processing is all boilerplate, and a reference implementation is given in speexdec.c. I took a copy of that about 7 years ago for Sweep, which I then adapted for libfishsound. The current reference speexdec.c does not have this bug.

For the Symbian port of Speex we created a function which returns the desired mode given a modeID, rather than having application code index into a global mode list. I wrote and committed speex_get_mode() to libspeex in September 2004, and it does the correct bounds checking. So if I'd been using that function in libfishsound then today's problem would never have happened. As it turns out, the libfishsound svn trunk version of speex.c does use that function. As far as I am aware, the OggPlay plugin binaries have always been built against the libfishsound svn trunk, so they were never vulnerable in the first place. However, recent tarball releases of libfishsound have been coming of a separate branch, so the advisory is valid for applications linked against those releases.

Finally, I sent a patch to Jean-Marc Valin yesterday which entirely removes the possibility of this bug happening again by bounding the mode values returned by speex_packet_to_header() in libspeex. It will be available very soon in a libspeex release.

Acknowledgements

Thanks to the team at oCERT for the efficient reporting of this advisory, and to the anonymous submitter for the details. I was able to patch the offending branches, which allowed j^ to build and upload new OggPlay plugin binaries (within 24 hours of contact by oCERT).

• libfishsound 0.9.1 release notes
• oCERT 2008-02

Release: HOgg 0.4.0

HOgg is a Haskell library and commandline tool for manipulating Ogg files. This release contains a bunch of code written during FOMS and LCA 2008, including a new sort subcommand and proper handling of Skeleton when merging and ripping files. Full details are in the release notes.

sort implementation

My favourite part is the implementation of the new sort subcommand:

sort :: [OggPage] -> [OggPage]
sort = sortHeaders . listMerge . demux

This is somewhat shorter than the equivalent C implementation, oggz-sort.c — Haskell affords abstraction whereas in C it's a trade-off. sortHeaders is a long (21 line) function that re-orders header pages according to the Theora and Skeleton specifications, and listMerge is a generic list merging function, also used in the merge subcommand. demux is tiny:

demux :: (Serialled a) => [a] -> [[a]]
demux = classify serialEq

You can read that as "demux is classification by serial number": classify is a generic list function, classifying list elements according to some criterion you give it. Here, for example, the list of pages:

[Video0, Audio0, Video1, Audio1, Audio2, Audio3, Video2, Audio4, Video3, ...]

will get classified into two separate lists:

[[Video0, Video1, Video2, Video3, ...],
 [Audio0, Audio1, Audio2, Audio3, Audio4, ...]]

This is done lazily, meaning that the processing is done on the fly and big intermediate lists are not constructed in memory. Video0, Audio0 will be passed through listMerge and sortHeaders and written to disk by the consumer of sort well before Video103 and Audio5007 are seen.

Documentation improvements and self-checking

The help for each subcommand now contains long descriptions, mostly similar to the man pages of the Oggz tools. The descriptions also have explicit sections describing how Theora, Skeleton and chained files are handled. The example commandlines for each subcommand use the Ogg MIME types and file extensions that we are now recommending in Xiph.Org.

The best bit though is hogg selfcheck, which checks that the help examples are valid. It checks that all the example commandlines pass through getOpt without errors, and that all file extensions used in options are valid. This is the kind of nice touch which would have been a pain to code up in C, but fell out cleanly in the Haskell implementation. As it is fairly cheap to run (and printing help text is hardly a performance-critical operation), this option is also silently run after printing out any help output at all, so that such errors are more likely to be found and reported. The same commit that introduced hogg selfcheck also fixed two such documentation errors which were found by this option :-)

Release: liboggz 0.9.7

There's been a whole bunch of work on liboggz recently; it deserves a few more weeks of shaking out and perhaps some updated Win32/MacOS support before it gets 1.0 slapped on it.

liboggz 0.9.7 includes a new tool called oggz-sort, which addresses a problem with some encoders that Shane Stephens brought up at FOMS. The discussion was going around in circles, so my response was to write this C code. It implements a function that Shane has written but not yet released in his OCaml implementation of Ogg (oogg), and which I've written but not yet released in my Haskell implementation (HOgg). Of course, people will take this version more seriously because it's written in C.

From oggz-sort (1):

oggz-sort sorts an Ogg file, interleaving pages in order of presentation time. It correctly interprets the granulepos timestamps of Ogg Vorbis, Speex, FLAC and Theora bitstreams, and all bitstreams of Annodex files.

Some encoders produce files with incorrect page ordering; for example, some audio and video pages may occur out of order. Although these files are usually playable, it can be difficult to accurately seek or scrub on them, increasing the likelihood of glitches during playback. Players may also need to use more memory in order to buffer the audio and video data for synchronized playback, which can be a problem when the files are viewed on low-memory devices.

The tool oggz-validate can be used to check the relative ordering of packets in a file. If out of order packets are reported, use oggz-sort to fix the problem.

This release also adds support for the experimental CELT audio codec, which is being developed by Jean-Marc Valin (the primary author of Speex). CELT is designed as a low-latency codec for high-quality audio. When wiretapping conversations encoded in CELT, we recommend that you record using the Ogg container format. You can then use oggz-tools to help with your analysis.

Release: liboggz 0.9.6

This release of Oggz 0.9.6 contains a new tool, oggz-comment, which can be used to edit the basic metadata (title, producer, copyright etc.) of Ogg Theora files. The library also has some pretty major improvements to the way it works out timestamps and does seeking, mostly the work of Shane Stephens.

In media files, timing and synchronization is extremely important. If the image and audio start to go out of sync, it is very noticeable and the video quickly becomes unwatchable. When you scan through a file you often need to decode a lot more data than you actually display. This is particularly the case when you jump backwards, which is common in a user interface that supports scrubbing. As video frames are stored as a difference relative to earlier (or later) frames, you end up needing to secretly jump further back in the file to the previous keyframe, and then decode many frames up to the one you actually want to show. For a smooth user experience you need to do this as quickly as possible.

Ogg has some interesting framing properties. Given that timing is so important, you might expect that every packet has its precise timing information associated with it. In Ogg, it turns out not to be so. Packets are stored in pages, and there is only one timestamp per page. It is common for many audio packets to be crammed onto one page; the timing information for all the rest is not stored in the file. On the other hand, the encoded data for video keyframes is usually much larger, and spans multiple pages. Only the last packet on a page has its timestamp recorded, so if the keyframe is followed by an a much smaller packet of frame data in the same page, the timestamp for the keyframe will be lost. For these reasons I tend to refer to Ogg as a "lossy" container.

In order to minimize these problems, liboggz now inspects the encoded data in order reconstruct the expected granulepos (corresponding to a timestamp) for every packet in an Ogg stream. This allows applications to use reliable timestamps, even though these are only sparsely recorded in most Ogg streams. This is not as easy as it sounds, particularly for Ogg Vorbis. To get a flavour of what's involved, read Shane's rant in the comments, explaining how to calculate Vorbis timestamps.

For an in-depth discussion, come to Ralph Giles' talk at linux.conf.au, Seeking is hard: Ogg design internals.

Release: libfishsound 0.9.0

Now libfishsound 0.9.0 supports FLAC, the Free Lossless Audio Codec. The patches were originally contributed by Tobias Gehrig in 2004. There hasn't been much use of Ogg FLAC, whereas FLAC in its native encoding is very popular. However, the point of the Ogg mapping is to allow FLAC to be used in parallel with other codecs, in particular as the audio codec for video files. The combination of Theora video and FLAC audio can be very useful for music videos, where you might not care too much if the image has lost some quality but you want the sound to be as good as possible.

However, creating such a file isn't so easy. Let's say you have a source video, like GrooveTV #204 - Jacob Fred Jazz Odyssey. I took the MPEG-1 file as recommended; for clarity, let's call it source.mpg. To make a video to test on, I did:

ffmpeg2theora source.mpg
to encode the video into an Ogg file containing Theora video and Vorbis audio. This produces source.ogv.

oggzrip -c theora source.ogv -o video-theora.ogv
to extract only the Theora video track, into video-theora.ogv.

mpg123 -w source.wav source.mpg
to extract the audio to a wav file, source.wav. Here the audio in the source material was encoded as MPEG I layer II; obviously if you were producing a music video, you'd skip this step and encode FLAC from the original recording. I didn't have that here, and I just wanted a file I could test on.

However, at the least this step means that no further artifacts are introduced into the audio, other than those which were present in the MPEG encoding. If the only source material you have is already encoded, you don't want to degrade it further by re-encoding it with a different codec.

flac --ogg source.wav -o audio-flac.oga
to encode the audio. This produces an Ogg FLAC file called audio-flac.oga.

oggzmerge video-theora.ogv audio-flac.oga -o final.ogv
to merge the video and audio tracks into the final Ogg video file, final.ogv.

Note that we're using the recently recommended file extensions for Ogg video and audio.

If you know an easier way to create Ogg Theora+FLAC files, please leave a note in the comments :-)

HTML5 for free media: Today on #whatwg

There has been a bit of FUD about Ogg Theora recently [2] [3]. So, over on #whatwg, one day before the W3C Video on the Web Workshop:

11:35:59	* Hixie casually removes Ogg from the spec and sees what happens
11:36:43	* othermaciej_ takes shelter
	...

The editor of the HTML5 draft specification, Ian Hickson (Hixie), sent this message :

I've temporarily removed the requirements on video codecs from the HTML5 spec, since the current text isn't helping us come to a useful interoperable conclusion. When a codec is found that is mutually acceptable to all major parties I will update the spec to require that instead and then reply to all the pending feedback on video codecs.

12:05:02	<kfish> Hixie!
12:11:47	* kfish throws a tantrum on behalf of the free software community
	...

However, the change didn't turn out to be so bad after all. The new text reads:

...; we need a codec that is known to not require per-unit or per-distributor licensing, that is compatible with the open source development model, that is of sufficient quality as to be usable, and that is not an additional submarine patent risk for large companies.

The previous draft stated no such requirements. As no rationale was given for choosing Ogg, that recommendation was easy to attack. Members of the MPEG LA, the cabal whose members receive money when people use content in MPEG formats, then had a fairly easy job of inciting flamewars on the whatwg list.

The new, clearer wording should allow more productive technical discussion, so that we can actually build an open standard which encourages anyone, anywhere, to publish their videos freely.

12:29:48	* kfish reads the replacement text and revokes the tantrum
12:30:15	<kfish> Hixie, actually you didn't casually remove Ogg, you made the case for Ogg stronger, so thankyou :-)
12:35:37	<Dashiva> "Lift the cat who was amongst the pigeons up and put him back on his pedestal for now."
12:35:40	<Dashiva> Poetic
12:37:49	<Hixie> kfish: :-)

Release: HOgg 0.3.0

Hogg is a commandline tool for manipulating Ogg files. It has subcommands, like hogg chop for cutting out bits of video, hogg info for telling you about the codecs, and hogg dump for hexdumping the packet data. It's basically a re-implementation of most of the stuff in liboggz, but the new features in hogg 0.3.0 such as chopping out a section of a file and adding Ogg Skeleton metadata, are not yet in oggz-tools.

$ hogg help chop
chop: Extract a section (specify start and/or end time)
Usage: hogg chop [options] filename ...

Examples:
  Extract the first minute of file.ogg:
    hogg chop -e 1:00 file.ogg

  Extract from the second to the fifth minute of file.ogg:
    hogg chop -s 2:00 -e 5:00 -o output.ogg file.ogg

  Extract only the Theora video stream, from 02:00 to 05:00, of file.ogg:
    hogg chop -c theora -s 2:00 -e 5:00 -o output.ogg file.ogg

  Extract, specifying SMPTE-25 frame offsets:
    hogg chop -c theora -s smpte-25:00:02:03::12 -e smpte-25:00:05:02::04 -o output.ogg file.ogg

Nevertheless, I'm continuing to work on both liboggz and hogg. liboggz, in pure C, is faster; hogg, in pure (but unoptimised) Haskell, is more correct. I spent a few hours earlier today tracking down a corner case in liboggz, coincidentally triggered by the chopping routines in libannodex. It reminded me that one of my first realizations about Haskell was that its sanity-checker often tells you about forgotten corner cases of algorithms.

Release: libfishsound 0.8.0

libfishsound provides a simple and consistent programming interface for decoding and encoding audio data using Xiph.Org codecs (Vorbis and Speex). This release includes compatibility with the floating point portion of the libfishsound development trunk API, in preparation for use with liboggplay. In order to build a minimal version of libfishsound for use with liboggplay, configure with encoding disabled in order to produce a smaller binary and to remove the dependency on libvorbisenc.

Trivial unit testing and coverage checking for C libraries

Recently I added a version script and the start of a test suite (under valgrind) to the Theora video codec. Apart from locating a trivial leak in the encoder, doing so turned up some interesting oddities.

The Version Script lists all public symbols, and tells the linker to only export these. This was added to avoid symbol clashes with other libraries. This usage is similar to a .def file for MSVC. A Version Script also allows pattern matching and definition of multiple API versions.

The tests I added are fairly trivial. One is a 'noop' test which simply creates and destroys each kind of data structure the library provides. By using GNU Automake's TESTS_ENVIRONMENT to (optionally) run the tests under valgrind, we can determine if the library contains memory leaks in its constructors and destructors.

One of the tests uses all of the theora_comment_*() API functions, and checks the correctness of return values and errors. If we're happy that a set of tests covers all API functions, then we can be reasonably happy that if it passes, the API is:

• completely exported by the linker (the test runs at all),
• does not contain any memory leaks (as valgrind doesn't complain),
• and is correctly implemented (as the test passes).

When using GNU Automake, make distcheck will fail if any tests fail. make distcheck should be used to create distribution tarballs, the point being that you ensure all tests pass before release. make distcheck also has other nice benefits like testing that install and uninstall works correctly.

One of the requirements for the theora reference implementation is to minimize dependencies. More detailed testing and analysis can be achieved with check and gcov, but the above is a fairly low-impact approach suitable for most C libraries.

Creative Commons tagging

To: linux-audio-user
Cc: advocacy@xiph.org

I've been following the rise and rise of music made with Linux, which have been announced on this list and Jan Weil has been listing.

Many of the released files have no licensing information. In most parts of the world, this implies "All Rights Reserved". If you are making music, or samples, that you are happy to share with others then you should consider tagging your files with a CreativeCommons license.

Embedding licensing information allows people using music browsers and search engines to _find your stuff_ (songs, samples, source materials -- it's up to you). We want Linux distributions to provide tools for people to find and use free media, and music made with Linux should be ready for that.

Creative Commons provide a guide to embedding licensing information and also more specific information about putting licensing information in Ogg Vorbis files.

This basically involves adding a LICENSE comment, such as:

LICENSE=Licensed to the public under http://creativecommons.org/licenses/by-sa/2.5/ verify at http://example.com/cclicenses.html

Using the commandline vorbis-tools, these tags can be added easily. To add licensing information to an existing Ogg Vorbis file:

  vorbiscomment -t "LICENSE=Licensed to the public ..." file.ogg

To add licensing information while encoding a WAV file to Ogg Vorbis:

  oggenc -c "LICENSE=Licensed to the public ..." file.wav

Please include the URL of the license you choose in the LICENSE tag. Information on CreativeCommons license choices is here.

Looking forward to a web of free music,

Conrad.

I ate snails

This week I have been staying with Thomas and Kristien, and their crazy cat Lunya. Last night I went to dinner at a Basque restaurant; when I got back, Thomas asked how it was.

<kfish> I ate snails. <thomasvs> Did you remove the shit from the end of the snails? <kfish> The wha???

In related work, we've been having Ogg fun preparing the GUADEC recordings.