Continuation Fest 2008: Continuations for video decoding and scrubbing

Yesterday was Continuation Fest 2008, at the University of Tokyo's campus in Akihabara (a very nice venue!). It was very well attended; latecomers overflowed to a second room and participated by video conference. It was a little strange to see so many people interested in such an obscure, troublesome and malignant expressively powerful programming construct; the breadth of talks made for a very inspiring and practical introduction to the theory, applications and implementation of continuations in many different languages.

I recommend reading Kenichi Asai's introduction to delimited continuations (slides [PDF]). He introduced the shift and reset operators through the problem of expressing exceptional control flow, and then explained how to use these to type (ie. determine a concrete type for) printf. The main point was that shift/reset provide a high-level abstraction over control flow, with minimal impact on the implementation of your existing functions.

Oleg Kiselyov demonstrated some new code for transactional web applications, using delimited continuations for explicit state sharing between parallel connections. The result is that the user has a consistent view across multiple tabs are open on the same site, and the state is transactional so that there is no need for warnings like "Do not press the BUY button more than once!". He said that everyone already understands delimited continuations, they just don't realize it.

The topic of my presentation at Continuation Fest was Continuations for video decoding and scrubbing:

Playback of encoded video involves scheduling the decoding of audio and video frames and synchronizing their playback. "Scrubbing" is the ability to quickly seek to and display an arbitrary frame, and is a common user interface requirement for a video editor. The implementation of playback and scrubbing is complicated by data dependencies in compressed video formats, which require setup and manipulation of decoder state.

We present the preliminary design of a continuation-based system for video decoding, reified as a cursor into a stream of decoded video frames. Frames are decoded lazily, and decoder states are recreated or restored when seeking. To reduce space requirements, a sequence of decoded frames can be replaced after use by the continuation which created them. We outline implementations in Haskell and C.

• Slides [383KB PDF]
• Article [215KB PDF]

I'll be introducing the code for this over the next few months. Whereas in my presentation about video player internals at BOSSA I outlined the problem space in designing a multimedia architecture, at Continuation Fest I tried to break it down into subproblems and considered useful data structures and programming techniques for dealing with them.

I got a lot of great feedback, and I think I succeeded in my mission to introduce this problem space to some really smart people. Thanks particularly to Chung-chieh Shan for some insightful ideas about how to deal with existing stateful codec implementations. It was also very interesting to talk with Shinji Kono about Continuation-based C (cBc) (slides [HTML tarball]), a C-like language capable of expressing continuations, non-local jumps, multiple function entry-points, and assorted other ways to shoot yourself in the foot. He suggested that it was designed for exactly the kind of thing I'm doing, and I'll be interested to try it out. It is implemented in a modifed GCC 4.x as an RTL code generator, so should now be (fairly) architecture-independent.

Thanks to the organizers of Continuation Fest 2008 for putting together such a useful and interesting event. I look forward to implementing just some of the things I learned :-)