blog.kfish.org

:rikaichan for Vimperator

Conrad Parker — Thu, 17 Apr 2008 14:45:00 +0000

Some of my favourite Firefox plugins are:

Rikaichan, a Japanese dictionary, which adds instant translation popups when you mouse over a word;
Vimperator, which provides vi-like user interface;
and Hide Tab Bar, because Vimperator's buffer list is more useful.

Vimperator hides the menu bar by default. Tools->Toggle Rikaichan has no default keybinding, and the keybindings to navigate the menubar are not available if the menubar is not visible, so Rikaichan can no longer be activated.

The following adds a vimperator command :rikaichan; save it to .vimperator/plugin/toggleRikaichan.js:

(function(){
    vimperator.commands.add(new vimperator.Command(
        ['rikaichan', 'rikai'],
        function(){
            rcxMain.inlineToggle();
        }
    ))
}) ();

It is aliased to :rikai for short, but unfortunately vimperator won't recognize :理解. Thanks to ktsukagoshi for the explanation of how to write a vimperator plugin (vimperatorのプラグインの作成).

Remember, the interface is inside your mind.

Continuation Fest 2008: Continuations for video decoding and scrubbing

Conrad Parker — Mon, 14 Apr 2008 07:30:00 +0000

Yesterday was Continuation Fest 2008, at the University of Tokyo's campus in Akihabara (a very nice venue!). It was very well attended; latecomers overflowed to a second room and participated by video conference. It was a little strange to see so many people interested in such an ~~obscure, troublesome and malignant~~ expressively powerful programming construct; the breadth of talks made for a very inspiring and practical introduction to the theory, applications and implementation of continuations in many different languages.

I recommend reading Kenichi Asai's introduction to delimited continuations (slides [PDF]). He introduced the shift and reset operators through the problem of expressing exceptional control flow, and then explained how to use these to type (ie. determine a concrete type for) printf. The main point was that shift/reset provide a high-level abstraction over control flow, with minimal impact on the implementation of your existing functions.

Oleg Kiselyov demonstrated some new code for transactional web applications, using delimited continuations for explicit state sharing between parallel connections. The result is that the user has a consistent view across multiple tabs are open on the same site, and the state is transactional so that there is no need for warnings like "Do not press the BUY button more than once!". He said that everyone already understands delimited continuations, they just don't realize it.

The topic of my presentation at Continuation Fest was Continuations for video decoding and scrubbing:

Playback of encoded video involves scheduling the decoding of audio and video frames and synchronizing their playback. "Scrubbing" is the ability to quickly seek to and display an arbitrary frame, and is a common user interface requirement for a video editor. The implementation of playback and scrubbing is complicated by data dependencies in compressed video formats, which require setup and manipulation of decoder state.
We present the preliminary design of a continuation-based system for video decoding, reified as a cursor into a stream of decoded video frames. Frames are decoded lazily, and decoder states are recreated or restored when seeking. To reduce space requirements, a sequence of decoded frames can be replaced after use by the continuation which created them. ~~We outline implementations in Haskell and C.~~

Slides [383KB PDF]
Article [215KB PDF]

I'll be introducing the code for this over the next few months. Whereas in my presentation about video player internals at BOSSA I outlined the problem space in designing a multimedia architecture, at Continuation Fest I tried to break it down into subproblems and considered useful data structures and programming techniques for dealing with them.

I got a lot of great feedback, and I think I succeeded in my mission to introduce this problem space to some really smart people. Thanks particularly to Chung-chieh Shan for some insightful ideas about how to deal with existing stateful codec implementations. It was also very interesting to talk with Shinji Kono about Continuation-based C (cBc) (slides [HTML tarball]), a C-like language capable of expressing continuations, non-local jumps, multiple function entry-points, and assorted other ways to shoot yourself in the foot. He suggested that it was designed for exactly the kind of thing I'm doing, and I'll be interested to try it out. It is implemented in a modifed GCC 4.x as an RTL code generator, so should now be (fairly) architecture-independent.

Thanks to the organizers of Continuation Fest 2008 for putting together such a useful and interesting event. I look forward to implementing just some of the things I learned :-)

Release: Sweep 0.9.3

Conrad Parker — Fri, 11 Apr 2008 21:43:00 +0000

This is a bugfix release of Sweep, addressing CVE-2008-1686. For details, see my earlier post about libfishsound 0.9.1. Thanks to Peter Shorthose for managing this release.

Release: libfishsound 0.9.1

Conrad Parker — Mon, 07 Apr 2008 01:08:00 +0000

This is a maintenance release, fixing a security vulnerability in Speex header processing as outlined in oCERT 2008-02. When used in a client for web video content, as in the OggPlay Firefox Plugin or the Ogg DirectShow filters, a specially crafted Ogg Speex stream hosted on a server could be used to allow an attacker to execute arbitrary code on the client system. The OggPlay plugin binaries available from www.annodex.net have already been updated.

Details

The Speex header contains a 32-bit modeID field, interpreted by libspeex as a signed int (spx_int32_t) The normal way to use this is to index into a global mode list to retrieve a SpeexMode *:

mode = (SpeexMode *)speex_mode_list[modeID];

and then use that to set up a decoder:

st = speex_decoder_init(mode);

This calls speex_decoder_init() in libspeex, which looks like:

void *speex_decoder_init(const SpeexMode *mode)
{
   return mode->dec_init(mode);
}

So if you don't check that the modeID given in the stream header is within the bounds of speex_mode_list[], arbitrary code can be executed. libfishsound was checking the upper bound (modeID < SPEEX_NB_MODES) but was not checking against negative values.

Discussion

This header processing is all boilerplate, and a reference implementation is given in speexdec.c. I took a copy of that about 7 years ago for Sweep, which I then adapted for libfishsound. The current reference speexdec.c does not have this bug.

For the Symbian port of Speex we created a function which returns the desired mode given a modeID, rather than having application code index into a global mode list. I wrote and committed speex_get_mode() to libspeex in September 2004, and it does the correct bounds checking. So if I'd been using that function in libfishsound then today's problem would never have happened. As it turns out, the libfishsound svn trunk version of speex.c does use that function. As far as I am aware, the OggPlay plugin binaries have always been built against the libfishsound svn trunk, so they were never vulnerable in the first place. However, recent tarball releases of libfishsound have been coming of a separate branch, so the advisory is valid for applications linked against those releases.

Finally, I sent a patch to Jean-Marc Valin yesterday which entirely removes the possibility of this bug happening again by bounding the mode values returned by speex_packet_to_header() in libspeex. It will be available very soon in a libspeex release.

Acknowledgements

Thanks to the team at oCERT for the efficient reporting of this advisory, and to the anonymous submitter for the details. I was able to patch the offending branches, which allowed j^ to build and upload new OggPlay plugin binaries (within 24 hours of contact by oCERT).

Release: HOgg 0.4.0

Conrad Parker — Mon, 24 Mar 2008 18:23:00 +0000

HOgg is a Haskell library and commandline tool for manipulating Ogg files. This release contains a bunch of code written during FOMS and LCA 2008, including a new sort subcommand and proper handling of Skeleton when merging and ripping files. Full details are in the release notes.

sort implementation

My favourite part is the implementation of the new sort subcommand:

sort :: [OggPage] -> [OggPage]
sort = sortHeaders . listMerge . demux

This is somewhat shorter than the equivalent C implementation, oggz-sort.c — Haskell affords abstraction whereas in C it's a trade-off. sortHeaders is a long (21 line) function that re-orders header pages according to the Theora and Skeleton specifications, and listMerge is a generic list merging function, also used in the merge subcommand. demux is tiny:

demux :: (Serialled a) => [a] -> [[a]]
demux = classify serialEq

You can read that as "demux is classification by serial number": classify is a generic list function, classifying list elements according to some criterion you give it. Here, for example, the list of pages:

[Video0, Audio0, Video1, Audio1, Audio2, Audio3, Video2, Audio4, Video3, ...]

will get classified into two separate lists:

[[Video0, Video1, Video2, Video3, ...],
 [Audio0, Audio1, Audio2, Audio3, Audio4, ...]]

This is done lazily, meaning that the processing is done on the fly and big intermediate lists are not constructed in memory. Video0, Audio0 will be passed through listMerge and sortHeaders and written to disk by the consumer of sort well before Video103 and Audio5007 are seen.

Documentation improvements and self-checking

The help for each subcommand now contains long descriptions, mostly similar to the man pages of the Oggz tools. The descriptions also have explicit sections describing how Theora, Skeleton and chained files are handled. The example commandlines for each subcommand use the Ogg MIME types and file extensions that we are now recommending in Xiph.Org.

The best bit though is hogg selfcheck, which checks that the help examples are valid. It checks that all the example commandlines pass through getOpt without errors, and that all file extensions used in options are valid. This is the kind of nice touch which would have been a pain to code up in C, but fell out cleanly in the Haskell implementation. As it is fairly cheap to run (and printing help text is hardly a performance-critical operation), this option is also silently run after printing out any help output at all, so that such errors are more likely to be found and reported. The same commit that introduced hogg selfcheck also fixed two such documentation errors which were found by this option :-)

Release: xsel 1.2.0

Conrad Parker — Mon, 24 Mar 2008 18:17:00 +0000

XSel is a command-line tool for manipulating the X selection. This is a maintenance release, improving argument handling, documentation and X11 library detection.

BOSSA 2008: Video Player Internals

Conrad Parker — Mon, 24 Mar 2008 17:29:00 +0000

Last week I attended BOSSA, a conference on open source software for mobile embedded platforms, organized by INdT. It was held in the town of Porto de Galinhas, Brazil. Since then I have been hanging out in the INdT labs in Recife, hacking on xine, catching up with friends and exploring the old city.

The topic of my presentation at BOSSA was Video Player Internals:

Embedded platforms put demands on latency and memory use. Video playback makes these difficult to guarantee. This presentation discusses the architecture of video players, and the problems imposed on them by the design of video codecs and their containers. To explain these problems we look at both proprietary and open source formats (MPEG, Ogg, Theora, Dirac, etc.) and evaluate open source video players in this context. We particularly examine xine and GStreamer, and introduce the minimal architecture of OggPlay.

Slides [613KB PDF]
Article [330KB PDF]

I'm very grateful to INdT for the opportunity to attend, it was an awesome conference in a very beautiful part of the world.

Release: liboggz 0.9.7

Conrad Parker — Fri, 15 Feb 2008 08:54:00 +0000

There's been a whole bunch of work on liboggz recently; it deserves a few more weeks of shaking out and perhaps some updated Win32/MacOS support before it gets 1.0 slapped on it.

liboggz 0.9.7 includes a new tool called oggz-sort, which addresses a problem with some encoders that Shane Stephens brought up at FOMS. The discussion was going around in circles, so my response was to write this C code. It implements a function that Shane has written but not yet released in his OCaml implementation of Ogg (oogg), and which I've written but not yet released in my Haskell implementation (HOgg). Of course, people will take this version more seriously because it's written in C.

From oggz-sort (1):

oggz-sort sorts an Ogg file, interleaving pages in order of presentation time. It correctly interprets the granulepos timestamps of Ogg Vorbis, Speex, FLAC and Theora bitstreams, and all bitstreams of Annodex files.
Some encoders produce files with incorrect page ordering; for example, some audio and video pages may occur out of order. Although these files are usually playable, it can be difficult to accurately seek or scrub on them, increasing the likelihood of glitches during playback. Players may also need to use more memory in order to buffer the audio and video data for synchronized playback, which can be a problem when the files are viewed on low-memory devices.
The tool oggz-validate can be used to check the relative ordering of packets in a file. If out of order packets are reported, use oggz-sort to fix the problem.

This release also adds support for the experimental CELT audio codec, which is being developed by Jean-Marc Valin (the primary author of Speex). CELT is designed as a low-latency codec for high-quality audio. When wiretapping conversations encoded in CELT, we recommend that you record using the Ogg container format. You can then use oggz-tools to help with your analysis.

Release: xsel 1.1.0

Conrad Parker — Sat, 09 Feb 2008 23:30:00 +0000

This is a story about the meaning of "version 1.0". A few weeks ago I released version 1.0 of xsel, a simple commandline utility for manipulating the X selection and clipboard. I chose to call it 1.0 after recalling a discussion with André Pang, about how the meaning of version numbers in open source software tends to differ from that in other software communities. For example, it is often advised not to buy the first version of a proprietary software product as it is sure to be buggy and incomplete; open source projects on the other hand often aspire to 1.0 being a major milestone, bug-free and fully-functional. The Windows and Mac freeware and shareware communities tend to follow a middle ground, content to release a useful but incomplete version 1.0, but thereafter avoiding the quick version creep that afflicts companies with marketing departments (and version-limited support contracts).

I'll argue that that middle way makes for more meaningful version numbers. Putting the label "1.0" on a release should be your way of saying that it's the first version that:

won't hose a user's system, and
hopefully does something useful.

Any version number less than 1.0 is sending out a signal that the software isn't quite ready yet; perhaps that you could lose or damage data by using it. Many people intuitively wait for version 1.0 before trying out some software, and this is fair enough. In fact, we need a way of warning that a project isn't ready for widespread adoption, that it could damage data, that the tarball is only out there so that other people can grab the code and help fix bugs. That's what version numbers less than 1.0 mean.

After 1.0, you can keep adding features and bumping the version number, working towards version 2.0 which perhaps does useful things in a different way. And from 2.0, onwards to 3.0 and beyond; integers are cheap. The important thing is not fall into the trap of thinking of 1.0 as some kind of asymptotic upper bound representing the perfect release.

Back to xsel. At first I wrote up release notes as version 0.9.7, but then then remembered that discussion with André and realized that it should really just be 1.0. More to the point the previous release (in July 2001, which went five years without a bug report or patch) should have been 1.0. So yeah, it was a good feeling to just write "1.0" and send it out.

The morning after releasing 1.0 I got a report from someone who couldn't get it to compile -- turns out they didn't have the X11 development libraries installed, and for some reason I had commented out the checks for that in configure.ac while testing something or other a while ago. As a result, the configure script wasn't check for its only dependency. I considered doing a canonical 1.0.1 (LOL) release. Within the next day, though, I got a report about how to fix handling of COMPOUND_TEXT, an archaic way of handling international text (since superceded by UTF8_STRING). And there follows the next lesson (as berated by Raster): random bug reports stream in after a release, not before. Erik de Castro Lopo is up to his 20th pre-release of libsndfile 1.0.18; each pre-release he gets bombarded with reports.

Anyway, the post-1.0 bug reports have died down, so today I'm releasing version 1.1.0 of xsel. "This release adds basic support for COMPOUND_TEXT and fixes a configuration bug". And I'm still waiting to hear good uses of xsel --append and xsel --follow.

FOMS, LCA Multimedia 2008: Videos

Conrad Parker — Fri, 08 Feb 2008 05:58:00 +0000

I arrived back in Japan after a few awesome weeks in Australia for FOMS and LCA. The weather in Melbourne was great, and the food was fantastic.

Between FOMS and LCA, dozens of free multimedia software developers were in town. It was the first time that developers of Dirac, Speex, Theora, Vorbis, Ogg, and most of the Annodex crew were all in the same place, so we spent most of the week of LCA holed up in a room designing content description and packaging formats. One immediate outcome will be finalization of the Dirac mapping into the Ogg container.

I organised the multimedia miniconf on the Monday of LCA, which was jam-packed with excellent presentations and lightning talks. Thanks to everyone who came, and talked, and video recorded. There were plenty of comments along the lines of it being "pretty hardcore for a miniconf". If you are interested in helping with next year's LCA Multimedia, or have friends in Hobart who might be able to help, let's start throwing around ideas. In particular, quite a few people asked what happened to the audio miniconf parties from a few years ago, and it might be a good chance to revive those ...

Videos

The following pages contain embedded videos of the presentations from these events, and the multimedia-related presentations from LCA:

FOMS Proceedings: introductions by the participants
LCA Multimedia: Dirac, Xiph, EngageMedia, FFADO and many others
Multimedia talks @LCA: PulseAudio, Ogg, Theora, Telepathy, Farsight ...

The videos on these pages are embedded with mv_embed, which supports playback via the OggPlay plugin for Firefox, vlc-plugin or generic application/ogg. mv_embed is a JavaScript library by Michael Dale of MetaVid. It is really easy to use, you just include that library (<script src="...">) and then write <video src="..."> anywhere in your page. No need to wait for native HTML5 support in your browser :-)

Release: liboggz 0.9.6

Conrad Parker — Sun, 13 Jan 2008 05:11:00 +0000

This release of Oggz 0.9.6 contains a new tool, oggz-comment, which can be used to edit the basic metadata (title, producer, copyright etc.) of Ogg Theora files. The library also has some pretty major improvements to the way it works out timestamps and does seeking, mostly the work of Shane Stephens.

In media files, timing and synchronization is extremely important. If the image and audio start to go out of sync, it is very noticeable and the video quickly becomes unwatchable. When you scan through a file you often need to decode a lot more data than you actually display. This is particularly the case when you jump backwards, which is common in a user interface that supports scrubbing. As video frames are stored as a difference relative to earlier (or later) frames, you end up needing to secretly jump further back in the file to the previous keyframe, and then decode many frames up to the one you actually want to show. For a smooth user experience you need to do this as quickly as possible.

Ogg has some interesting framing properties. Given that timing is so important, you might expect that every packet has its precise timing information associated with it. In Ogg, it turns out not to be so. Packets are stored in pages, and there is only one timestamp per page. It is common for many audio packets to be crammed onto one page; the timing information for all the rest is not stored in the file. On the other hand, the encoded data for video keyframes is usually much larger, and spans multiple pages. Only the last packet on a page has its timestamp recorded, so if the keyframe is followed by an a much smaller packet of frame data in the same page, the timestamp for the keyframe will be lost. For these reasons I tend to refer to Ogg as a "lossy" container.

In order to minimize these problems, liboggz now inspects the encoded data in order reconstruct the expected granulepos (corresponding to a timestamp) for every packet in an Ogg stream. This allows applications to use reliable timestamps, even though these are only sparsely recorded in most Ogg streams. This is not as easy as it sounds, particularly for Ogg Vorbis. To get a flavour of what's involved, read Shane's rant in the comments, explaining how to calculate Vorbis timestamps.

For an in-depth discussion, come to Ralph Giles' talk at linux.conf.au, Seeking is hard: Ogg design internals.

Release: xsel 1.0.0

Conrad Parker — Sat, 12 Jan 2008 15:53:00 +0000

XSel is a command-line program for getting and setting the contents of the X selection. You can use xsel in shell scripts and desktop keybindings, so that the contents of the X selection are available to command arguments:

mozilla --remote "openurl(`xsel`)"

This release adds UTF-8 support and fixes various bugs. The last version of XSel was 0.9.6, released sometime around 2001. It may have been the first version also. For some reason a bunch of patches came in recently, and I've had the joy of revisiting this project.

For old time's sake, my thoughts on ICCCM. (Warning: explicit language). Back then I made a point of implementing as much of that crack as possible. You can even tell applications to delete their selected text:

To delete the contents of the selection: xsel --delete

(This really works, you can try it on xedit to remotely delete text in the editor window).

This time around, of course, nothing does what the docs say anymore. So we ignore the details in the 2001 proposal for Inter-Client Exchange of Unicode Text and just grunt atoms at the selection owner until they yield all their secrets. And now, finally, xsel works on Japanese.

People have come up with some interesting uses for xsel over the years, but nobody has yet come up with a nifty use for the following options:

To append to the X selection: xsel --append < file

To follow a growing file: xsel --follow < file

Any ideas?

Release: libfishsound 0.9.0

Conrad Parker — Sat, 12 Jan 2008 10:34:00 +0000

Now libfishsound 0.9.0 supports FLAC, the Free Lossless Audio Codec. The patches were originally contributed by Tobias Gehrig in 2004. There hasn't been much use of Ogg FLAC, whereas FLAC in its native encoding is very popular. However, the point of the Ogg mapping is to allow FLAC to be used in parallel with other codecs, in particular as the audio codec for video files. The combination of Theora video and FLAC audio can be very useful for music videos, where you might not care too much if the image has lost some quality but you want the sound to be as good as possible.

However, creating such a file isn't so easy. Let's say you have a source video, like GrooveTV #204 - Jacob Fred Jazz Odyssey. I took the MPEG-1 file as recommended; for clarity, let's call it source.mpg. To make a video to test on, I did:

ffmpeg2theora source.mpg
to encode the video into an Ogg file containing Theora video and Vorbis audio. This produces source.ogv.

oggzrip -c theora source.ogv -o video-theora.ogv
to extract only the Theora video track, into video-theora.ogv.

mpg123 -w source.wav source.mpg
to extract the audio to a wav file, source.wav. Here the audio in the source material was encoded as MPEG I layer II; obviously if you were producing a music video, you'd skip this step and encode FLAC from the original recording. I didn't have that here, and I just wanted a file I could test on.
However, at the least this step means that no further artifacts are introduced into the audio, other than those which were present in the MPEG encoding. If the only source material you have is already encoded, you don't want to degrade it further by re-encoding it with a different codec.

flac --ogg source.wav -o audio-flac.oga
to encode the audio. This produces an Ogg FLAC file called audio-flac.oga.

oggzmerge video-theora.ogv audio-flac.oga -o final.ogv
to merge the video and audio tracks into the final Ogg video file, final.ogv.

Note that we're using the recently recommended file extensions for Ogg video and audio.

If you know an easier way to create Ogg Theora+FLAC files, please leave a note in the comments :-)

HTML5 for free media: Today on #whatwg

Conrad Parker — Tue, 11 Dec 2007 04:04:00 +0000

There has been a bit of FUD about Ogg Theora recently [2] [3]. So, over on #whatwg, one day before the W3C Video on the Web Workshop:

11:35:59 * Hixie casually removes Ogg from the spec and sees what happens

11:36:43 * othermaciej_ takes shelter

...

The editor of the HTML5 draft specification, Ian Hickson (Hixie), sent this message :

I've temporarily removed the requirements on video codecs from the HTML5 spec, since the current text isn't helping us come to a useful interoperable conclusion. When a codec is found that is mutually acceptable to all major parties I will update the spec to require that instead and then reply to all the pending feedback on video codecs.

12:05:02 <kfish> Hixie!

12:11:47 * kfish throws a tantrum on behalf of the free software community

...

However, the change didn't turn out to be so bad after all. The new text reads:

...; we need a codec that is known to not require per-unit or per-distributor licensing, that is compatible with the open source development model, that is of sufficient quality as to be usable, and that is not an additional submarine patent risk for large companies.

The previous draft stated no such requirements. As no rationale was given for choosing Ogg, that recommendation was easy to attack. Members of the MPEG LA, the cabal whose members receive money when people use content in MPEG formats, then had a fairly easy job of inciting flamewars on the whatwg list.

The new, clearer wording should allow more productive technical discussion, so that we can actually build an open standard which encourages anyone, anywhere, to publish their videos freely.

12:29:48 * kfish reads the replacement text and revokes the tantrum

12:30:15 <kfish> Hixie, actually you didn't casually remove Ogg, you made the case for Ogg stronger, so thankyou :-)

12:35:37 <Dashiva> "Lift the cat who was amongst the pigeons up and put him back on his pedestal for now."

12:35:40 <Dashiva> Poetic

12:37:49 <Hixie> kfish: :-)

Release: HOgg 0.3.0

Conrad Parker — Thu, 06 Dec 2007 13:31:00 +0000

Hogg is a commandline tool for manipulating Ogg files. It has subcommands, like hogg chop for cutting out bits of video, hogg info for telling you about the codecs, and hogg dump for hexdumping the packet data. It's basically a re-implementation of most of the stuff in liboggz, but the new features in hogg 0.3.0 such as chopping out a section of a file and adding Ogg Skeleton metadata, are not yet in oggz-tools.

$ hogg help chop
chop: Extract a section (specify start and/or end time)
Usage: hogg chop [options] filename ...

Examples:
  Extract the first minute of file.ogg:
    hogg chop -e 1:00 file.ogg

  Extract from the second to the fifth minute of file.ogg:
    hogg chop -s 2:00 -e 5:00 -o output.ogg file.ogg

  Extract only the Theora video stream, from 02:00 to 05:00, of file.ogg:
    hogg chop -c theora -s 2:00 -e 5:00 -o output.ogg file.ogg

  Extract, specifying SMPTE-25 frame offsets:
    hogg chop -c theora -s smpte-25:00:02:03::12 -e smpte-25:00:05:02::04 -o output.ogg file.ogg

Nevertheless, I'm continuing to work on both liboggz and hogg. liboggz, in pure C, is faster; hogg, in pure (but unoptimised) Haskell, is more correct. I spent a few hours earlier today tracking down a corner case in liboggz, coincidentally triggered by the chopping routines in libannodex. It reminded me that one of my first realizations about Haskell was that its sanity-checker often tells you about forgotten corner cases of algorithms.

Survey: Haskell Unicode support

Conrad Parker — Thu, 15 Nov 2007 11:50:00 +0000

Haskell source is interpreted as UTF-8, but internally the data is stored as Unicode code points. However the generic show method does not serialize Strings as UTF-8 (when using GHC). So, when reading or writing documents it is necessary to introduce an explicit conversion from or to the desired character set. This article outlines how to use Unicode in Haskell, and surveys three alternatives for character set conversion: iconv, utf8-string and encoding, providing working examples for each.

Unicode in Haskell source

The Haskell Prime standardization wiki contains discussions of Unicode in Haskell Source, and of ways of handling Char as Unicode. In particular, GHC (as of release 6.6, early Jan 2006) interprets source files as UTF-8. Hence the following is a valid source file:

import System.Time

main :: IO ()
main = do
  time <- getClockTime
  cal <- toCalendarTime time
  putStrLn $ dayName $ ctWDay cal

dayName :: Day -> String
dayName d = case d of
              Monday -> "月曜日"
              Tuesday -> "火曜日"
              Wednesday -> "水曜日"
              Thursday -> "木曜日"
              Friday -> "金曜日"
              Saturday -> "土曜日"
              Sunday -> "日曜日"

The dayName function provides the Japanese name for a given Day. However the main function, which tries to print that onto stdout, dumps it without any character set conversion, truncating each character to 8 bits. In order to control the output charset, we need to use a Unicode conversion library. The three libraries iconv, utf8-string and encoding have similar purposes but some different features.

iconv

Description:	Binding to C iconv() function
Author:	Duncan Coutts
darcs get	`http://code.haskell.org/iconv/`
Exports:	`Codec.Text.IConv`
Interface:	`ByteString.Lazy`
Advantages:	Speed, coverage of charset support
Disadvantages:	Portability: requires POSIX `iconv()`

This is a Haskell binding to the iconv() C library function, providing a lazy ByteString interface. The only module exported is Codec.Text.IConv, which provides a single function:

-- | Convert fromCharset toCharset input output
convert :: String -> String -> Lazy.ByteString -> Lazy.ByteString

where fromCharset and toCharset are the names of the input and output character set encodings, and input and output are the input and output text as lazy ByteStrings. An example program to convert the encoding of an input file, similar to the GNU iconv program, is given in examples/hiconv.hs. The guts of that program is:

        output = convert (fromEncoding config) (toEncoding config) input

which is somewhat clearer than the brain-damaged interface exported by the C library. Exceptions are provided for handling unsupported conversions, invalid and incomplete characters. These errors can be silently ignored if desired by calling convertFuzzy instead. As this library wraps the system iconv() implementation, all character sets supported on the underlying system are available. The Lazy.ByteString interface works directly on the memory buffers used by the C library, which may give a speed advantage for large conversions. Note however that the iconv() C library function is defined by POSIX.1-2001 and may not be available on some older systems. In most such cases it should be possible to install GNU libiconv separately.

utf8-string

Description:	Simple UTF-8 conversion library
Author:	Eric Mertens
darcs get	`http://code.haskell.org/utf8-string/`
Exports:	`Codec.Binary.UTF8.String, System.IO.UTF8`
Interface:	`String`
Advantages:	Simplicity
Disadvantages:	Only supports UTF-8 conversions

This library contains both a simple module for data conversion with a String interface, and a useful IO module. The String conversion module, Codec.Binary.UTF8.String, provides two pairs of complementary encoding and decoding functions:

-- | Encode a string using 'encode' and store the result in a 'String'.
encodeString :: String -> String

-- | Decode a string using 'decode' using a 'String' as input.
-- | This is not safe but it is necessary if UTF-8 encoded text
-- | has been loaded into a 'String' prior to being decoded.
decodeString :: String -> String

-- | Encode a Haskell String to a list of Word8 values, in UTF8 format.
encode :: String -> [Word8]

-- | Decode a UTF8 string packed into a list of Word8 values, directly to String
decode :: [Word8] -> String

I guess "not safe" in the comment for decodeString refers to type-safety; for example this function doesn't stop you from trying to decode the same text twice, whereas if you tried that with the plain decode function, the compiler would point out your bug for you. To see how this might look in the wild, the following is a complete "Hello World" web application (err, CGI script) in Japanese:

import Codec.Binary.UTF8.String
import Network.CGI hiding (Html)
import Text.Html

main :: IO ()
main = runCGI $ handleErrors cgiMain

cgiMain :: CGI CGIResult
cgiMain = do
    setHeader "Content-Type" "text/html; charset=utf-8"
    output $ renderHtml $ h1 << encodeString "おはよう御座います！"

The utf8-string library also includes an entire IO module, System.IO.UTF8, exporting print, putStr, putStrLn, getLine, readLn, readFile, writeFile, appendFile, getContents, hGetLine, hGetContents, hPutStr, hPutStrLn. These essentially wrap the default IO functions in encodeString and decodeString, which you may find convenient if you are doing lots of UTF-8 processing. This library is tiny, and implemented natively in Haskell so there are no portability issues. As it works directly on ByteStrings it should be sufficiently fast for practical purposes. Of course, if you need to do conversions to or from character sets other than UTF-8, you will need to use a different library.

encoding

Description:	Native Haskell charset conversion library
Author:	Henning Günther
darcs get	`http://code.haskell.org/encoding/`
Exports:	`Data.Encoding.*, System.IO.Encoding`
Interface:	`ByteString.Lazy`
Advantages:	Portable; covers more charsets than utf8-string
Disadvantages:	Covers fewer charsets than iconv

Data.Encoding provides native Haskell implementations for encoding and decoding of many common character sets: ASCII, UTF8, UTF16, UTF32, ISO8859[1-16], CP125[0-8], KOI8R, and GB18030, as well as BootString (for Punycode). For each of these, it implements an Encoding interface:

{- | Represents an encoding, supporting various methods of de- and encoding.
     Minimal complete definition: encode, decode
 -}
class Encoding enc where
        -- | Encode a 'String' into a strict 'ByteString'. Throws the
        --   'HasNoRepresentation'-Exception if it encounters an unrepresentable
        --   character.
        encode :: enc -> String -> ByteString
        -- | Encode a 'String' into a lazy 'Data.ByteString.Lazy.ByteString'.
        encodeLazy :: enc -> String -> LBS.ByteString
        encodeLazy e str = LBS.fromChunks [encode e str]
        -- | Whether or not the given 'Char' is representable in this encoding. Default: 'True'.
        encodable :: enc -> Char -> Bool
        encodable _ _ = True
        -- | Decode a strict 'ByteString' into a 'String'. If the string is not
        --   decodable, a 'DecodingException' is thrown.
        decode :: enc -> ByteString -> String
        decodeLazy :: enc -> LBS.ByteString -> String
        decodeLazy e str = concatMap (decode e) (LBS.toChunks str)
        -- | Whether or no a given 'ByteString' is decodable. Default: 'True'.
        decodable :: enc -> ByteString -> Bool
        decodable _ _ = True

Notice that this interface provides exceptions for handling unrepresentable characters. Instances of Encoding can be found by importing charset-specific modules; each simply exports a value with the same name as the module, ie. Data.Encoding.ISO88592 exports ISO88592, which is an instance of Encoding. Here is a "Hello World" CGI in Polish, using ISO-8859-2:

import Data.Encoding
import Data.Encoding.ISO88592
import Data.ByteString.Char8
import Network.CGI hiding (Html)
import Text.Html

main :: IO ()
main = runCGI $ handleErrors cgiMain

cgiMain :: CGI CGIResult
cgiMain = do
    setHeader "Content-Type" "text/html; charset=iso-8859-2"
    output $ renderHtml $ h1 << (unpack $ encode ISO88592 "Cześć")

You'll notice the call to the unpack to convert the ByteString into a plain String as expected by Html. The encoding library also provides a way to select an encoding by name:

-- | Takes the name of an encoding and creates a dynamic encoding from it.
encodingFromString :: String -> DynEncoding

(Anything which is a DynEncoding is by definition an instance of Encoding). So we could choose the encoding at runtime, or we can just be lazy and pick encodings by name. If we do this, we don't need to import the charset-specific module, and we can replace the last line of our CGI with:

    let enc = encodingFromString "ISO-8859-2"
    output $ renderHtml $ h1 << (unpack $ encode enc "Cześć")

The encoding library also provides a pair of functions for converting character sets directly between two ByteStrings:

-- | This decodes a string from one encoding and encodes it into another.
recode :: (Encoding from,Encoding to) => from -> to -> ByteString -> ByteString

recodeLazy :: (Encoding from,Encoding to) => from -> to -> Lazy.ByteString -> Lazy.ByteString

The System.IO.Encoding module does not try to provide as many convenience functions as the similar module provided by utf8-string, providing only the generic hGetContents and hPutStr. However, it does provide a way of retrieving the current system's default encoding (when used on systems supporting POSIX.1-2001 nl_langinfo()), which utf8-string lacks.

-- | Like the normal 'System.IO.hGetContents', but decodes the input using an
--   encoding.
hGetContents :: Encoding e => e -> Handle -> IO String

-- | Like the normal 'System.IO.hPutStr', but encodes the output using an
--   encoding.
hPutStr :: Encoding e => e -> Handle -> String -> IO ()

-- | Returns the encoding used on the current system.
getSystemEncoding :: IO DynEncoding

As this library is native Haskell it is portable, and as it uses lazy ByteStrings it can be fast. While it does not (yet) provide as many character sets as your system's iconv(), it does support many of the most commonly used ones.

Notes

The libraries surveyed here are under fairly active maintenance, and there are rumours of unifying their implementations. Nevertheless the existing interfaces are fairly similar where common functionality exists. Historically, all serialized data was handled in Haskell as Strings, and there was a legitimate concern that transparently converting the character set of arbitrary Strings could mangle data. The newer ByteString and Binary interfaces may allow future Haskell standards to clearly disambiguate binary and textual data, and simply serialize Strings as UTF-8 by default. Although it might be nice to "simply" serialize Strings as UTF-8, show is the wrong place to do it. Haskell's Read/Show serialization serializes to String, which is a list of Char, ie. a list of abstract Unicode code points. Character set conversion should rather happen on conversion to [Word8], at which point byte values become significant. This also encompasses direct conversions to ByteString, and the internals of primitive IO functions such as:

putChar    :: Char -> IO ()
putChar    =  primPutChar

getChar    :: IO Char
getChar    =  primGetChar

, getContents, readFile, writeFile, and appendFile defined in the Haskell Prelude, and the various character IO functions on Handles defined in System.IO. Whether or not this conversion can be done everywhere transparently, and backwards-compatibly, is an open issue for Haskell Prime. Meanwhile these libraries provide useful interfaces for explicit [Word8] and ByteString conversion, and various IO wrappers.

Summary

Although all Haskell Strings are Unicode, Haskell98 does not specify a character set representation for their IO. Unicode strings can be written directly into Haskell source files and hence exist as data within a program, but character set conversion is required if you wish to read or write these Strings in files, user input or on the network. We looked at ways of dealing with Unicode in Haskell, surveyed some useful libraries and provided working examples. Although we might hope that a future version of Haskell will provide a way to handle UTF-8 conversions, in the meantime we need to choose an appropriate library for each project that handles Unicode text.

Updates

Fri Nov 16: Edited to incorporate some feedback from #haskell:

Thanks to Tim Newsham for clarifying GHC's default character encoding when printing Strings.
Thanks to Stefan N. O'Rear for pointing out that Show/Read is not the right place for serialization, but that it should instead occur on conversion to/from [Word8].

Release: libfishsound 0.8.1

Conrad Parker — Fri, 09 Nov 2007 07:37:00 +0000

libfishsound provides a simple programming interface for decoding and encoding audio data using Xiph.Org codecs (Vorbis and Speex). libfishsound 0.8.1 is a maintenance release, fixing a build error when configured with encoding disabled. Full documentation of the FishSound API, customization and installation, and complete examples of Ogg Vorbis and Speex decoding and encoding are provided.

Type-Level Instant Insanity

Conrad Parker — Mon, 10 Sep 2007 12:56:00 +0000

I wrote an introductory tutorial about type-level programming, "Type-Level Instant Insanity", which is now available in the Monad.Reader, Issue 8 [PDF]. The tutorial implements a solution to the puzzle Instant Insanity, entirely in the Haskell type-system.

Familiarity with the syntax of the Haskell Type System is a prerequisite for understanding the details of general Haskell programming. What better way to build familiarity with something than to hack it to bits?

The point of the exercise is to demonstrate how the type system can be used to tell the compiler about details of logic, such as pre- and post-conditions of functions. Although type annotations are optional in Haskell, they can often be used to more clearly describe what the shape of a correct solution is; or, conversely, to tell the compiler what a bug looks like.

There is also a talk page for this article, so feel free to add random thoughts there, or here.

The Monad.Reader is "a quarterly magazine about functional programming. It is less formal than a journal, but somehow more enduring than a wiki page or blog post."

Please enjoy :-)

Review: TagSoup

Conrad Parker — Sat, 09 Jun 2007 08:09:00 +0000

This week I've been playing with TagSoup, a Haskell library by Neil Mitchell and Henning Thielemann "for extracting information out of unstructured HTML code, sometimes known as tag-soup". This article introduces the basic usage of TagSoup, and discusses the functional approach to mining XML-like data.

Name: TagSoup
Version: darcs as at June 9, 2007; prior to release version 0.2
Functionality: Parsing possibly malformed HTML/XML
Inputs: String (also includes a URL -> IO String helper function)
License: BSD3

parseTags :: String -> [Tag]

The first thing you do with TagSoup is parse the document into a list of [Tag]. The Tag type is fairly general, and can represent the various things that can occur when reading an HTML document: an opening or closing tag, the text between tags (like <strong>this</strong>), comments, or special tags like <!DOCTYPE ...>. It is also used to mark the location of syntax errors, though of course these are not fatal as the whole point of the library is to robustly work around badly-formed input.

XML Parsing

TagSoup actually contains no HTML-specific code, other than that it knows about HTML entities. To demonstrate that it can be used for other kinds of possibly malformed XML, I added an example which extracts information from an RSS feed (now part of Example.hs):

-- rssCreators Example: prints names of story contributors on
-- sequence.complete.org. This content is RSS (not HTML), and the selected
-- tag uses a different XML namespace "dc:creator".

rssCreators :: IO [String]
rssCreators = do
    tags <- liftM parseTags $ openURL "http://sequence.complete.org/node/feed"
    return $ map names $ partitions (~== "dc:creator") tags
    where
      names xs = innerText $ xs !! 1

This function is of type IO [String]: it uses IO, and returns a list of Strings -- the names of contributors. The first line:

    tags <- liftM parseTags $ openURL "http://sequence.complete.org/node/feed"

uses openURL (part of TagSoup) to read the contents of the given RSS feed into a String, and then runs parseTags on that, calling the result tags.

Extracting information

Now that we can use the XML as a list of Tags, the second line:

    return $ map names $ partitions (~== "dc:creator") tags

splits it up into separate partitions, starting a new partition wherever there is a tag that roughly matches <... dc:creator ...>. It then runs the function names on each partition, and returns the result. names simply grabs the text inside the first thing in a partition, ie. the content of the <dc:creator> tag itself. Done:

*Example.Example> rssCreators 
["dons","dons","dons","jgoerzen","dons","dons","dons","dons","dons","dons"]

A more complex example, using an external HTTP library

Simon Peyton-Jones is a Free Software developer working on the GHC compiler at Microsoft Research in Cambridge, England. One of the examples given in Example.hs attempts to extract a list of his current research papers. Using TagSoup's convenient but simple HTTP library, it fails to terminate due to a hanging server. Here is a working version of that example using the new lazy ByteString version of the Haskell HTTP library:

-- compile with: ghc --make -o spj -Ldist/build -lHSHTTP1-3000.0.0 spj.hs

module Main where

import Text.HTML.TagSoup

import qualified Data.ByteString.Lazy.Char8 as BS
import Network.HTTP (rspBody)
import Network.HTTP.UserAgent as UA

spjPapers :: IO ()
spjPapers = do
        rsp <- UA.get "http://research.microsoft.com/~simonpj/"
        let tags = parseTags $ BS.unpack $ rspBody rsp
        let links = map f $ sections (~== "a") $
                    takeWhile (~/= TagOpen "a" [("name","haskell")]) $
                    drop 5 $ dropWhile (~/= TagOpen "a" [("name","current")]) tags
        putStr $ unlines links
    where
        f :: [Tag] -> String
        f = dequote . unwords . words . innerText . head . filter isTagText

        dequote ('\"':xs) | last xs == '\"' = init xs
        dequote x = x

main = spjPapers

This example is obviously a little more involved, but this is typical for a real-world example of scraping HTML -- the main content of the page is handwritten, and the enclosing content management system uses many non-standard elements and attributes.

The guts of that example is the links = ... declaration. It looks for the part of the page roughly between <a name="current"> and <a name="haskell">, breaks it up into sections (starting each section wherever there is a new <a ...> tag), and then runs some function f on each of those sections. To see this in more detail, we can read the declaration from the far end backwards: with the parsed list of tags :: [Tag], do the following:

drop everything until you get an opening tag matching <a ... name="current" ...>
drop the next 5 tags (no matter what they are)
take everything until you get an opening tag matching <a ... name="haskell" ... >
We now have a big list of tags (:: [Tag]): split it up into sections, starting each section wherever there is a new <a ...> tag. This will give us a list of lists of tags (:: [[Tag]]).
run f on each section's [Tag].

And what about this magical function f? It takes the first item of TagText (text between tags) in a section, runs unwords . words on it to clean up the whitespace (by splitting it up into words, then joining the words back up with a single space between each), and finally removes any surrounding quotes if present. You could use a function like f to clean up tag text in any page you are scraping. Here, the final result is a nice, clean, plaintext list of the titles of Simon Peyton-Jones' current research papers:

Constructor specialisation for Haskell programs
Faster laziness using dynamic pointer tagging
Scrap your type applications
A History of Haskell: being lazy with class
...

Functions for extracting information

TagSoup provides a few useful functions for pulling apart web pages. Taken together with Haskell's list processing functions, the result is a very expressive, concise language for extracting information from web data. In the above examples, we've seen how various String and list handling functions (like lines, words, unlines, unwords, takeWhile, drop, filter, defined in the Haskell Prelude), can be used together with predicates like isTagText from TagSoup. TagSoup also provides operators for inexact matching, ~== and its negative ~/=. These allow you to do a loose match on tag contents, so that your HTML-scraping application has some resilience to minor changes in page generation.

Lastly, functions like sections make use of the above predicates to divide the page up into similar parts. As we have seen in the above examples, this is useful for the common case where a page contains a list of items, and we want to extract the same kind of information from each of those items. We can write a function f to handle one item, then simply map f across all the contents of the page.

Comparisons

The similarly-named Java TagSoup is "a SAX-compliant parser written in Java that, instead of parsing well-formed or valid XML, parses HTML as it is found in the wild". This tries to make badly formed HTML usable through a conventional SAX interface. This makes it useful for more general applications than the Haskell TagSoup, but far less expressive for the task of scraping a known web page.

The Python Beatiful Soup (and similarly its Ruby counterpart Rubyful Soup) "can turn even invalid markup into a parse tree." This takes quite a similar approach to the Haskell TagSoup, and also provides functions to print out the modified parse tree. It attempts to create a full DOM parse tree (using HTML-specific heuristics), so extracting information can involve walking the tree with syntax like head.nextSibling.contents[0].nextSibling. Nevertheless it provides content-based search functions like soup.find('p', align="center"), and with a few lambda functions can be quite expressive.

Conclusion

This TagSoup does one thing and does it very well: it provides a small set of useful abstractions for extracting information from HTML pages and other XML data, As the markup you are scraping is badly formed, TagSoup provides operators for inexact matching and does not attempt to coax the page into a conventional tree structure. The result is a very fast, list-based representation of page content which can be mined using TagSoup's Tag-specific functions and Haskell's usual list operations.

Release: libfishsound 0.8.0

Conrad Parker — Sun, 03 Jun 2007 06:48:00 +0000

libfishsound provides a simple and consistent programming interface for decoding and encoding audio data using Xiph.Org codecs (Vorbis and Speex). This release includes compatibility with the floating point portion of the libfishsound development trunk API, in preparation for use with liboggplay. In order to build a minimal version of libfishsound for use with liboggplay, configure with encoding disabled in order to produce a smaller binary and to remove the dependency on libvorbisenc.

XTech 2007

Conrad Parker — Fri, 04 May 2007 13:28:00 +0000

I'll be giving a general talk about Annodex at XTech 2007 in Paris. The conference theme is "The Ubiquitous Web", and covers emerging standards for web technology.

My presentation slides and a short paper are available: Building video webs with Annodex.

Greek, Cyber-virgins, in LaTeX

Conrad Parker — Mon, 12 Feb 2007 12:35:00 +0000

For some reason, recently I've been reading a lot of math and writing a lot of LaTeX. I was getting lost in the symbols so I wrote up a chart of the Greek alphabet in LaTeX math mode, in pretty black, white and grey. The source is annotated with instructions for Web 2.0 programmers to replace the grey with cornflower blue.

While putting this together I came across the book TeX by Topic, a TeXnician's Reference, by Victor Eijkhout. The original book, published in 1992, has been out of print for a few years, but it is avaliable from the author's web site as a 289pp PDF. Chapter 1 covers the low-level input processor and expansion, and the subsequent 269 pages go below that, into the nasty implementation details of TeX:

"The four levels [of input processing] (corresponding roughly to the 'eyes', 'mouth', 'stomach' and 'bowels' respectively in Knuth's original terminology ..."

Thanks to the quick search capabilities of my PDF reader I was able to find the details I needed about macro expansion within the few minutes of sanity that this book affords the casual reader. If you are planning to start a neo-tribalistic techno-cult, I highly recommend this book as a means of torturing hapless cyber-virgins during their last few hours before sacrifice.

And keep it real: Respect Knuth's original terminology.

Atmel AVR ISP mkII, avrdude, Ubuntu

Conrad Parker — Tue, 06 Feb 2007 15:03:00 +0000

Inspired by Vik Olliver and Geoffrey Bennett at LCA 2007, and by seeing a class full of 9 year old Japanese kids building and programming robots (part of a field trip at C5 2007), I grabbed an AVR microcontroller and started playing.

This post just contains some gotchas for what seem to be common problems.

My aim tonight was just to build and program a simple test circuit, as outlined in Guido Socher's article Programming the AVR microcontroller with GCC. As I have plenty of USB ports but no parellel port, I bought a USB based in-system programmer, the AVR ISP mkII. My laptop is running Ubuntu 6.10 (Edgy), but most of the software required comes straight out of universe, ie. basically pulled directly from Debian.

To use the mkII rather than a parallel port programmer, refer to the pin-out for the AVR ISP header. Add a separate Vcc line to the programmer, the rest of the circuit is unchanged.
The cross-compiler toolchain is packaged in Debian, as is avrdude (to upload and download the code, dude):
```
apt-get install binutils-avr gcc-avr avr-libc avrdude
```
There are some buildable examples in /usr/share/doc/avr-libc/examples.
Guido Socher provides an updated tarball for the demo code in his tutorial. It contains a pre-built image and a useful Makefile.
To add an entry for the AVR ISP mkII to udev, add the following to /etc/udev/rules.d/40-permissions.rules:
```
#AVRISP mkII
SUBSYSTEM=="usb_device",SYSFS{idVendor}=="03eb",SYSFS{idProduct}=="2104",MODE="0666"
```
(from Yuki's Linux Memo).
Apparently the factory setting for the clock on these chips is fairly random, so you need to set that explicitly before you try doing anything else like uploading code. If you don't do that, then you'll get a bunch of nasty timeout errors like avrdude: stk500v2_paged_write: write command failed with 128. So set the clock by dropping into the avr terminal:
```
$ avrdude -c avrispmkII -P usb -p m8 -F -u -t
avrdude> sck 10 
>>> sck  10
Using p = 10.37 us for SCK (param = 7)
avrdude> quit
>>> quit 

avrdude done.  Thank you.
```

Introductory Haskell Programming in the UNIX Environment

Conrad Parker — Sun, 17 Dec 2006 14:42:00 +0000

A few months back I was chatting to Don Stewart about scripting in Haskell, and he pointed me towards some Haskell shell scripts he's written.

This weekend, Don wrote some introductory tutorials. Part 1 introduces Haskell in a similar style to how the Camel book introduces Perl -- quite readable, and fairly low on mathematical jargon. Part 2 introduces character and file IO, which I'll dig into below.

Why bother?

It turns out that you can re-implement the core of many simple UNIX tools as one-liners in Haskell. This is interesting because, like C, Haskell compiles to a binary and runs like a real program. Its also interesting because, unlike C, Haskell provides lots of error checking, as well as guarantees against segfaults and memory leaks, for free.

Lazy evaluation

Consider the following implementation of cp (from Part 2), which copies its standard input to standard output:

import System.Environment

main = do
  [infile, outfile] <- getArgs
  s <- readFile infile 
  writeFile outfile s

Although this is pretty simple to understand, it looks like it reads the entire contents of the input file into the variable s, and then writes that to the output file. That would be a huge memory hog, so let's take a look at what's actually going on.

Haskell compiles to a binary, so we can strace the resulting program:

$ strace -o /tmp/cp.out ./cp bigfile.ogg /tmp/bigfile-copy.ogg
$ less /tmp/cp.out
...
read(3, "\300\23n\261\205\v\fD$\r\330,\260\2172Zp\241h\306<\216"..., 8192) = 8192
write(4, "\300\23n\261\205\v\fD$\r\330,\260\2172Zp\241h\306<\216"..., 8192) = 8192 
read(3, "\2646\353t\304\300\f9|\36\10|O@r|\3149\3\340v{4\366|\17"..., 8192) = 8192
write(4, "\2646\353t\304\300\f9|\36\10|O@r|\3149\3\340v{4\366|\17"..., 8192) = 8192
...

We see that it has actually set up an 8K temporary buffer to funnel data back and forth, keeping the memory requirements very low. So the code was not a memory hog at all, even though its pretty simple to understand.

The way this works is that s is not a normal String variable at all. It is an IO String, the embodiment of everything String-like in IO. It lives in a very beautiful, transient and continually changing state of interaction where it might read some chars, write some, read some, write some, and so on until EOF. This is all that an IO String could want from its brief yet pristine existence, and nothing more.

Pass the pipe

Giving our instance of this IO String a name is conceptually similar to the use of named pipes in shell scripts. A direct translation of the above Haskell script into sh might be:

#!/bin/sh

infile=$1
outfile=$2

s="${TMPDIR-/tmp}/$$.fifo"
mkfifo $s

cat < $s > $outfile &
cat < $infile > $s

rm $s

Of course, this example is trivial; you'd only use named pipes for more complex tasks, such as setting up transcoding pipelines, where you might not know the names or parameters of the commands to be run up front. So, what if your shell script doesn't need to be so complex? What if you don't need to name your intermediate pipe?

cat $infile | cat > $outfile

Well, that's fine in Haskell too:

readFile f >>= writeFile g

No more naming our intermediate IO String. But now we know that it's still there, lurking inside that little >>=. This uses lazy evaluation, and we read in the Camel book that laziness is the first virtue of a programmer; Haskell gives it to you in spades.

Living in Kyoto

Conrad Parker — Sun, 03 Dec 2006 10:52:00 +0000

Autumn leaves are falling, and the days are becoming cooler. Awesome hacking weather.

It's now two months since I came to Japan. I spend half my time studying Japanese, half my time coding and doing research, and half my time partying. I've done a fair bit of sightseeing around Kyoto's temples, shrines and gardens, partied in Tokyo with Alex, Horms and Raster a couple of times, and learned my way around the local ramen joints. My favourite cocktail bar stocks over 20 different varieties of gin, including their own pepper-infused variety which tastes great with stingray (flambéd in your face, with spiritus). Student life is hell.

I've been car-free for two months now; it feels great to kick the gasoline habit. I ride a mama-chari, a steel-framed bicycle with a basket on the front. At first I thought of it as the cheap, chunky brick of a bike that it is. Then I realised that I have about the same strength to bike-weight ratio as I did when I was a kid on a BMX. So now I take every opportunity to bunny-hop random obstacles, get air off pavement and jump gutters. For some reason old people look at me funny when I do that.

As soon as I arrived and got my alien registration sorted, I sat down on a borrowed laptop to review the 350 awesome paper submissions for linux.conf.au 2007. The programme is packed full of more awesome than a noseful of wasabi. Go there or lose, your choice.

More recently I've been helping organise FOMS 2007, a developer workshop for free and open source media software, the week before LCA. It too will rock, and give us a great opportunity to plan some real integration between projects, like video editors + annodex + wikipedia. Rock on!

Recent hacking:

HOgg: a new commandline tool and Haskell library for manipulating Ogg files.
Blender Scripts: Getting Blender to model, animate and render non-interactively.
Croquet: An open source, fully hackable 3D metaverse. I'm just learning my way around -- come play in #croquet on irc.freenode.net!

11:35:59	* Hixie casually removes Ogg from the spec and sees what happens
11:36:43	* othermaciej_ takes shelter
	...

12:05:02	<kfish> Hixie!
12:11:47	* kfish throws a tantrum on behalf of the free software community
	...

12:29:48	* kfish reads the replacement text and revokes the tantrum
12:30:15	<kfish> Hixie, actually you didn't casually remove Ogg, you made the case for Ogg stronger, so thankyou :-)
12:35:37	<Dashiva> "Lift the cat who was amongst the pigeons up and put him back on his pedestal for now."
12:35:40	<Dashiva> Poetic
12:37:49	<Hixie> kfish: :-)