Mozilla launches video accessibility drive

Video and audio support will soon be built directly into Firefox, by way of the free Vorbis and Theora codecs, and Mozilla is using the opportunity to advance multimedia accessibility for hearing-impaired and seeing-impaired users. Although HTML 5 does not officially include Ogg Vorbis and Theora as baseline codecs for the new VIDEO and AUDIO tags, Mozilla has adopted them for its own implementation. Researcher Silvia Pfeiffer is leading a Mozilla Foundation-funded effort to integrate support for closed captioning and other multimedia accessibility features into the Ogg formats and their implementation in Firefox.

Pfeiffer has worked on multimedia accessibility for more than 10 years, and is a long-standing member of the Xiph.org community. She set up a public mailing list about the video accessibility work, that along with a public wiki is designed to garner wider community involvement. Such involvement is critical, because the decisions made will need to prove acceptable to several groups: Mozilla, Xiph.org, the World Wide Web Consortium (W3C), Web Hypertext Application Technology Working Group (WHATWG), and Opera Software, which will also deploy Ogg support for VIDEO and AUDIO in its browser.

Why accessibility matters to everyone

Closed captioning tracks for the hearing impaired are the most widely discussed accessibility feature, but the field encompasses far more. Audio annotations and sign language tracks reach additional disabled users, while transcription, alternate languages, and embedded metadata tracks can be useful for all users.

The value of multimedia accessibility for users with disabilities is clear, but the issues faced regularly by deaf and blind users confront all users at different times and under different circumstances. In loud surroundings, users may need closed captioning to understand a video. Mobile device users may not be able to maintain eye contact with the screen, in which case audio annotations become necessary. And, perhaps most importantly, making a text transcription of multimedia files available opens the possibility of indexing and searching video and audio alongside HTML.

Similarly, a scheme for moving to a specific time code inside a video is useful to blind users who cannot effectively drag a slider back and forth in a timeline widget, but it also makes it possible to hyperlink to any point in the video, rather than just the beginning. The W3C’s Media Fragments working group is developing just such a time code addressing scheme.

Timed text for Ogg

The first step in Mozilla’s accessibility work, according to Pfeiffer, is deciding on a captioning format suitable for Ogg containers. There are plenty of alternatives, including Continuous Media Markup Language (CMML), TimedText, and VOBSub. They differ in complexity and technical detail. Some, like CMML, were designed for embedding into a multitrack media format like Ogg. Others originated in different domains, such as SubRup, which was designed for DVD subtitles, or were designed for different purposes, such as Kate, which was designed for karaoke.

The common denominator is that each format makes up a “timed text” track, encapsulating text to be multiplexed with existing video and audio tracks inside a container file, ensuring that the text arrives in time to be displayed in sync with the video and audio data. Support for multiple text tracks, multiple languages, and the occasional graphic (e.g., the note icon denoting playing music) complicate the choice.

Pfieffer is in the midst of cataloging the state of the art among the various captioning formats in preparation for her recommendation for Ogg. She says, “I do not expect any problems between Xiph, Opera, and Mozilla, which is great because at this stage I want to focus on the technical issues and make the best possible technical choice before being confronted with politics.”

The sailing might not be as smooth when it comes to the W3C, Pfieffer says, pointing to the difficulty that arose over the inclusion of Vorbis and Theora in HTML 5. But she remains optimistic. “If we’ve done our work well and the choices we have made are obvious for technical reasons, I think the political debate may turn out to be small eventually.”

Implementation in Firefox

But the team’s work will be far from complete when the captioning format is chosen; crafting the user interface within Firefox is as hard or harder. Although Ogg containers are the only type set to be displayed directly via the VIDEO and AUDIO tags, the interface exposed to activate and manage timed text and other accessibility features will need to integrate with accessibility frameworks at the Web page and operating system level, including specialized components like screen readers and Braille displays. Ideally the solution will be the same on all platforms, and will extend to other embedded media types, including those depending an external back end like GStreamer and QuickTime.

The draft plan on the wiki is to select the most important subset of desirable features first, saving the rest for further iterations. It also includes building a suite of open-licensed documentation and test cases for use by developers of authoring software and content creators; support for captioning and audio description in the playback engine is of little value if the authoring tools and content hosting sites do not buy in to the solution or find it too difficult to implement.

Pfeiffer describes Web media accessibility today as the exception and not the rule. “The possibilities are there for Flash video. They are not widely implemented and are all implemented in a different way, so they are incompatible.” She cites YouTube’s recent addition of captioning support, experimental players like Easy YouTube, and independent projects like Jeroen Wijering’s FLV Media Player.

Web video itself is still a young phenomenon, she continues, whose full impact on the Web we are only now starting to understand. “We are now exploring means of making Web video accessible, which is a precursor for the development of standards. Within the next few years we will see standard solutions on this space also being a standard part of online services.”

Whether Mozilla’s implementation of captioning and other accessibilty features in Ogg will significantly change the landscape remains an open question. One of the biggest roadblocks, says Pfeiffer, is the latent resistance to Ogg formats themselves demonstrated by the proprietary browsers. “One browser implementing Ogg won’t make the other browser vendors take it up,” says Pfeiffer. “Ogg is already supported by two browsers: Opera (in an experimental branch) and Mozilla (in the next release). The others — in particular IE and Safari — will not easily move toward Ogg support.”

But she remains optimistic about its effect on accessibility. “I believe that when we decide on a particular technology to provide accessibility support in Ogg, it has a high chance of being taken up by all Ogg supporters, as long as the technology choice was an informed one and had input from everybody. That’s what I’m working toward, which is why the process is so open.”