Sun’s OMS Video codec project is a means to an end

Sun Microsystems is setting out to create an open source, royalty-free video codec. Given the considerable head start of well-known, royalty-free video codecs like Dirac and Theora, you might ask why the world needs another. The answer, according to Sun, is the process the company will use to develop it — starting with a full-on, careful examination of the patent situation.

I spoke with Gerard Fernando and Rob Glidden about the project, which was unveiled in April. Fernando is a senior staff engineer at Sun, and Glidden is the company’s global alliance manager for TV and media.

Fernando says the video codec idea dates back to 2005, when the Open Media Commons (OMC) initiative was launched by Sun CEO Jonathan Schwartz. It just took a back seat to OMC’s first large-scale project, the DReaM digital rights management (DRM) system.

DRM rubs many in the open source and free software communities the wrong way, Glidden admits. But for DRM detractors, he says, the important thing about DReaM is the process that OMC used to develop it. Sun spent a full year conducting a thorough analysis of the patents and associated claims in DRM, and built DReaM on top of security systems that were either unpatented or whose patents had expired. You can read the fruits of Sun’s patent examination for yourself on the OMC Web site. Regardless of how you feel about DRM, in a field dominated by patent-holding commercial interests and standards bodies that promote member products to collect royalty checks, DReaM is both open source and — more importantly — composed entirely of patent-unencumbered, royalty-free technology.

Submarine patents, industry FUD

By systematically examining the patent landscape, Glidden says, OMC can document the intellectual property rights asserted by other players, including valid patents to be worked around, and bad patents that need to be challenged. That strikes at the heart of one of the chief criticisms of codecs created through an open source development model, the so-called “submarine patent threat.” Glidden calls that threat FUD, a scare tactic used by those collecting royalties on their own codecs to discourage the development of others.

“It’s a call to inaction,” he says. “When you are surrounded by a fleet of battleships all taking aim at you, the possibility of a submarine is not really a problem.” The battleships in question are the patent-holding participants in royalty-charging standards groups like the ISO and the ITU. MPEG-LA exists solely for the purpose of selling licenses to MPEG codec patents, Glidden says, so of course it is going to spread the notion that competing technologies are fraught with risk and unspecified infringements. That is why OMC’s development process begins by proactively challenging the allegation.

The “how can you be sure you don’t infringe on someone else’s patent?” question has a straightforward answer, Glidden says. You research the field, taking a conservative approach, and build up a taxonomy of the relevant work. “The fact is that these standards have known patent pools. They are listed in the documents, on the Web sites, even in the press releases.”

H.261 baseline

OMC has already begun its patent evaluation for OMS Video, and has preliminary ideas sketched out for the structure of the codec. You may have heard it reported that OMS Video will be based on the H.261 codec, whose first wave of patents has already expired. Critics object that H.261 is irrelevant because of its age, arguing either that OMS Video will be shackled to archaic technologies, or that OMS Video will contain so little of H.261 that its expired patents make no difference.

That OMS Video will be based on H.261 is true only in the most basic sense; as Glidden explains it, H.261 is the starting point for the OMS codec because it is the grandparent of all discrete cosine transform (DCT)-based codecs — a family that includes the majority of the patent-encumbered codecs popular today. But by citing H.261 as the basis for the new codec, OMC makes a statement about its lineage: demonstrably unencumbered by patents.

Glidden and Fernando both insist that “based on H.261” is oversimplifying too much anyway. Fernando explores the structure of the codec in a PDF presentation that you can download for free. As he puts it, what we often refer to as the “codec” is actually better understood as a suite of tools performing different functions — spatial prediction, motion region coding, arithmetic coding, and so on. A patent holder might make a claim on one or more of those specific functions, not on the codec as a whole.

That distinction is important both for tracking the patent issues and for understanding the amount of new work that remains for the OMS Video project. If you search the ITU’s Web site, you will find seven specific patents listed as parts of H.261. The block diagram of the OMS Video codec in Fernando’s presentation (which is also used in an OMC blog post about the project) references just two components from the original H.261: arithmetic coding and 1/4 pixel motion estimation.

Taken as a whole, the few expired patents implemented by H.261 will not constitute much of the final OMS Video. But the H.261 heritage will make it harder for opponents to argue that the new codec infringes on more recent codecs, since those recent codecs are derived from the demonstrably historic H.261 as well.

Free perspective

Christopher Montgomery of The Xiph Foundation is well-versed in codec creation, and no stranger to patent infringement FUD either. He agrees with Sun’s basic claim that the vast majority of modern video codecs share the same mathematical lineage, but is not persuaded by its arguments regarding the value of basing new work on H.261. “If you’re limited to an executive summary flowchart of no more than 10 boxes that fit on one page, you’d be hard-pressed to tell the differences between any of these codecs. The differences are all in the details (of course, the details are very important).

“The claim that building on top of H.261 frees you from patent concerns and that’s a reason to use H.261 over Theora is dubious, as Theora is based on a codec of approximately the same age. The truth is that as soon as you start trying to move forward (using any of the discoveries published since then) you immediately run into patent concerns again.”

Montgomery also referred me to a blog post he wrote in response to the HTML 5 Video element debate. There he recounts his personal experience with years of patent FUD aimed at the Xiph codecs, Vorbis and Theora, noting “part of the problem is that there is no legal test for ‘unencumbered.’ Nor do you have to be infringing to get sued. We will always have a halting problem here. And yet, both Vorbis and Theora are 10 years old with no claims against them. Would another 10 be sufficient? Twenty? Vague threats that come to nothing (nothing that is except eliminating the monopoly interest’s competition) are the story of my professional life. Patent law is being used to extort outright, and most of the interests represented here are willingly buying into it. This isn’t even getting to the trolls. And we sit around and wonder why this industry is so dysfunctional.”

Standards and beliefs

Glidden also points to the W3C’s withdrawn support for Vorbis and Theora in HTML 5 as evidence that the standards bodies — like the ITU, ISO, and MPEG-LA — are falling victim to FUD from vendors of competing codecs. When participants in the process stand to profit through patent royalties, he concludes, the process cannot be trusted to make objective technical decisions. It is lamentable to see in the W3C, he says, because it had long enjoyed a reputation for fair and pragmatic decision-making. By caving to FUD over codec patents, it is acting just like everyone else.

Sun’s work on OMS Video is an attempt to challenge the patent-crazed belief system that dominates the codec industry, Glidden says. “We don’t accept the stock answers on FUD, such as ‘How can you ever be sure you aren’t violating a patent?’ You can be sure by doing a taxonomy; you dig into the process and ask questions.”

So what impact will Sun’s work have on the prevailing attitudes towards software patents? Only time will tell. Glidden and Fernando could not predict when the OMS Video patent audit would produce its first fruits or when a general plan would be available for the project.

I see evidence of the FUD campaign that Glidden describes whenever I read the discussion threads attached to news story about codecs: Fear that developers (or even users) will get sued for patent infringement. Uncertainty about what patents exist and whether it is even possible to know. Doubt that new ideas or patent-unencumbered ideas could ever match the patented, for-profit products.

Regardless of whether OMS Video bests H.264 technically or ever makes a dent in the MPEG-LA’s market share, this initiative from Sun is important to watch because the development process is a counterattack against the it’s-not-worth-trying mindset. That mindset is costing consumers millions of dollars a year in licensing fees, and it has a chilling effect on new work.