Improved Ogg Theora coming soon to an Internet near you

Free software for video is currently in sad shape. The only two widely distributed free video editing programs, Kino and Cinelerra, are nowhere near as capable as competing commercial software. Hardly any professional video producers use either one. But most people aren’t as concerned with the software used to make video as with the software they need to play it, especially in their Web browsers, and on this side of the video equation it looks like things are going to get a lot better for free software enthusiasts in 2008.

The two most prominent video codecs currently available under free software licenses are Xvid and Ogg Theora. In a technical sense, Xvid is clearly the better of the two. Any videomaker who renders a work in Xvid, then renders that same work in Theora with settings that produce a similar file size for the resulting video, will instantly prefer Xvid not only for output clarity but also for rendering speed. Xvid is not just an excellent free software video codec, but is an excellent video codec, period. It is often used in commercial video production, and many proprietary video editing programs render directly into Xvid format, while I have found none that directly render Ogg Theora files.

The problem with Xvid is that it works within the MPEG-4 framework, which is so patent-encumbered that today no one really knows who has “rights” to it. Indeed, right now, no new MPEG-4 licenses are even being issued. If you are using typical free GNU/Linux video players, and you play Xvid videos in a country whose government believes in software patents, you may be violating patent law every time you use Xvid.

But then, according to Microsoft CEO Steve Ballmer, you may be violating 200+ Microsoft patents just by running GNU/Linux. And not everyone believes Xvid violates anyone’s patents. In an email, Michael Militzer of Xvid.org said:

Xvid is an implementation of the MPEG-4 standard, which is an international standard under ISO/IEC 14496-2. We support open standards and have chosen to follow the MPEG-4 standard to ensure interopability also with other, third-party applications and hardware devices.

It is well known that some parties believe they hold patents that are essential to the MPEG-4 standard. The interests of these patent holders are bundled by the MPEG Licensing Association (MPEG-LA). The MPEG-LA offers patent portfolio licenses for different MPEG standards, including MPEG-4.

We never have experienced any problems with MPEG-LA. And from the patents we’ve studied we have yet to see one that’s not either trivial or where not prior art can be found (or which actually has been granted unlawfully). So whether these patents really cover the MPEG-4 standard [and] are valid and enforcable may depend on your jurisdiction. Here in Germany, algorithms, processes or computer-implemented inventions are not patentable. Nonetheless of course, these patents exist. We however do not consider the situation for Xvid to be much different than for other applications or other open source projects. Also, other application fields are heavily patented (like image or word processing), so I suppose you could find some potentially dangerous patents for many projects if you perform careful research. The existence of the MPEG-LA as a royalty-collecting body for the MPEG standards just highlights these patent issues, while they exist (unnoticed) for many other applications just the same.

The aim of Theora is to circumvent such software patents (to avoid the MPEG-LA patents) by creating [another] bitstream format. This is however very difficult, if not impossible, because of the sheer mass and triviality of the existing software patents.

Theora claims to be not encumbered by patent issues but IMHO that is just a claim and nothing more (because of the above). Theora is obviously not applicable to the MPEG-LA patent portfolio license because it’s not compatible to a MPEG standard (it stems from On2’s proprietary VP3). However, the employed underlying compression methods are essentially very similar to MPEG compression: input images are converted into YUV colorspace, the image is partitioned into smaller blocks for compression, there’s MPEG-like intra and inter prediction and a block-based transform is applied for energy compaction. All these basic concepts are already patented.

In 2002 ON2 open-sourced VP3 and handed its maintenance and future development to Xiph.org, which was already the steward of the non-proprietary Ogg Vorbis audio format.

Ogg Vorbis has developed a user base that, while nowhere as large as the one for the patent-encumbered MP3 audio format, is certainly respectable. Virtually all Linux audio players handle Vorbis, and so do many that run on Windows and Mac OS. I personally own several royalty-free sound effect and music libraries that came to me in Vorbis format, and I have no trouble using them with commercial video editing software.

But when we move from talking about Vorbis to talking about Theora, we come to a screeching halt. Outside of Linux-land, hardly anyone uses it, and Theora encoding is in a sad state, especially for those computer users (estimated at 95%+) who prefer a GUI to command-line work. Not only that, the DirectShow filters for Theora don’t work correctly, so Windows-based video editing programs can’t render directly to Theora by using simple plugins the way they render to Xvid or QuickTime once you install the correct codecs for those formats.

Ogg honcho Christopher “Monty” Montgomery says he’s well aware of Theora’s encoder problems, and that he and others are working to solve them. He has pointed a lot of people — including me — to this screed about what Theora needs in order to be a competitive method of delivering video over the Internet. Even if all the faults mentioned on that page are corrected, there will still be those who say H.264 is better than Theora from a technical standpoint, and they’ll probably be right. But, beyond a certain basic point (clear colors, lack of blocky “digital artifacts,” well-synchronized sound, reasonable file size:clarity ratio), quality may not matter to anyone except videophiles.

Face it: millions of people are satisfied listening to MP3 music and watching YouTube (FLV) videos, neither of which give anything like the best reproduction of their original source material. I suspect that, if Theora becomes “good enough” in the same way MP3 and FLV videos are “good enough,” and tools for converting mainstream video formats to Theora become as easy to use as an MP3 encoder or — better yet — as easy to use as YouTube’s upload utility, we will see a dramatic increase in Theora’s popularity.

Monty says the Theora converter is “obsolete by 10 years,” that he’s now writing a “real encoder” for Theora, and that it will “be another couple of months until its mainline release.” He also says a lot of the problems with the original Theora code and encoder aren’t because they’re buggy in the usual sense, but because the original codebase was “written by people who were self-taught,” and that their lack of experience shows.

On the Windows and DirectShow front, Monty says Xiph lost one of its most talented Windows coders — he went to work for Microsoft — and that one of his dreams is to be able to hire volunteer Xiph coder Timothy Terriberry to work on the project full-time. (Monty himself works for Red Hat on free media software.)

Future prospects

Meanwhile, there’s the FSF-endorsed Gnash project, which will play some online FLV videos, but won’t currently work with the most popular Flash video sites. Gnash is codec-agnostic. It will play videos encoded in H.264, Xvid, Theora or (supposedly) almost any other video codec you happen to have on your computer.

Another hope for the future is Fluendo’s Cortado Java applet, which streams Theora video to any computer that has Java installed. It could presumably be used with other codecs and media frameworks just as easily, thereby eliminating the problem of users needing this or that codec on their computers to watch videos from different sources. Now that Java is Free Software, no one can really object to having it on their computer on licensing grounds (except possibly a few Microsoft executives), and it is available for a wider variety of platforms than Flash, too.

Meanwhile, there are other 100% free and open source video codecs under development. They may or may not violate someone’s patents. As Xvid’s Michael Militzer says, “I think you can just conclude here that software patents are evil.” And so they are, but for those of us in less-advanced countries, including the USA, they are a fact of life at this point in time.

On the video editing side, Christian Einfeldt of Digital Tipping Point believes Cinelerra may become much more useful in the near future than it is now. I, myself, am about to start experimenting with Kino for all of my simpler video editing projects. And these aren’t the only two Linux/FOSS video editing projects out there, just the best-known and most mature ones. There are many others under development, and some of them show a lot of promise.

A bright FOSS video future, but don’t expect miracles by tomorrow afternoon

Xiph.org’s people are working on a better Theora encoder, and are aware that the best hope for increased Theora spread is not yelling at people to get them to use their video delivery system, but in making it work better.

The people who scream, “How dare you use proprietary software to edit videos! Haven’t you ever heard of Cinelerra?” all have one thing in common: they’ve never tried to use Cinelerra to produce sophisticated video on a tight deadline. If they had, they’d know why dead-broke indy film kids would rather beg, borrow, or steal to get Final Cut Pro or one of its commercial competitors than use free-as-in-beer Cinelerra. But Cinelerra is improving, and it may be only another year or two before low-budget video and film producers start using it of their own free will, without anyone yelling at them.

In fact, the screaming is probably counterproductive. It makes Free Software advocates sound like Steve Ballmer complaining about Linux. It’s better to take the Firefox tack: improve your software’s usability and reliability, and market the positive things about it in a positive way.