New York Times opens up code

The New York Times likes open source — so much so that, as it gradually moves more of its print operations online, it is nurturing a Web development team that has released two of its own open source projects.

XSL Cache is a PHP extension the Times is using to cache stylesheets on its Web site. DBSlayer is a tool the team developed to overcome LAMP scaling limitations that caused database replication processes to overwhelm the DB connection limits.

New York Times senior software architects Jacob Harris and Derek Gottfrid say they’ve received a mixed reception from the community, because some people just can’t understand why a print media company would jump feet first into the open source philosophy. But open source software use isn’t new to the Times, says Gottfrid. “I’ve been here a number of years, and open source has always played an integral part in everything we do.”

Recently, the team has experienced growth, according to Gottfrid, in that custom applications developed in-house are “shifting from a proprietary posture. As we were building out and replacing old infrastructure, there were some gaps, so we wrote additional code. And some of those things we’re open-sourcing. It’s a small, humble effort.”

Gottfrid says that upper management has been supportive. “Open source has grown so much in terms of people being aware of it, and our managers have grown up in that environment. Five or 10 years ago we wouldn’t have had this reception.”

To support their community efforts, Harris and Gottfrid launched a blog at open.nytimes.com where they and other members of the development team share their thoughts about code, tools, the New York Times site, and sometimes even fun little widgets they’ve written. Harris, who join the Times only about a year ago, was a big influence on the launch of the open source projects. “We have a lot of internal data that we could potentially share,” he says, “and we have applications we’re interested in exposing.”

Harris says he’s received some pushback from members of the open source community. “We went to OSCON, and some people were very excited that we were there, and some were like, what the hell are you doing here. A lot of people feel like we’re pretty much irrelevant, and that they know what our business model should be. When you talk to geeks, they say ‘just stop printing the newspaper,’ but that would pretty much bankrupt us overnight.”

Gottfrid says they didn’t take it personally. “We’ve all been involved in the community for a long time. We want to figure out how to have a meaningful engagement with the community, to try to explain some of our problems so we’re not just thought of as a monolith. We want to have a discussion with the community we’re wading into. So we’re figuring out the right way to do outreach; we’re trying to find a sympathetic ear and engage some feedback.”

Gottfrid says the most frequent comment he hears about the New York Times’ foray into open source Web development is “‘That’s surprising.’ They general think of us just as a newspaper, so they don’t understand why we would be doing this. Is there a secret angle? But as soon as they read some posts, they start to break it down and understand. Now there’s a bunch of people interested in this.”

Gottfrid’s first release was DBSlayer. “It’s a lightweight database connection-pooling tool that we use internally. We were always saying, ‘We should open-source this thing.’ And that got a positive response from our manager — they said, ‘Go ahead, let’s see where you can take this.’ It hasn’t taken the world by storm, but it is somewhat of a niche thing. It’s a new program, so we haven’t developed it fully and it is still in the early stages. But people have submitted patches and we’ve gotten good comments. It’s been a very positive response. If we want to open-source other stuff in the future, it makes it a lot easier because we’ve established that we can.”

Harris says one of the things they’ve talked about doing in the future is “opening up some of our content in the form of widgets or APIs, and see what the general public would do with it.”

Gottfrid hints at the ultimate goal of NYT’s move toward open source. “The key goes back to the title of the blog: Open. We wanted to have an open dialogue because we value the conversation with the development community. We want to be a part of that, and have a place at the table and listen and give feedback and have that respect. As we open more and more, it gives us a built-in audience that we can seed these things with, and a number of people that we already have a good relationship with.”

Gottfrid says there are things you can do to make it easier to jump into the open source community — among them, “Be humble.” He says they don’t try to act like they already know where their place is. “We’ve never presented our solution as a definitive solution for all problems. It worked well for us, and hopefully someone else can make it work for them.” And don’t forget to be persistent. “Just because you release something today doesn’t mean everyone will show up the next day and download it and praise you and shower you with accolades.”

Harris and Gottfrid stress the importance of trusting your developers. “This [move toward open source] wasn’t a corporate mandate,” Gottfrid says. “There were no vice presidents involved in this at all. This was Jake and me monkeying about and saying, ‘Wouldn’t this be cool?’ The fact that they empowered us speaks volumes about the New York Times entrusting non-senior people to have a voice on the outside and represent the company. There’s a growing awareness that developers play an integral role in how the medium is evolving.”

Harris agrees. “We’re no longer just a print company, we’re a technology company. We need to express ourselves in technological terms. The best way to do that is to give the developers a voice.”