Spotify can aptly be called the “Netflix for music.” The company started off in 2008 and by 2014 it boasts more than 40 million users with 10 million paid subscribers. The Swedish company is yet another example of how Linux and open source enable businesses to serve millions of customers using state-of-the-art, shared technologies.
How much Linux & Open Source does Spotify use?
The simple answer is ‘a lot’. There are two components of the Spotify experience: the client which is used by customers to access the service, and the back-end infrastructure which powers the service.
The client is proprietary or closed source, but it uses a lot of open source technologies. Noa Resare, free software ombudsman at Spotify, told me, “In the client we use Chromium Embedded Framework, which is an open source project related to Google’s Chrome browser, to provide a JavaScript execution environment and HTML renderer so the interface is built using HTML5.”
“If you look at the software stack from the client then of course the client is proprietary, the source is not available but looking at the various building blocks that together comprise the back-end services, there are lots and lots of open source components,” Resare said.
Linux is everywhere at Spotify
As far as Linux is concerned, he said, “Linux is used pretty much everywhere, for the machines that serve music files, to machines that respond to various API calls from the clients, to the machines that hold the databases storing all kinds of information.”
Linux is a natural choice for companies like Spotify that deal with huge amounts of data, where faster access to data is also critical. When asked about the factors that lead companies like Spotify to choose Linux, Resare said, “cost, and the ability to look inside the software and figure out how it works.”
Spotify has more than 5,000 physical (not virtual) servers and four different data centers running Debian GNU/Linux. The songs are encoded using yet another open source project, Ogg Vorbis, which is a free software audio codec.
Commercial support and timed release is important
Earlier this year the company made public its switch from Debian GNU/Linux to Ubuntu, which is now underway. There are many reasons why a company like Spotify would prefer Ubuntu over Debian, and the reasons are purely technical. Debian doesn’t have a time-based release cycle; they release it when it’s ready. And it’s ready when it is ready. It is not very predictable which makes it harder, or almost impossible, for companies with huge infrastructure to plan.
Resare says, “We have so many machines and so many services running those machines. Migrating to the next supported version once an older version is declared obsolete by the Debian project gives us very little time.”
Debian supports the older version for only a year once the next stable version is released so if Debian Y was released in May 2014 the support for it will end in May 2015. That changes if the next version is released within a year as the project doesn’t have resources to support three versions.
Ubuntu, on the other hand, has time-based releases. They have one release every six months and the dates are well known. Then they have LTS or long-term support releases every two years and these LTS releases are supported for 5 years. This model allows companies to plan ahead and also have more than enough time to upgrade machines.
“You know that a new release will come at this point of time and and you can plan around that, was kind of the most important thing,” Resare said.
Another reason behind this switch is commercial support. Companies like Spotify can easily sign a contract with Canonical to get support for their infrastructure, which is not the case with Debian as the project doesn’t offer any commercial support.
Resare said, “As we grow larger it’s kind of convenient to have a company that can be paid to help out with various things.” He clarified that they have not decided to buy any contract with Canonical at the moment but, “it is nice to have Canonical there to be able to interface with someone if we would like to get help in some specific parts.”
Other Open Source components at Spotify
In addition to Linux, Spotify uses a lot of other open source components; they are heavy users so an obvious question was, “How much do they contribute back?”
Resare said something really important and something I discussed with Linus during LinuxCon in Prague – contributing back must not be the condition to using open source. “Of course we use a lot of free software, using itself is good. You should not be ashamed of using things even if you are not contributing actively back that much.”
The fact is, you don’t need to give the patches back if you have not made any changes or if you have not released your software. That said, Spotify is involved with quite a lot of projects.
“We are a big user of Apache Cassandra, for example, the distributed database management system, and we contributed upstream to Apache Cassandra,” Resare said.
Spotify is also actively contributing to the Chromium Embedded Framework. One great way to contribute to open source is to fund the people developing the code (in the end it’s all about people and not the code). Spotify employs the project leader of CEF, Marshall Greenblatt.
“We are also contributing a fair bit to Puppet configuration management utility, mostly bug fixes and other smaller things but we are fairly involved in the development there. Those are the main areas where we have done substantial contribution to upstream,” he said. “Other than that our open source and free software engagement is mostly about us releasing internal software as open source. That’s something we are getting better at.”
What Spotify is doing with Linux is not unique to the service they offer. A lot of what they do is IT infrastructure which could be similar to what many other organizations build.
“What most people don’t realize is that 90 percent of what we do is build systems to manage our service, to make sure that we can create and maintain thousands of servers. A huge amount of stuff that we build is infrastructure things that has nothing to do with specifics of doing streaming music but instead is more like building generalized frameworks for doing efficient back-end request handling and stuff like that,” Resare said. “In that area we are contributing to the open source community by releasing some of our software as free software.”
Why no Linux client?
The irony here is that while Spotify runs entirely on Linux it is not an officially supported platform. Resare admits there are some unofficial clients and since 60–70 percent of the backend engineers at Spotify run Linux on their systems they have their own client which is supported by those engineers. He also pointed at a harsh reality: Since Spotify is a subscription based service, they have a crystal clear database of where their users are coming from and the Linux user-base is way too small to make any business sense to officially support a Linux client.
But unlike other services, Spotify has made a smart choice which doesn’t alienate Linux users. “In the case of Linux you can either use the unsupported Linux desktop version or you can use the web client. It works, but the integration is a bit limited,” Resare said. “I really enjoy having my media keys working for example, and that is functionality that is not provided by the web client, but music playback runs fine.”
