Apache Mesos is one of a select group of open source projects shaping the future of enterprise cloud infrastructure and distributed computing.
The rise of container technology in the past few years has created a unique opportunity for the project, which has been quickly growing its code base and contributions. As companies increasingly turn to Linux container technology to package and run their web applications, they are looking for software to help distribute and orchestrate those containers across data centers, says Dave Lester, open source advocate and software engineer at Twitter.
“We’re already seeing more companies looking at Mesos this year,” Lester said. “Many of the folks vetting it are looking to understand how this technology will play into their container strategy.”
Mesos: The Data Center OS
Originally a UC Berkeley research project, Mesos was adopted early-on by Twitter, and became a Top-Level Apache Software Foundation project in 2013. It’s now used by other web giants including AirBnB, Atlassian, eBay, Hubspot, Netflix, and PayPal.
Mesos is, essentially, an abstraction layer in the cloud stack between the hardware in your data center – whether it’s three machines, or 3,000 – and applications above it. As a cluster management tool, Mesos knows what hardware resources are available at any given point, including CPU and memory, and provides that information to application frameworks to schedule tasks, and in the process, minimizes operational costs by automating this coordination.
“By having the layer Mesos provides, you can bring together your compute and batch workloads,” he said. “It creates one system that you can manage both on.”
But Mesos is also evolving beyond a tool into something more akin to an operating system kernel. When combined with the set of application frameworks built on top of it, including Apache Aurora and Kubernetes, the Mesos ecosystem works like a data center operating system.
“The vision of Mesos frameworks is to make it as easy to write and run applications in the data center as it is for applications on your desktop or cell phone,” Lester said.
How to get involved
For open source developers who are interested in next-gen cloud technology, Mesos is a great project to get involved in. Contributors at all experience levels are invited to submit patches, attend a local Mesos user group meeting, and participate in the project’s second annual conference, MesosCon, to be held Aug. 20-21 in Seattle following LinuxCon, CloudOpen and ContainerCon. (Register for MesosCon by Feb. 14 for a $200 discount!)
Here, Lester, who is also the MesosCon program committee chair, talks more about how to get involved in the project, some upcoming features that are a good place for newcomers, and what to expect from the next MesosCon.
MesosCon is a great place to learn more about Mesos. What’s new at the conference this year?
In our second year of the conference things are going to be somewhat different. This year we’ll have longer talks and many more opportunities for folks to present what they’re doing. The community has grown significantly in the past year, so we’re expanding the conference to reflect that.
Last year was a single track, and fairly packed. This year we’ll break into multiple tracks to do deeper dives on frameworks people are writing, and practical talks on how engineering teams are operating Mesos in production. We’ll have the opportunity to feature use cases from companies, with the goal of providing a more holistic picture of what people are doing in the community.
The CFP deadline for MesosCon speakers is Feb. 14, who should speak and what topics are you looking for?
Talks will be presented by a combination of adopters and core developers, but also developers within the ecosystem. There are tons of topics that directly relate to Mesos, that aren’t Mesos proper, including monitoring, service discovery, load balancing, microservice frameworks, writing and running distributed systems, etc. (You can submit your talk proposal at the event website.)
We’re co-located again this year with CloudOpen so it’s a good opportunity for folks attending to get a broad view of what’s happening. The two conferences are nicely paired. We’re solving similar problems, just for a specific platform.
What’s the best way for contributors to get started with Mesos?
A good place to start is at a local level by joining or starting your own Mesos user group. These meet ups are held in major cities all around the world, and it’s a simple way to get more exposure to how people are using the project. We also have a fairly standard open source process that you see in other Apache Software Foundation projects. The mailing list is a source of truth and the best place to follow ongoing development, and our IRC channel is fairly active which is a good place to ask questions. We’re also active on Twitter: @ApacheMesos.
If you’re interested in contributing and don’t have a specific project in mind, we have tickets labeled “newbie.” Many are simple enough that if you get up and running with Mesos and want to tinker with C++ you could probably submit a patch.
Google Summer of Code or Outreachy (formerly the FOSS Outreach Program for Women) are opportunities to participate as well. People who are curious about the project can pop in and submit a patch or two and see how they like it.
(You can get more info on how to contribute in the Mesos developers guide.)
What exciting new features are upcoming for Mesos?
One of the biggest features added to Mesos this year will be what that community has referred to as persistent storage primitives.
Out of the box, Mesos is stateless. In a distributed system of many servers in a data center, and a scheduler that’s intelligently moving applications around, by default Mesos provides no guarantee that you will continue to run your app on the same machine. So if your app wants to save something to your local hard drive, when the application is rescheduled on a different machine you won’t have access to that data you saved.
It would be great to have storage supported directly. Storage primitives being added to Mesos will allow storage to operate as frameworks. So just like Aurora is a framework for services, you’ll be able to run HDFS or MySQL as a framework on Mesos. That will be a game changer for the project.
What other areas are new and interesting?
Another big feature is optimistic offers. Mesos currently has an intricate knowledge of all the hardware available at any point and it tells the frameworks which machines with various profiles are available and asks “would you like to run applications on it?” If they say yes, it will schedule a job. It currently does that one framework at a time. Optimistic offers will parallelize that communication between Mesos and frameworks, and make it easier to run different frameworks on the same Mesos cluster. This will bring Mesos closer in alignment to Google’s Omega project.
