The bigger you are, the more small efficiencies add up. Manivannan Selvaraj’s talk from LinuxCon North America gives us a detailed inside view of how PayPal cut operating costs by a factor of ten, while greatly increasing performance and user convenience.
Everything has to be fast now. We can’t have downtimes. No going offline for maintenance, no requesting resources with a days-long ticketing process. Once upon a time virtual machines were the new miracle technology that enabled more efficient resource use. But that was then. Selvaraj describes how PayPal’s VMs were operating at low efficiency. They started with a single giant customized Jenkins instance running over 40,000 jobs. It was a single point of failure, not scalable, and inflexible.
The next iteration was individual VMs running Jenkins for each application, which was great for users, but still not an optimal use of hardware. Selvaraj notes that, “Only 10% were really used. The rest of the time, the resources were idle and if you think about 2,500 virtual machines, it’s millions of dollars invested in hardware. So, although it solved the problem of freedom for users and removed the single point of failure, we still had the resource management issue where we didn’t use the resource optimally.”
Docker is the key
The solution was a continuous integration (CI) system built on Git, Docker, Mesos, Jenkins, Aurora, and the Travis CI API. Docker is the key to making it all work the way they want. Selvaraj explains how Docker provides five key benefits: task isolation, eliminates host dependency, reproducibility, portability, and cloud native.
Selvaraj says, “Once we decided that Docker is the way to go, we started dockerizing most of our applications. We have dockerized CI API, which is our orchestration engine, which takes in the CI provisioning request, creates the CI to the user. We have dockerized the Jenkins master. We have dockerized Jenkins slaves. So, everything is running in Jenkins, in Docker, so that we don’t really rely anything on the host and it’s very easy from our maintenance perspective.”
Selvaraj shares a wealth of great insights on PayPal’s CI infrastructure in the conference video (below) and gives a live demonstration.
