China Mobile is one of the biggest telecom companies in the world, with more than 800 million users in China — all of whom are served with open source technologies. During the 2016 Mobile World Congress, China Mobile declared that the operational support system running their massive network would be based on open source software. China Mobile is not alone; many major networking vendors are moving to open source technologies. For example, AT&T is building their future network on top of OpenStack, and they have invested in software-defined technology so significantly that they now call themselves a software company.
I sat down with Zhang Zhihong, Deputy General Manager of Cloud Computing Products, China Mobile Suzhou R&D Center to discuss how the company is embracing open source and Linux, and how they are giving back.
China Mobile is not the only player embracing Linux in this industry. Zhihong said that companies like Alibaba and Baidu also have internal groups whose job is to just build optimized Linux distributions for their own consumption. Not only does Linux cut costs heavily (you don’t have to pay millions of dollars to acquire subscriptions or licenses for thousands of machines), but the company can also fine tune it to get the most out of their network and infrastructure.
“We thought, when they can do it, why can’t we? We built an internal team at China Mobile and created our own Linux distribution,” said Zhihong.
China Mobile buys around 4000-5000 servers every year, and most of these servers deploy Linux. Previously, they used commercial versions of Linux — mainly SUSE Linux Enterprise Server and RHEL — but in 2015 their team created a custom version of Linux that gave them more control over their infrastructure while also cutting costs.
The new operating system is based on CentOS, and in 2016, China Mobile deployed more than 10,000 physical servers running this customized version of Linux in a production environment.
Cost and Control
When asked about the advantages of using their own custom Linux, Zhihong pointed out two deciding factors: cost and control. Cost has been the most important factor, Zhihong said. The purchasing department makes all decisions and controls all deals for the company, and they think it’s too expensive to pay for a commercial operating system as the cost can run into hundreds of millions of dollars. That’s a lot of money.
The second reason was better control over their infrastructure. “With custom Linux distribution, we can push our limits as we have a lot of low-level software. We use KVM for virtualization and Ceph for storage, with a lot of fine tuning and optimization at the kernel level. If there are bugs, or if we need a new feature it can take a lot of time to talk to the vendor and get those changes into the OS. By using our own distribution we gain this capability.”
However, running their own distribution doesn’t mean that they don’t contribute to the Linux community. Zhihong said that contribution is the core part of using Linux or another open source software. He said more than 100 contributors from the company contribute to the kernel. Whenever there is a bug, they fix it and submit the patch.
Zhihong gave an example of working with upstream when they hit a KVM bug in the public cloud production environment, they were running several virtual machines with several CPUs and when they attached more than two disks it would crash. Their kernel teams traced the problem, which had something to do with buffer overflow, fixed the bug, and submitted the patch upstream.
He also added that Linux is very strong in China, with many local Linux groups, and he said many employees from the company are part of those local Linux communities. “Linux is very much welcomed in China, there are lot of Linux programmers,” Zhihong said.
OpenStack Superuser
In addition to Linux, China Mobile is a heavy user of other open source technologies. “We use a lot of open source technologies: OpenStack, Hadoop, Zookeeper, Tomcat, Ceph, and so many that I can’t list them all,” said Zhihong.
China Mobile is more than just a mobile carrier; they offer many more services. They have many IT applications so they have been running their own private cloud for many years. Their private cloud is spread across three pools in three regions of China. There are thousands of servers running in these pools, but the cloud is proprietary and not open source.
By 2015, OpenStack had stabilized and matured enough to be considered seriously by the likes of China Mobile. So, China Mobile began building a new OpenStack private cloud spanning across two pools, with each pool running more than 3,000 servers. Once the project is complete, they will connect it to the existing proprietary cloud and little by little replace it with OpenStack.
Their commitment to open source and OpenStack led them to win the OpenStack Superuser award last year. OpenStack is mostly seen as a private cloud answer to AWS and Microsoft Azure, but China Mobile uses OpenStack both in its private and public clouds. Their public cloud has more than 3000 servers. It’s similar to AWS where it provides virtual machines, object based storage, and other such services to customers. It has more than 20,000 registered users and around 2,000 enterprise users.
China is a huge market for companies like China Mobile with more than a billion potential customers, and it’s also the manufacturing hub of the world. As more and more big companies embrace Linux and open source, China may evolve from a consumer of Linux to one of its leading contributors.
The program for Open Networking Summit is now available!
Look forward to over 75 sessions led by networking visionaries including Martin Casado, General Partner, Andreessen Horowitz; Amin Vahdat, Google Fellow and technical lead for networking, Google; Justin Dustzadeh, VP, Head of Global Infrastructure Network Services; Dr. Hossein Eslambolchi, Technical advisor to Facebook, Chairman & CEO, 2020 Venture Partners; and many more. Register now >>
