Linux Foundation System Administrator Konstanin Ryabitsev works on the team that runs the systems behind Linux kernel development on kernel.org. As part of the Collaborative Projects team he’s also responsible for providing IT hosting for projects including Yocto Project; Code Aurora Forum; OpenDaylight; AllSeen Alliance; OPNFV; and Iotivity.
This week Ryabitsev answered questions in a Reddit AMA that covered topics ranging from his favorite distro and tools to the inner workings of kernel.org infrastructure. Some of his more interesting answers included showing the trust paths from his PGP key to Linus’s, explaining his love of a fermented Russian drink called Kvass, and confessing that not all of the systems he oversees run Linux.
Below is a sampling of some of the responses. Visit the r/linux subreddit for the full AMA. For previous Linux.com articles featuring Konstantin, see On the Job with a Linux Foundation Systems Administrator and Linux Foundation SysAdmin Konstantin Ryabitsev, an SELinux Expert.
On kernel.org infrastructure
Which distro do you guys run on production? Do you prefer a mono-culture or do you see different distros doing different jobs better? (See the full thread by minimim)
We’re an RHEL shop for a number of both historical and pragmatical reasons. The only thing we have that’s not RHEL is the Raspberry Pi that’s doing auto-signing for sha256sums. That’s running Raspbian.
Vi or e-macs? (minimim)
I’ve used both, but vi is the tool that’s most likely to be installed on any given Linux system, so that’s my preferred editor. If you do sysadmin work, you pretty much have to know VI, or the day cometh when you’ll find yourself with a console terminal and no way to run “yum/apt-get install emacs/nano” (pity the fool). 🙂
What is the gear for the various services? Where is it hosted? (minimim)
We split our infrastructure into three main components — core infrastructure, interactive web services, and frontends. Core infrastructure runs our gitolite server, kup server for tarball uploads, and internal tools. Interactive web runs things like bugzilla.kernel.org, patchwork.kernel.org, wiki.kernel.org, etc. The frontends run www.kernel.org and git.kernel.org.
Excepting the frontends, everything is in Portland, Oregon. The frontends are hosted by:
ISC, in Palo Alto and San Francisco
Tizen, in Portland, Oregon
Vexxhost, in Montreal, Quebec
These lovely people donate us 1Gbps of bandwidth at each location — for which we very, very thankful.
Gear-wise, we have some older donated HP servers, but most of the stuff is running on Dell PowerEdge R610s, with a large NetApp on the backend for networked storage. All recent hardware is funded by the Linux Foundation.
Favorite automation tool and which one (if one) do the team uses in kernel.org? (minimim)
We’re a Puppet shop, though if I could do it over, I’d switch to either Ansible or SaltStack. I hate the fact that Puppet is Ruby, as it’s the only thing that pulls in the whole Ruby stack onto my systems. Honestly, Ruby VM is awful — Puppet had to switch to Clojure just to get over the fact that admins had to set up Passenger just to stop Puppet server from falling over when your system count gets into hundreds.
(Disclosure: I have no love of Ruby.)
How many users do you have? (minimim)
We currently count ~300 users, who are usually either kernel module maintainers or high-profile developers. To qualify for a kernel.org account, people have to either be listed in MAINTAINERS or receive a special approval from the steering committee (Linus, Greg KH, H.P.Anvin, Ted Ts’o). We also require that people are in the kernel.org PGP web of trust, which means that before anyone is given access, they must have PGP signatures from at least 3 other kernel developers who already have a kernel.org account.
Do you run all Linux systems, or do you run other OSes too? ( Goofybud16)
We run some gasp Mac and Windows systems that serve as builders for Collaborative Projects using our CI infrastructure (Allseen Alliance, mostly).
It looks like the old fingered daemon had been replaced. https://www.kernel.org/finger_banner I was curious what other archaic services were still running at kernel.org. (stonefoz)
I’m amazed how many people still ask for fingerd. It’s dead, Jim. Honestly, come on. It’s not 1988 any more. I would love to kill FTP, too, but that’s not likely to happen any time soon.
What kind of security challenges do you face? Is kernel.org unusually more or less targeted than most websites, or about the same? (DJWalnut)
All of them. So do you. 😉
It’s been a few years since the kernel.org hack and we still don’t have any details even though it was said a writeup was coming after the investigation is done. Where is the write up.. ? (bcopy)
I don’t have too much detail, as this both happened before I started at the Linux Foundation, and because, to my knowledge, this is still an active investigation by the FBI. Therefore, I can only provide what is already publicly known anyway — the attackers managed to obtain private ssh key credentials from the laptop of one of the administrators (how exactly, that is not known to me). That allowed attackers to ssh in and elevate their privileges on the servers. Then they installed a rootkit that allowed them to get in via a backdoor. That’s basically the extent of it. There is nothing hush-hush about it.
These days, we have a strict policy that all administrators must keep their ssh private keys on PGP smartcard capable devices, such as Yubikey NEO or a Gemalto smartcard, plus everyone must additionally provide a 2-factor token when performing sudo.
I can’t tell you much about any promises of write-ups, as that was before my time.
Is France still blocked? (dagbrown)
Oui.
Longer story, since someone will go “huh?” A while ago we discovered that something is absolutely hammering ftp.kernel.org from all over the French IP space by opening a connection and then immediately closing it (SYN-SYNACK-ACK-FIN). We counted about 100-200 such connections per second, all from France, all from what looked like mobile IP ranges. The best we figured, there’s some kind of a mobile app popular in France that uses “am I able to connect to ftp.kernel.org” as a sort of a “do I have an Internet connection” test. Unfortunately, the only sane mitigation strategy was to block all of France from being able to use ftp.kernel.org.
Wouldn’t have been a problem if they used http, but the way vsftpd works, this was causing a fork/destroy for each connection, such as our PID counter wrapped around every 3-4 minutes.
On working for the Linux Foundation and being a system administrator
How does your day go about? Can you work from home like Linus does? (minimim)
Everyone employed by the Linux Foundation works remotely, the IT team included. A lot of our team is US West Coast (Portland, Seattle), but we also have quite a number of people working from Montreal.
We love hiring in Montreal — province-funded programs such as universal healthcare, subsidised childcare, subsidised parental leave, etc, make Quebec a top destination for well-educated, bilingual or tri-lingual remote employees. </shamelessplug>
At what point does one know that they’re ready to start applying for Linux Admin. Jobs? (Meth_Tical)
<shill>When they have passed the Linux Foundation Certified Systems Administrator Exam, of course. ;)</shill>
Did you take the test? Did you help develop it?
I have taken the LFCE (tougher). I didn’t develop it, but our team was involved in early try-outs. Everyone passed. 🙂
What do you run on your workstation? (minimim)
I’ve been a part of Fedora Project since it’s very early days, so that’s what I run on my workstation. We are distro-agnostic on our team — as long as basic security guidelines are followed.
Is it BYOD or what the foundation gives the team? Does it work 100%?
Sysadmin staff is given a budget to spend on their preferred hardware. To a sysadmin, their laptop is like their second pair of hands, so forcing them to use this or that brand is just wrong.
