It’s still a bit unsettling to see a Microsoft speaker at a Linux Foundation conference. Yet, at the recent Linux Security Summit, Ryan Fairfax, Microsoft’s head of OS development for Azure Sphere, quickly put the audience at ease with his knowledge of tuxified tech. His presentation on “Azure Sphere: Fitting Linux Security in 4 MiB of RAM” fits into the genre of stories in which developers are challenged to strip down their precious code to the spartan essentials for IoT.
As we saw last year in Michael Opdenacker’s presentation about reducing the Linux kernel and filesystem for IoT, Linux can be made to run — just barely — in as little as 4MB of RAM. That was Microsoft’s target for Azure Sphere OS, the open source Linux-based distribution at the heart of the Azure Sphere platform for IoT. Azure Sphere also includes a proprietary crypto/secure boot stack called the Microsoft Pluton Security Subsystem, which runs on an MCU, as well an Azure Sphere Security Service, a turnkey cloud service for secure device-to-device and device-to-cloud communication.
Last week, Seeed launched the first dev kit for Azure Sphere. The Azure Sphere MT3620 Development Kit features MediaTek’s MT3620, a 500MHz Cortex-A7/Cortex-M4F hybrid SoC that runs the lightweight Azure Sphere OS on a single -A7 core. The SoC’s 4MB of RAM is the only RAM on Seeed’s Grove compatible dev board. Other SoC vendors besides MediaTek will offer their own Cortex-A/Cortex-M SoCs for Azure Sphere, says Microsoft.
Major shrinkage
Fitting an entire Linux stack into 4MB was a tall order considering that “most of us hadn’t touched Linux in 10 years,” said Fairfax. Yet, the hard part of creating Azure Sphere OS was not so much the kernel modification, as it was the development of the rest of the stack. This includes the custom Linux Security Module, which coordinates with the Cortex-M4’s proprietary Pluton security code using a mailbox-based protocol.
“We decided early on to go with Linux,” said Fairfax. “Most of our changes to the kernel were small, and the core Linux features ‘just worked’ even with limited resources. That’s a credit to the effort of the community and flexibility of the kernel.”
Fairfax’s team started working on Azure Sphere in secret in 2016 after struggling to convince Microsoft leadership that working with a Linux kernel “was viable,” said Fairfax. The project was unveiled in April 2018, and the first public preview will be released soon.
One of the main goals of Azure Sphere was to bring security to the MCU world where “security is basically nonexistent,” said Fairfax. Microsoft somewhat confusingly refers to the MediaTek MT3620 as an MCU rather than an application processor due to its inclusion of Cortex-M4 MCU cores. In part, this may be a marketing ploy since Microsoft intends to compete directly with the Cortex-M oriented Amazon FreeRTOS.
Sign up to receive updates on Open Source Summit and ELC+OpenIoT Europe:
Azure Sphere OS sits on top of the MCU’s Pluton stack, architecturally speaking, and the base layer is a security stack based on Arm TrustZone. This is followed by the custom Linux kernel, which in turn is topped by a cloud services update connectivity layer. The top level is for POSIX and real-time I/O app containers.
The custom kernel is currently based on mainline Linux 4.9. Patches are merged upstream every month, and there are plans to upgrade to LTS branches yearly.
The first step in reducing the kernel was “to avoid putting text into memory,” said Fairfax. To do this, the OS depends a lot on Execute-In-Place (XiP) technology, which is commonly integrated in MCU flash controllers. “XiP lets you take a flash region and map it into the address space in read only, but also in a mode where you can execute it as code.”
In addition, “we tuned the kernel to make things modular so we could turn things off,” explained Fairfax. “We tuned cache sizes and patched to tweak default sizes.”
The team turned off a lot of the memory tracking options and things like kallsyms. They reluctantly cut sysfs, which saved almost 1MB, but for Fairfax was the coder equivalent of the writer’s challenge to kill your darlings. In the end, much of the kernel space was taken up by the network stack and hardware drivers.”
A lightweight Linux Security Module
Initially, the Azure Sphere OS team tried using SSH server with a fixed root password for security, but they quickly realized that this “was not going to cut it long term,” said Fairfax. To reduce the attack surface, they experimented with different security models, including “baking things into the file system and leveraging set UID and SGID to create predictable environments.”
These approaches caused some IPC problems and were otherwise flawed because “they put all the burden at build time,” said Fairfax. “Any mistake would propagate through the system and leave you vulnerable.”
Fairfax and his team revisited existing Linux technologies that might help make permissions more granular and “create a model where apps can access resources with the principle of least privilege,” said Fairfax. They finally decided on a stripped-down version of the Linux Security Model (LSM), a set of kernel extensions that “would reduce attack surface by taking certain features completely off the table. There’s no shell or user account management, which really isn’t relevant for an IoT device, and there’s no sophisticated job and process management.”
Fairfax also added fields that created an app identity for every task. “Applications and kernel modules can use these new fields for extended access control,” said Fairfax. “Values are immutable — once set, they inherit by default.”
The developers “experimented a lot with file systems,” said Fairfax. They tried the read-only cramfs with XIP patches, as well as writable file systems like ext2, jfffs, and yaffs, but “they all took hundreds of kilobytes to initialize, or about 1/16th of the total system memory available.” In the end, they ported the ultra-lightweight littlefs from Arm’s Mbed OS to Linux as a VFS module.
One problem with securing a Linux IoT device is that “Linux treats the entire GPIO infrastructure as a single resource,” said Fairfax. “In the real world not everything connected to your chip has the same sensitivity. I might have one GPIO pin that toggles an LED saying I’m connected to the network, which is not super sensitive, but another GPIO might open the solenoid on my furnace to start gas flow, which is more worrisome.” To compensate, the team added access control to existing features like GPIO to provide more granular access control.
User and application model
If Azure Sphere’s kernel is not radically different than any other extremely reduced Linux kernel, the user mode differs considerably. “The current Linux model is not designed for resource constrained environments,” said Fairfax. “So we built a custom init called the application manager that loads apps, configures their security environments, and launches them. It’s the only traditional process that runs on our system — everything else is part of an application.”
Azure Sphere applications are self describing and independently updatable. In fact, “they’re actually their own independent file systems,” explained Fairfax. “They run isolated from each other and cannot access any resource from another app.”
There are initially four pre-loaded system applications: network management, updates, command and control via USB, hardware crypto and RNG acceleration. GDBServer is optional, and OEMs can “add one or two apps that contain their own business logic,” said Fairfax.
One Azure Sphere rule is that “everything is OTA updatable and everything is renewable,” said Fairfax. In addition, because “quick OTA is critical” in responding to new threats, the team is aiming for OTA security patch updates within 24 hours of public disclosure, a feat they achieved with the Crack virus. Microsoft will manage all the OS updates, but OEMs control their own app updates.
The Microsoft team tried hard to find a way to run containers, including using LXC, but “we couldn’t get it to fit,” said Fairfax. “Containers are great, but they have some serious RAM overhead.” They also tried using namespaces to create self-contained apps but found that “many peripherals such as GPIO don’t play right with namespaces.”
For now, “we have pivoted off of containers and are focused on isolating apps and making sure that our permission model is sane,” said Fairfax. “We ensure that a buffer overrun in an application only gives you what that application can already do. We build each app as its own file system so they mount or unmount as part of install or uninstall. There’s no copying of files around for installation.
“Each application has metadata in the file system that says: ‘Here’s how to run me and here’s what I need,” continued Fairfax. “By default, all you get is compute and RAM — even network access must be declared as part of the manifest. This helps us reason about the security state and helps developers to do least privilege in apps.”
Future plans call for revisiting namespaces to create “something like a container,” and there’s a plan to “reduce cap_sys_admin or make it more granular,” says Fairfax. He also wants to explore integrating parts of SELinux or AppArmor. More immediately, the team plans to upstream some of the work in memory improvements and file systems, which Fairfax says “are applicable elsewhere even if you’re talking about something like a Raspberry Pi.”
You can find more information about Azure Sphere on Microsoft’s product page, and you can watch the complete presentation below.
Join us at Open Source Summit + Embedded Linux Conference Europe in Edinburgh, UK on October 22-24, 2018, for 100+ sessions on Linux, Cloud, Containers, AI, Community, and more.
