The Embedded Linux Development Process
The Linux kernel can run on many different computer architectures, most of which are quite popular in the embedded world. All of the base packages allowing the OS to perform the basic tasks are suitable for cross-compilation, therefore Linux can be as pervasive as microcontrollers and Systems on Chip (SoCs).
A Linux distribution is an operating system made from a software collection, which is based upon the Linux Kernel and – often – a package management system. The distribution may come as pre-compiled binaries and packages put together
by the distribution maintainers, or as sources paired with instructions on how to (cross-) compile them.
In the embedded domain, since the hardware platform is usually bespoke, the OS designer generally prefers to generate the distribution from scratch, starting from sources. This gives the designer absolute control of what ends up in the
product. Furthermore, the Board Support Package (BSP) engineer modifies the low-level code in order to make the core functionality of the OS work on the specific hardware product.
Getting all the necessary software components together to generate the Linux distribution for a particular embedded product used to be a nightmare, butthankfully this is no longer the case.
Many have shared with the open source community the sources of build systems capable of fetching all of the Software components off the Internet, compile them and link them together, up to the generation of installation images of fully fledged operation systems. A few companies are developing and maintaining their own build system, others compile just few of the core components and then take pre-built binaries to finalize the OS.
In 2010, a workgroup from the Linux Foundation started to address tools and processes to allow the creation of Linux distributions for embedded software (a.k.a. Embedded Linux). Such a workgroup, known as the Yocto Project, aligned itself with Open Embedded, a framework with similar goals.
The Yocto Project is an open source project whose focus is on improving the software development process for embedded Linux distributions. The Yocto Project provides interoperable tools, metadata, and processes that enable the
rapid, repeatable development of Linux-based embedded systems.
The Yocto project is currently powering the most popular Linux distributions for embedded system, to a point where sometimes the terms “Embedded Linux” and “Yocto Project” are easily confused as synonyms. Yocto it’s not an Embedded Linux distribution, it creates a custom one for you.
Yocto’s meta-layers layout
The modern version of Yocto’s architecture is based on meta-layers, which are directories containing configuration files and rules on how to compile and assemble Linux distributions for embedded systems.
Usually, but not always, a meta-layer lives on its own git repository, and provides:
- its own packages (defined by recipes, .bb files),
- modifications to packages provided by other meta-layers (.bbappend files),
- machines (.conf files),
- configuration files (.conf files),
- common code (.bbclass files),
- licenses,
- and other minor bits and pieces.
A single meta-layer normally addresses a specific purpose. Therefore to achieve a fully working system, more meta-layers need to be combined together.
Versions picking and matching
When putting different software components together, one needs to be mindful of the version of each component, as the wrong version may not work well together with the other components or even break the system.
The Yocto project provide releases of components known to work well together, but that is just the starting point for your product.
The Linux kernel is a big chunk of code that needs to expose the right interfaces to user space, and has to contain the right drivers, for the system to work properly. Therefore, the role of the silicon vendor has become more and more important these days, as they usually have their own development repositories for the Linux kernel and the bootloader, hence, they are the best people to put together a working base system based on their technology.
Google’s repo
Originally developed to cope with the multitude of Git repositories in an Android project, repo has become quite popular among Yocto developers too.
Repo is a tool built on top of Git; it uses a “manifest file” to clone and pull a set of Git repositories all the same time.
A repo manifest is an .xml document containing references to git repositories (along with their versions), repo can use the manifest to populate a directory with all of the sources coming from the several Git repositories required to build a project.
Also, the same manifest may be used by repo to keep in check (sync) the project sources when upstream make changes.
A few silicon vendors provide manifests for their development and release branches these days, so that designers can easily check out the starting point for their own products.
Yocto-based product development
The BSP engineer usually starts from the silicon vendor repo manifest to checkout the version of the software for the reference design (which is, a design provided by the silicon vendor itself or one of its partners, containing the same or similar SoC to the one the new product is based on). The engineer makes changes to the bootloader and the Linux kernel to make sure the hardware selected by the Electronics Engineer has proper low-level software support (e.g. device drivers, device tree, kernel configuration, etc.).
The purpose of the product is to run one or more applications, therefore the BSP/OS engineer makes sure that all of the dependencies of the application(s) are being built for the system. The engineers developing the application need a Software Develpment Kit (SDK) to cross-compile and link the application, therefore the BSP/OS engineer
will provide them with such a kit, and thanks to Yocto this has become quite straightforward.
Embedded Linux good practice
The repo manifests used for development usually contain reference to development branches, which mean repo will fetch the latest commit for those branches.
If you use the same manifest to fetch the project at a later date, you may fetch a different version of the code! This is perfectly fine for development because you want to stick with the latest version of your project, but one of your development versions will eventually become a release, and therefore you need to “take a picture” of that precise version of the sources used to generate the software release that goes on the project. Failing to do so can expose you to legal troubles as you won’t be able to regenerate the same build starting from sources, therefore you won’t be able to make a change on top of a specific release, forcing the customer to re-test the entire system as you’ll be forced to fix the bug or add the new feature on top of the latest version of the software.
Also, if you don’t take those snapshots, there is no way you can run a bisect on the project sources to find out which commit has broken the functionality you so desperately need. When designing your development process, find a way to automatically generate repo manifests with precise commits in them, so that you can save them alongside releases to checkout the same sources again at a later date, and do whatever you are paid to do.
Copy sources in house
Keep also in mind that 99.9% of the sources that go inside your product come from the open source community, which means you have no guarantee the same sources will be available for download again. As a designer, you need to protect yourself against changes and mistakes made upstream. Keep a copy of all the relevant sources in house, and find a way to plug them back into your build system. Also, you may want to mirror some of the repositories you use the most, as sometimes upstream git servers may suddenly became unavailable. If you don’t have an internal copy you’ll be stuck until the servers will come back online.
At ByteSnap, we have a fully automated way of releasing Yocto based projects, such that we can get recover the sources that go into a release, and also, re-build the same release at a later date. We keep copies of open source packages in an automatic fashion, so that we experience no down time caused by faulty servers around the world. Furthermore, we back everything up every single day so that we can guarantee no work will be lost even in case of disaster on site.
Fabrizio Castro
Fab is a senior software engineer. He gained his bachelor and master degrees at Politecnico di Milano, in Milan, Italy. He has 20 years’ experience of all-round software development (services, data bases, applications, scientific software, firmware, RTOS, device drivers, Linux kernel, etc.), spent working in academia and industry. Fab has co-authored scientific papers and books, and worked on patents. As well as research and development, he specialises in Embedded Linux development – delivering state-of-art designs powering successful scientific, industrial, commercial, and military products. Fab has also been a lecturer and has taught undergraduates at some of the most prestigious universities in Europe. For more information about ByteSnap Design visit http://www.bytesnap.co.uk.
