DevOps has become a very popular field and buzz term within the software industry but what is it exactly? Amazon AWS defines DevOps as “the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity.” More concisely: the ability to deliver products faster using toolsets and processes. Processes such as Continuous Integration (CI) and toolsets such as automated build systems are some examples of what can be used to speed up the software development lifecycle (SDLC). The use of these toolsets and processes have fundamentally changed the way software is developed and the rate at which releases can be delivered to customers. What’s also important (and should not be dismissed) is the level of quality that comes with introducing DevOps into the SDLC. By integrating automated pipelines, testing, and other DevOps concepts, users release higher quality quality software at lightning speed.
Embedded Systems: The Abandoned Sibling
Software systems have made significant advances with the use of DevOps but their neglected sibling, embedded systems, has not. A significant number of embedded hardware and software teams still use older version control systems (if any at all), little or no automated testing, and virtually no automated build processes. The concept of Embedded Devops is to take those philosophies, practices, and tools and apply them to embedded systems.
Embedded DevOps
In the Venn diagram below (Figure 1) you will notice three circles overlapping: Electronics, Software, and Devops.
Figure 1: Depiction of various fields overlapping to make Embedded DevOps
The electronics and software fields have coexisted for ages (as embedded systems) but never with DevOps. When you introduce DevOps into embedded systems you change the whole paradigm of the development lifecycle. Within Embedded DevOps there are a few key practices that, when adopted properly, accelerate product development at greater efficiencies and quality. These practices include:
Automated compilation: Anyone, Anywhere, Anytime
Self checking simulations: Anyone, Anywhere, Anytime
Hardware in the loop testing: Anyone, Anywhere, Anytime
Automated deployments: Anyone, Anywhere, Anytime\
Note the repeating qualifiers stated after each Embedded DevOps component: Anyone, Anywhere, Anytime. These qualifiers ensure that each one of these components can be executed by anyone at any time and anywhere they would like it to happen. Let’s review each qualifier in detail.
Automated Compilation
The “build anywhere” is a key element for Embedded DevOps (and DevOps in general). The ability to create repeatable builds by anyone on any machine (i.e. anywhere) at any time is critical to avoiding the classic, “but it built fine on my machine!” We enable such a process with containerization. The objective is to encapsulate our build environment into a very lightweight virtual machine - also known as containers. One can choose to run these containers on their machine, in the cloud, or even replicate the container (i.e. install all the dependencies) locally on their machine. This Dockerfile grabs an image of Debian and adds the necessary GCC tools to compile and build for various Atmel devices. The container is built and pushed to a registry which can then be used for individuals or cloud based machines. After every commit, for example, this pipeline gets triggered and generates the programming files needed to flash an Atmel SAM4E device. If the project can be built within a container (essentially a brand new computer) then it can certainly be built on your machine running the same environment.
Self Checking Simulations
In the software world running linting, static analysis, and other code checking tools have become common practice. For embedded software teams this is equally important but also extends to software simulation tools. For embedded software using a full stack linux this could be simulating your code on an ARM based Docker container. For FPGA designs it can be running simulation pipelines on Docker containers.
Hardware in the Loop Testing
The concept of hardware in the loop testing functions the same way automated tests work for software only projects. The idea is to have automated code checking and testing but for your embedded target. There are simulators that can support this but nothing replaces true hardware tests. When integrating hardware in the loop testing with Embedded DevOps platforms the process looks something like this:
Write Code → Test Locally → Commit/Push Changes → Build Binaries (Server side) → Perform Hardware in the Loop testing (Server side) → Report Results (Server side)
Tests run locally by developers may not be exhaustive enough (e.g. test one feature but not others) or can sometimes be overlooked. Having the automated build system trigger hardware in the loop testing attempts to cover the test cases often missed by developers’ “bench tests.” As a bonus, hardware in the loop test systems can also be doubled as hardware verifiers after new boards are fabricated or reworked.
Here is a simple example demonstrating an Atmel target using a Raspberry Pi Docker image for hardware in the loop testing: https://gitlab.com/ci-cd-examples/SAM4E-XPlained-Pro-HIL-Example.
Automated Deployments
Having automated mechanisms to deploy binary images is critical to Embedded DevOps practices. Continuous Delivery/Deployment (CD) is achieved by deploying binary packages for production whether it goes directly to the released product or automated production testing. This ensures everyone gets the same code at the same time and no one releases binaries on their local machine. Atlassian Bamboo, a CI tool, provides a built-in deployment system whereas other CI tools, such as Gitlab CI or Jenkins, use third party binary managers such as Artifactory or Amazon S3 storage. The latter systems usually require extra pipeline steps to deploy to different development stages (i.e. staging versus production).
