Embedded Open Source Summit 2024 Recap

by Thomas Sarlandie

April marked the return of the Embedded Open-Source Summit, this year in Seattle. I was lucky enough to be able to attend and split my time between the Memfault booth in the exposition hall and many of the captivating presentations. Since the videos have just been published on the the Linux Foundation’s YouTube account, we thought it would be a good time to highlight some of the talks and give you a quick summary which will, hopefully, inspire you to go watch them!

We cover the talks I was able to see in person, as well as some talks seen by my colleagues since they were posted. Obviously this is just our little biased selection, we have not been able to see everything, let us know in the comments what we missed!

Comparing Linux OS Image Update Models

Drew Moseley, from Toradex and ex-Mender engineer, did some great research to compare different strategies to update Linux systems. He starts with a little explaination of what delta updates can look like for an A/B system: he covers SWUpdate with ZChunks, Rauc, and Mender proprietary delta-update. Of course, he also includes Toradex and starts with a little refresher of how it works differently using libostree (a git like approach to your entire filesystem).

Then he devised a few “update scenarios”, including one very small change in a configuration file, installing a big package, removing a big package. The choice of scenarios seemed very relevant to me and I was glad to see someone take the time to “run the numbers” and compare the number of bytes downloaded for each scenario and each solution.

As always, the perfect solution will depend on your use-case. This talk gives a solid methodology and some solid starting numbers to compare different approaches. Thank you Drew for taking the time to do this and share it with the community!

Maximizing SDCard Life, Performance and Monitoring with KrillKounter

(EDIT: An initial version of this post incorrectly attributed the “silent-read-only behavior to Samsung - Andrew clarified that this was observed on a SanDisk card. We regret the error.)

Andrew Murray of the good penguin shared of their research on SDCards. Like Andy, we often run into customers who use SDCard as their primary flash storage and have lots of questions about the reliability of these cards. I will now systematically point them to this great talk!

Andy started with a very thorough explaination of how flash storage works. I thought I knew the topic well but I still learnt quite a bit, especially what multi-layer flash storage really is and how it works (the 0 and 1 on the flash are really analog values - instead of reducing one analog value to a 0 or a 1, the multi-layer flash will infer 2 or 4 bits from one analog value - this is why these types of flash memory are more “fragile” - fascinating stuff). He also covered in great details the role of the flash controller and its different roles.

He then presented a test setup that his team and him devised to take some real-world measurement of flash storage and see how it would fail. He discusses how different flash brands can fail in different ways (some Samsung SanDisk apparently goes read-only - without telling the OS that it’s not writing, some other flash just start timing-out on all read and write operations).

KrillKounter is their attempt to build a tool from all this research. It will keep track of how many writes to a flash (not easy because it needs to be tracked across reboots and for each SDCard that might get pulled/inserted in the device).

There was a lot more in this talk and I can’t recommend it enough!

Engineering Secure SSH Access for Engineers

Colin (Garmin) goes through the challenges for enabling SSH access to devices in production. Passwords and secret keys are not great solutions: if you share one across all your devices then a leak would be catastrophic. And a leak is hard to protect against because the engineers need the keys with them when doing maintenance work. So instead, he recommends using certificates to create keys that are only valid for one hour or one day. The device SSH daemon validates that the engineer key has been signed by the proper certificate, and that we are still within the correct validity period. This way, the only real secret (the private key used to sign the keys) can be stored in a very safe storage, engineers are only granted access for a few hours (so an ex-employee will not have access anymore).

The talk goes in much more details on how to actually implement this. Demonstrates the commands to create certificates, options available in SSH to restrict permissions of the key, mentions a few tools that can be used to manage the certificate and sign the keys for the engineers, etc. He also discusses some challenges that are specific to embedded devices such as how to deal with devices that may not have the correct time set (he also reminds us that NTP is not a safe protocol and time attacks are possible).

A great talk from someone who has obviously gone pretty far into the implications of keeping SSH open on production device. Another one that I will be recommending for years to come!

From C to Rust: Bringing Rust abstractions to the Embedded Linux

Fabien Parent, of Linaro, is a well known name of the community. I was curious to hear him talk about Rust. The talk did not disappoint.

Fabien starts with a good overview of what is available for Rust in the Kernel. He explains that Rust is limited to the device drivers because “core” components of the Kernel needs to remain compatible with all architectures and some of these architectures are not supported by the LLVM backend used by rustc (LLVM is shiny and new but the kernel built with gcc still supports more architectures).

A big challenge for Rust kernel developers today is that although tooling was merged, there is very little in terms of abstraction. So to develop a new device driver in Rust, you first need to write some abstractions. Many people have started working on this but this work is often left unmerged because code cannot be merged in the kernel unless it’s actually used somewhere. Any serious device driver will require many abstractions, making for some big patch requests that are complicated, hard to review (especially by non-Rust developers) and hard to merge.

At this point, you probably share a bit of my disappointment. The best lesson of the talk was Fabian’s optimistic approach to all of this. Instead of giving up, he put some thoughts into the best way to move everything forward. His strategy is to find some small drivers (that do not exist today because just replacing C code with Rust code is also not acceptable for merge) that require only a few abstractions and to use them to progressively introduce Rust abstractions in the Kernel.

I loved hearing this positive take on all of this. A good reminder that you can sit and complain about the state of things or take it as a challenge ;)

Yocto Project: 5 Year Plan, Now What?

Megan Knight (AWS) and Andrew Wafa (ARM) went through the 5 year plan for Yocto, highlighting all the results and sharing more on what is coming next.

Yocto is such a big project, and we are often working with pretty old versions. It was refreshing to hear about the “modern” side of Yocto and got me interested in testing some of the new stuff (maybe even give the graphical builder another try 😛). I heard that a big part of this 5 year plan effort was to catch up on technical debt and that now that it’s done, they are hoping to be able to attract more volunteers to stay on top of the bug flow and also work on some exciting new features.

It was also clear from the talk that although there is a lot of interest in Yocto, there is not as much money flowing into this project. If your company can contribute money or resources to Yocto, now might be a great time to do it!

Delta Updates: Making Updates Leaner

I only caught glimpses of this talk but I understand they have been doing some research on how to strategically prepare the filesystem images so that delta updaters would be able to compress the delta more efficiently. This is a fascinating topic as we know how important compressing updates can be for some of our customers, especially these relying on expensive cellular modem radios!

Unwrap()-ing Rust on Embedded Linux

Obviously a bit biased here as this was my own presentation. This talk was very much from the angle of “everything I would have liked to know when I started writing Rust for embedded Linux”. We covered different ways to cross-compile and package Rust apps, what to expect in terms of binary size, some strategies to reduce the size of the binary and a few other lessons-learnt.

Status of Embedded Linux

Marta (Syslinbit) and Tim (Sony) discussed everything they saw of interest in the embedded linux world. The scope was wide, covering GCC improvement all the way to discussing the status of the elinux.org website.

I won’t try to summarize this as this was already a very dense summary of embedded linux news. I do encourage you to take a listen to get a better sense of what is happening in our world.

Rusty swapping: Rewriting a zswap backend in Rust

Vitaly (Konsulko) shared his experience writing a kernel driver in Rust. After listening to Fabian talk the day before, I was curious to hear another take on Rust in the kernel. Vitaly shared his experience writing a new zswap backend in the kernel. For fun, he wrote it both in C and Rust and compared the results.

His conclusions are similar to Fabian’s: Rust in the kernel is hardly ready for prime time today. The Rust learning curve is a challenge, getting the tools set up correctly seems like a major pain, collaborating with existing kernel C API requires a lot of unsafe code which reduces the “memory safety” advantage of Rust. And once again, figuring out the way to get Rust code merged seems to be the biggest issue. One encouraging point was a slight improvement in performance and size with the Rust version.

Patterns and Anti-Patterns in Embedded Development

Marta Rybczynska walks through security incidents from 2023 concerning devices running Embedded Linux, protocols used by devices, and the Linux kernel itself while going over what patterns and anti-patterns we can take away from the incidents. In particular, I found the segment about the train software fascinating. That the security researchers found that almost every train had a different software version really shows the importance of having a defined OTA and versioning process, not just for software management but also security. As a big fan of the Yocto project, I also great enjoyed the fun fact about how the now-infamous XZ backdoor wouldn’t build on the Yocto Release Candidate that included it!

Charging a Battery with Zephyr

The maintainer of the charger drivers API, Ricardo Rivera-Matos from Ciirus Logic, explains the role that a Zephyr host can play in controlling battery charging. Notably, in enables setting the charge profile, setting protections and limits for battery charging which could include limiting the maximum charge percentage to improve battery life, and getting status info on charging. The charger driver API was inspired by the Linux Power Supply Class, so for those familiar with power management in Linux, the charger API should be familiar. We also got a sneak peak at some of the improvements they want to make to the charging API, including adding support for charger telemetry to observe a charging session with ADCs, for example. This was a great talk that got me wanting to try out this API!

Enabling Real-Time Secure Connectivity to the Industrial Edge with Single-Pair Ethernet and Zephyr

Jason Murphy from Analog Devices discussed the usage of Zephyr on the industrial edge using support for 10BASE T1L Single-Pair Ethernet. He explained how Zephyr is great for industrial edge applications because of its native TCP/IP networking stack supporting Ethernet protocols and low-latency networking, among the well-known benefits like portability and community support. He also walks through the board configuration (devicetree and Kconfigs) to use the ethernet drivers and board support for the ADI eval boards for 10BASE T1L. I liked learning about this option for wired applications using Zephyr since most of the conversation in IoT is around wireless protocols!

Conclusion

Does this feel like a lot of great content? We are pumped just writing about it! And we have only scratched the surface: there are 60 videos for the Embedded Linux Conference, 53 from the Zephyr Developer Summit, and many more from the Open-Source Summit. Let us know what we missed in the comments below!

You can count on us to be there against next year!

See anything you'd like to change? Submit a pull request or open an issue on our GitHub

Thomas Sarlandie is the Linux Tech Lead at Memfault.