Why You Don't Hear About Firmware

The Hidden Ubiquitous World Around You

On Substack, you’ll find incredibly thoughtful posts about engineering culture, machine learning, web development, and AI (I highly recommend Society’s Backend if you’re looking for an example). However, if you search firmware on Substack the first entry is “Chicks Dig Muscles” with a thumbnail of an AI-generated musclebound man. I’m not sure why or how, but I think we need to change that.

Firmware is code baked into the devices you use. It’s not hardware as it can be changed, but it’s low-level difficult to write code. And it is everywhere. It’s in your dishwasher, car, iPad, stereo, street lights, and phone charging cable. At this point, almost anything that “turns on” has firmware. Given the sheer scale, why don’t you hear from the engineers responsible for it all?

Editor’s note: my experience is working at Microsoft, Apple, and developing small embedded devices. My views are biased heavily towards those types of firmware. My opinions are my own.

The Industry

Firmware engineers often come from an EE background instead of a CS background, which means they tend to run in different circles than the software folks. So I bet you would be surprised about the large number of firmware engineers. There are dozens of us! Dozens!

Image is from Arrested Development, may be subject to copyright. Please don’t get mad, I’m new to this.

I’ve spent several years in the boot firmware space (UEFI) and have fond memories of going to some of conferences. At one of the largest conferences of the year there were folks from Microsoft, Apple, IBM, Lenovo, Dell, Red Hat, Intel, AMD, and ARM. Some of the world’s largest and most powerful companies were represented. Combining the market cap of all the companies would easily total in the double digit trillions. There were 200ish people at the event and many companies sent their entire team (my entire team for example). That was it.

I have set forward a bit of a conundrum. How can there be landfills of firmware but a handful of firmware engineers? The industry as a whole has a bit of a fork and run model problem (I‘m generalizing here, based off my experience in UEFI, but similar problems apply to other parts of the firmware world).

Fork And Run

The industry has a bit of a problem with fork and run. What I mean is that for many projects that are similar to an existing project, just take what you have, create a new fork, hack it until it works, then ship it. New dishwasher model? Just take the old dishwasher code and add in support for the new microcontroller. You save time, but now have fragmented codebases that are all subtly different from each other. If there’s a bug, it’s painful to apply the fix to all the repos/branches. Some are trying to move away from a branch model (see Microsoft’s Project Mu) towards a common core model.

But even with those efforts, it’s just a common core not a core firmware. Each firmware is highly specific to a specific model. An iPhone 15 and iPhone 15 Plus have different firmwares, despite being very very similar internally. From an external perspective, the only thing different is the screen. And you’re right, but that means the GPU firmware needs to be tweaked. And the image assets that get baked into the firmware are different. A Surface Studio Laptop and a Surface Laptop are completely different and might only share a few things code wise.

So each product gets its own firmware build and often its own repo. What’s that you say? Firmware developers must be drowning in technical debt? Not so fast. You can’t have technical debt if you never revisit a codebase. For many consumer products, they receive somewhere between zero and ten firmware updates. They ship with the QA verified code plus some last minute factory fixes (and there are almost always factory fixes). You don’t need to worry about dependencies if you’re only building the firmware a handful of times. Part of that mentality comes down to the fact that updates are incredibly dangerous.

Risky

Firmware is fundamentally different from software in a few key areas. It is often risky to update it and in some cases, fatal. A medical devices company a friend of mine worked at shipped a problematic firmware and the medical implant started boot-looping in a certain percentage of people. Those people needed operations to physically remove the implant in their brain and reset it, which is never without risk. Hopefully every engineer has heard of the story of the Therac-25, which killed at least six people due to buggy firmware that had been forked from another model.

For non-medical devices, every device is one bad firmware update away from being a near permanent brick. Obviously, with physical access and the right tools, almost anything is possible to fix. But for most devices, the old firmware is in charge of installing the new firmware. And if it’s faulty, you just get no new firmware, or worse, corrupted firmware.

For this reason, updates that do get shipped out are scrutinized heavily. And because all the products get their own build, each product must be validated individually. To save on quality costs, it can be easy to rationalize starting with an existing code base or to not touch the code. Just leave it alone unless a seriously critical bug shows up.

Very Little User Interaction

At the end of the day, most people don’t care about the firmware around them. You don’t care what your dishwasher does while it runs. For PC hardware, the firmware is a temporary trampoline. Just some bootstrapping code that gets you to the OS. It runs for a few minutes, leaves a few pieces of itself resident, and lets the OS take over the rest of the system. Unless you’re a security research or big on open source, you likely just don’t care what the CPU in your laptop does while it’s booting. You don’t interact with the firmware in a meaningful way outside of the occasionally journey into the BIOS and how many people think that’s an enjoyable experience?

It’s a somewhat thankless job. But when people complain about some issue with their iPhone or Surface, I can smile and ask

“Did it boot?”.
“Well yeah? But when I launched this app it start to…”
”If it booted, it’s not my problem.”

Consider subscribing to Rambleware, it’s free and encouraging to me to continue writing. Gotta get those fake internet points somehow.