Firmware Update Failure: Why Devices Must Fail Gracefully

A firmware update failure is a major project red flag.

Yet, most firmware engineers think the goal of an update is to succeed.

It’s not.
It’s to fail gracefully.

The Firmware Update Failure Bug

This week’s lesson came from my own home.

A treadmill I use nearly every day forced me into a firmware update.
No option to skip. No option to delay.

Then . . . failure!

The screen displayed a “retry” button, but no input was accepted. Each tap gave me nothing but a short beep. The update had locked up completely.

The treadmill was effectively dead. A textbook firmware update failure.

The Firmware Update Failure Problem

The update process was designed only for success.
No validation of preconditions.
No fallback image.
No recovery mode.
No path out of a failed update loop.

In the lab, it probably looked perfect. But in the field, it turned into a brick with a power button.

This is a failure we see often across embedded systems: developers test only the “happy path.”
Everything works until something goes wrong.

When it does, users pay the price.

The Real Lesson Behind Firmware Update Failures

Firmware isn’t judged by how it behaves on the bench.
It’s judged by how it behaves in the field, under stress, when something fails.

If your update mechanism can’t tolerate failure, your product can’t tolerate the real world.

Devices in the field deal with:

• Partial downloads due to poor connectivity
• Power interruptions during flashing
• Corrupted images
• Bad flash sectors
• User impatience

And yet, most update logic assumes a perfect environment.

That’s not engineering, that’s wishful thinking.

How to Design Firmware Updates That Survive Failure

Here’s how to prevent your product from becoming an expensive brick in someone’s living room:

1. Build fail-safes into every update cycle.
   Always assume something will go wrong such as a power loss, corrupt data, or user interruption. Design a mechanism to detect and recover.
2. Include a known-good fallback image.
   Dual-image bootloaders or A/B partitions can recover automatically if the new firmware fails validation.
3. Validate system readiness before updating.
   Check power levels, available flash, and communication stability before starting the update. If the system isn’t ready, postpone it.
4. Test under worst-case conditions.
   Simulate power cuts, corrupted transfers, and bad CRCs. See what happens, not just in the happy path, but in the messy real-world ones.
5. Preserve data integrity.
   If an update wipes calibration or user history data, you’ve failed your users. Data migration and backup should be integral to the update process.
6. Don’t assume the user can recover the system
   A commercial product should have automated recovery, not one that relies on the customer and customer service to restore the system to working order.

Want to learn about how to create fail-safe bootloaders successfully? Check out my course on Bootloader Design Techniques for MCUs, or check out the Embedded Software Academy.

The Aftermath of My Firmware Update Failure

I eventually found a hidden reset button that triggered a factory restore.
It failed three times.
When it finally succeeded, seven years of maintenance data were gone. (Not critical to me, but it could have been to someone else!)

That’s a real-world lesson in what happens when firmware update design stops at the success case.

The Takeaway

The next time you design or review your firmware update system, ask yourself:

“If this update fails halfway through, what happens to the device?”

If your answer isn’t clear or comforting, it’s time to redesign.

Because firmware that can’t fail gracefully will eventually fail publicly.
And once a product loses user trust, no update can fix it!

Want to learn more about firmware updates? Check out these additional resources:

• Blog – Jumping from the Bootloader to the Application Code Cleanly
• White Paper – Bootloader Design for Microcontrollers in Embedded Systems
• 7 Tips for Designing a Robust Bootloader