3 Lessons Learned from an Embedded Systems Security Breach

Several years ago, I was working on a project for a client that involved an embedded Linux machine. I was planning to travel over the weekend but given the project timeline it was critical that progress still be made. The prototype device would not travel well, so we decided to connect it to the internet so that I could easily access it remotely. Friday evening the system was connected to the web and when I returned Sunday evening, I discovered that Chinese hackers had managed to get access to the system and added it to a botnet. The meantime to compromise (MTTC) for this system turned out to be relatively short, less than 48 hours. While the system was as easily recovered as it was hacked, there are several points in this little incident that I think can provide extremely good lessons for embedded systems developers. 

Lesson #1 – Know your systems meantime to compromise (MTTC)

Developers are probably familiar with common terms such as meantime to failure (MTTF) or meantime to repair (MTTR), but system security can also be looked at from a meantime to compromise (MTTC) perspective. (I’ve also heard this referred to as meantime to breach MTTB). MTTC can be defined as the average time it takes an attacker to breach, or gain access, to the system and obtain the desired assets. Those assets could be data, or they could the systems firmware or the ability to remotely control the device.

Obviously, the goal is to make the MTTC large enough that would be attackers decide that the time and resources required to breach the system is not worth the payout and therefore their time and effort. This requires that developers understand three key points:

  • The data and assets they have on their system that should be protected
  • The attacks and methods that are used gain access to those assets
  • The methods to protect their system against those types of attacks

In fact, you’ll find that these three points are included in the Arm Platform Security Architecture (PSA) and in documents that describe how to perform a security threats analysis.

Developers also need to ensure that they balance the MTTC to adequately protect their system and clients while at the same time the development costs and schedules.  

Lesson #2 – Always have secondary security measures

Considerable effort can be used to try to maximize a systems MTTC but embedded systems, especially IoT connected systems, face an interesting challenge. They may be connected to the internet indefinitely which means that even if the MTTC is very large, say three years, a system may still be vulnerable because the system may be in use for a decade or more and can be attacked 24/7 using automation.

In order to try and keep the MTTB for connected devices reasonable, it may make sense to have secondary security measures in place which is often done for physical security. For example, a physical safe is often rated based on how long it would take someone with knowledge of the safe to successfully breach it. You may find that on a home firesafe for important documents, the ratings are anywhere from a few hours down to about 15 minutes. Even sophisticated safes may not have ratings beyond several hours. This is why there are often secondary security measures such as security cameras or security guards who make their rounds in timeframes that are less than the rated breach time!

The same idea can be applied to an embedded system. If you understand your systems MTTC, you can place security watchdogs that periodically scrub memory, restart the system or perform other activities designed to thwart any software that has started to find its way through the system. (This is by no means a new concept, many space systems clients I work with will often have an automated hardware reset in order to ensure long lasting reliability and recover the space craft in the event that it has locked up or experienced some unexpected event).

Lesson #3 – Hackers are always faster than you think

Right before placing the system on the internet, if you had asked any of us how long it would take for the system to be breached, no one would have given an answer less than 48 hours. In fact, I know that we all though exposing the system for 48 hours would be fine. With so many systems on the internet, how could this one little node be found amongst the billions of devices in 48 hours? Even if it was found, surely it would still take considerable time to apply known exploits and access the system. This is all optimistic thinking!

There are programs running 24/7 that are probing, searching and testing the defenses of systems looking for vulnerabilities. It can be quite enlightening to examine your home or businesses router logs and statistics. You’ll undoubtedly find several attacks per day that are thwarted or blocked. Hackers have the tools to find your devices quickly and if you have not followed security best practices when you developed your system, your data, your customers data and your intellectual property will be at risk. As an optimistic engineer out to change the world, you can’t, can’t be optimistic when it comes to protecting your embedded systems.


There are so many companies that are now connecting their products to the internet and unfortunately there are many that are taking an optimistic approach to security. Companies often rush to market, leaving security as something that can be added later. It’s critical that developers understand their systems MTTC. In fact, it would be fantastic if this was something that had to be advertised on the box of every product. Wouldn’t you be interested to know what the MTTC is on that new smart lock you just bought and assume is securing your home?

The only way we are going to usher in a secure IoT is to develop our security strategies at the start of our development efforts, use isolation to partition our software, understand our MTTC and put in place secondary security measures designed to augment our first lines of defense. Yes, it will require more time and money but isn’t the effort worth protecting your customers and your own intellectual property?

Share >

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.