The chip shortage has forced many embedded systems companies to rethink their software architecture. Many embedded systems have tightly coupled embedded software that makes the hardware-dependent microcontroller code indistinguishable from the application. As companies have scrambled to port their firmware to new microcontrollers, they have discovered that they need to refactor and, sometimes, redesign their software architectures. Unfortunately, a software architecture redesign can lead to the same problems if the team is not careful. In this post, I will share five software architecture killers that I have seen companies stumble into, leading them to the same problems or worse, after a software architecture redesign. These architecture killers are more team related than technical. We will cover technical killers in a future post.
Software Architecture Killer #1 – Design by committee
When word comes down from on high that the development team has permission to redesign their software architecture, one of the first activities I often see is a committee assembled to develop the new software architecture. Designing the architecture by committee is a big mistake. What usually happens is that you get six to eight developers in a room, all with their agendas, ideas, and concerns. On its own, this isn’t a bad thing; however, what I’ve seen ensue is that architectural elements don’t get nailed down until there is consensus. Have you ever tried to get six engineers to agree on anything? You’ll get nine answers if you put six engineers in a room and ask them a question!
To successfully develop software architecture, one developer should be assigned to be the software architect of the system. The software architect will then have a few simple roles:
- Develop the software architecture
- Engage with other developers as needed to flesh out architecture pieces they may not be familiar with
- Hold software architecture review meetings
The goal is to remove the nearly unending; slow progress made when a team of engineers tries to design by committee. Everyone’s concerns, knowledge, experience, and feedback are addressed in the review meetings, but the software architect is unleashed and able to make decisions that quickly move the design forward.
Software Architecture Killer #2 – Relying on emergence
I often see embedded software teams adopting Agile principles and methodologies start to believe that the software architecture of a system should emerge naturally from the firmware. Developers rely on the idea that software architecture is emergent and that banging out a bunch of code during their sprints will result in an elegant, scalable, embedded software architecture. Unfortunately, this misinterpretation of how to apply Agile to a software architecture leaves teams believing they don’t need to do any upfront design work. If a team relies on emergence, they will end up right back where they were before! The original problem with many embedded software architecture redesigns is that there was no guiding architecture in the first place. The result was a big ball of mud!
Instead of relying on emergence, the software architecture should be iterated upon and evolve over each sprint. The software architecture isn’t just built upfront and set in stone. That’s a great way to end up with problems as well. Instead, developers, during each sprint, should design and evolve the software architecture around the user stories they are developing. Usually, at the start of the project, the first sprint will be used to get the 30,000-foot view of the software architecture. The details are then filled in as the project progresses.
Software Architecture Killer #3 – Ignoring the data
One of my core embedded software architecture design principles is “Data Dictates Design”. I’ve been mentioning this in many architectures and posts recently because I often encounter teams that don’t let their systems data dictate their architecture! The primary purpose of nearly all embedded systems is to acquire, process, store, and output data. Yet, teams try to architect their system around design patterns or the latest and greatest architectural structures. Unfortunately, you can’t successfully leverage a design pattern or typical architecture until you understand what your systems data is asking of you!
Now the way to solve this problem in your architecture is to examine your systems data. Identify your systems data assets early in product development. Create flowcharts showing how those data assets flow through the system and the operations performed on them. If you follow the data, you’ll discover that the software architecture will emerge. The data will show you what tasks to create. The data will show you where data transfer mechanisms are needed and what design patterns to use. Don’t ignore the data.
(If you want to learn more about how to do this, check out my article series on 5 Steps to Designing an Embedded Software Architecture)
Software Architecture Killer #4 – Keeping the same team
When it comes time to redesign an embedded product, I often don’t recommend keeping the same team for the redesign. If you have a team of engineers who designed and built a product and evolved it into a big ball of mud that is buggy, slow, and has a bunch of problems, you can’t expect to get anything different by letting them do it again. Sure, they’ve learned a lot from their mistakes. They have a lot of experience with the system. But, yes, they’ve been clamoring to redesign from scratch now that they understand what needs to be built. Fool me once; shame on you. Fool me twice; shame on me!
Whenever we redesign a product because of issues with the embedded software, I prefer to keep half the team working on the redesign and use the other half in design reviews and consult on how the product behaves or should behave. Obviously, I’m not saying fire the engineers. The idea is to get fresh eyes and new blood onto the program while not losing the lessons learned and system knowledge. In a small company, this might not be possible. However, I have found that you must carefully select the team for the redesign project, or else you might as well just live with what you have because it won’t get better.
Software Architecture Killer #5 – Ignoring the consultant
Getting outside help and advice can be a great way to ensure that software architecture heads in the right direction. Sometimes engineers and management can be too invested and too close to their products that they can’t see the forest through the trees. A consultant can bring an unbiased opinion, alternative experiences, and expertise not present on the development team.
So far in my career, I’ve worked on more than 100 projects with several dozen companies in a dozen or so countries. Once or twice, I’ve had a company decide to ignore my advice. I’ve had colleagues who are also consultants tell me stories about a customer here or there that didn’t heed their advice either. Interestingly, in all the cases I’ve heard of teams that ignored the consultant, the result was that a year later, they ended up right back where they started, only with a lot less cash in their pockets. I suspect these results are statistically skewed because I don’t think consultants are too keen to share times when they were wrong. Which is too bad because no one gets it right 100% of the time, and there can be great lessons learned through sharing.
If you go down the route of hiring a consultant to help you, listen carefully to their advice. A consultant’s job isn’t to make you happy. It provides an unbiased opinion and advice to help you achieve your goals. At the end of the day, they move on to the next thing while you’re stuck with the decisions you’ve made.
Software Architecture Killer Conclusions
A scalable, flexible, and evolvable software architecture is almost necessary for any modern embedded system. However, there is no guarantee that developing a new embedded software architecture will solve your problems if you aren’t careful. I’ve had the privilege of helping dozens of companies design and evolve their software architectures. These five software architecture killers are the most common issues I see with development teams when it comes to creating a software architecture.
