Open Source versus Commercial Compilers

Using open source and free software seems to be engrained in the DNA of software developers. Given a choice to pay or build a developer will usually choose build. When it comes to compilers, building their own compiler would take far too long and be very error prone so developers choose open source tools such as GCC to save the few thousand dollars’ investment in a commercial compiler. The justification is usually that they do the same thing so why pay the money? The interesting truth is that commercial compilers offer developers many benefits the most important being performance improvements.

It’s easy to imagine that one compiler is just as good as any other. After all, aren’t all those optimization algorithms that are used to compile C code been well documented and published for almost 50 years now? There really hasn’t been a good way for developers to compare one compiler to another as far as how well the compiled code runs on a target microcontroller. In the last few years a new benchmark was released by EEMBC known as the CoreMark which allows developers to put their microcontrollers through performance testing and see how well they do. The cool thing about it is that we can also compile the benchmark code on different compilers for the same microcontroller and see what the performance difference is in the generated code.

Before diving into the interesting results, it is useful to understand a little bit about the CoreMark. The CoreMark is interesting to low level developers due to the fact that it is written in C and designed to test a cores ability to process common embedded software tasks such as processing lists, state machines and matrices. The benchmarking software can be ported to any microcontroller or processor and once executed provides a measure in CoreMarks and CoreMarks per MHz. In general, the higher the result, the better performing the processor was.

Developers don’t necessarily need to perform the benchmarking themselves but instead could check the CoreMark website to see if their processor (and compiler) are listed (check here). Over 500 different benchmarks have been submitted although duplicates do exist, sometimes with different or similar toolchains. Looking through the results, developers could pick through and find a few microcontrollers that have been benchmarked using various compilers. Below are a few examples:

01-CoreMark

As you can see from the image, the exact same processor running the exact same code compiled by different compilers results in completely different performance results. Unfortunately, there aren’t many benchmarks in the list that compare GCC to commercial compilers. The ones that do exist are for application processors which are outside the range for microcontrollers which are more ARM Cortex-M class. As it turns out, the Renesas Synergy™ Platforms SSP datasheet has an interesting comparison using CoreMark. They performed the benchmark between their open source GCC toolchain and their IAR Embedded Workbench for Synergy toolchain. The results can be seen below (The table is taken from chapter 15 in the datasheet located at https://www.renesas.com/en-eu/doc/products/renesas-synergy/doc/r01ds0272eu0111_synergy_ssp.pdf):

01-Renesas

After reading through the results it becomes very obvious: All compilers are NOT the same. In fact, commercial compilers tend to outperform GCC by as much as 50%! With a commercial compiler, developers could literally select a 48 MHz part over a 72 MHz part and get the exact same performance while saving hardware costs. The next time you plan to default to GCC, you may want to stop and do the math to figure out if a commercial compiler might just save your company money in the long run.

If you are interested in learning more about CoreMark and the resulting differences in performance between different compilers I highly recommend you check out the CoreMark white paper located at http://www.eembc.org/techlit/coremark-whitepaper.pdf. The results section is particularly illuminating.

Leave a Reply

Your email address will not be published. Required fields are marked *