Arduino mass-builder

The script supports building given list of sketches for a given set of boards. The builds can happen multiple times (for different compilers, settings, etc.) producing multiple buildsets.

Furthermore, the script can output the build results, including comparisons between buildsets, in CSV format that can be processed using e.g., Google Fusion Tables, Excel, etc.

Building

To build sketches, run the build subcommand. Pass the .ino filename (just one per sketch for sketches with multiple .ino files) of each sketch as a commandline argument.

By default, it builds for the Arduino uno. Use the --boards option to specify other boards.

For example:

arduino-mass-builder.py build --boards arduino:avr:uno,arduino:avr:leonardo \
	01.Basics/BareMinimum/BareMinimum.ino 01.Basics/Blink/Blink.ino

Build results are placed in the result directory by default, pass --results-dir to put them somewhere else.

You can run this command mutiple times with different sketches or boards and the results will just be merged together.

Add -v or --verbose before the build command to enable verbose mode.

See arduino-mass-builder.py --help and arduino-mass-builder.py build --help for more options.

Creating a report

To create a report (CSV file) from existing build results, run the report subcommand.

For example:

arduino-mass-builder.py report

Build results are read from the result directory by default, pass --results-dir to get them somewhere else. The CSV file is also output into that directory, in the report subdirectory.

The CSV generated by the build example above looks like this (extra whitespace added for display here):

Buildset , Sketch                , Board                , Status , Program size , Data size
base     , 01.Basics/BareMinimum , arduino:avr:leonardo , OK     , 4262         , 157
base     , 01.Basics/BareMinimum , arduino:avr:uno      , OK     , 450          , 9
base     , 01.Basics/Blink       , arduino:avr:leonardo , OK     , 4798         , 157
base     , 01.Basics/Blink       , arduino:avr:uno      , OK     , 1030         , 9

See arduino-mass-builder.py --help and arduino-mass-builder.py report --help for more options.

Comparing multiple buildsets

Sometimes, you might want to build the same set of sketches, for the same set of boards multiple times, but with changes in the Arduino IDE or core files applied. To supports this, the script can distinguish between multiple "buildsets".

You can specify which buildset a build belongs to by using the --buildset option. If not specified, the buildset name is set to base.

For example, to find out the effects of a different compile flags, we modified platform.txt to use -O3 instead of -Os. Then, we ran another buildset:

arduino-mass-builder.py build --buildset o3 --boards arduino:avr:uno,arduino:avr:leonardo \
	01.Basics/BareMinimum/BareMinimum.ino 01.Basics/Blink/Blink.ino

After regenerating the report:

arduino-mass-builder.py report

The CVS now looks like this. Note that there are now delta columns in there. Also note that everything compares with the base buildset by default, this can be changed using the --base-set option.

Buildset , Sketch                , Board                , Status , Program size , Data size , Δ status , Δ program size , Δ data size , In base buildset
base     , 01.Basics/BareMinimum , arduino:avr:leonardo , OK     , 4262         , 157       , Is base  , 0              , 0           , Yes
base     , 01.Basics/BareMinimum , arduino:avr:uno      , OK     , 450          , 9         , Is base  , 0              , 0           , Yes
base     , 01.Basics/Blink       , arduino:avr:leonardo , OK     , 4798         , 157       , Is base  , 0              , 0           , Yes
base     , 01.Basics/Blink       , arduino:avr:uno      , OK     , 1030         , 9         , Is base  , 0              , 0           , Yes
o3       , 01.Basics/BareMinimum , arduino:avr:leonardo , OK     , 5052         , 157       , Modified , 790            , 0           , No
o3       , 01.Basics/BareMinimum , arduino:avr:uno      , OK     , 460          , 9         , Modified , 10             , 0           , No
o3       , 01.Basics/Blink       , arduino:avr:leonardo , OK     , 5626         , 157       , Modified , 828            , 0           , No
o3       , 01.Basics/Blink       , arduino:avr:uno      , OK     , 1146         , 9         , Modified , 116            , 0           , No

This shows that adding -O3 changes the resulting binaries ("Modified"), and increases the program size (increase is a lot bigger for the leonardo), but doesn't affect the RAM size.

Visualizing results

To visualize the results, the data can be imported into Google Fusion Tables, or plain Excel. As an example for Fusion Tables, see this document. To re-use the same graphs, you can best make a copy of the document, clear out the data (Edit -> Delete all rows) and import your own (File -> Import more rows).

License

This script and accompanying docs are written by Matthijs Kooijman, and are made available under the MIT license.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.