A Scalable Software Build Accelerator: Break the Build Bottleneck with Faster, More Accurate Builds

A Scalable Software Build Accelerator:
Break the Build Bottleneck with Faster, More Accurate Builds












John Ousterhout Founder and Chairman
John Graham-Cumming Chief Scientist Electric Cloud, Inc. February, 2006



1.0 Executive Summary For organizations that depend on software innovation, a slow software build process can be a bottleneck for the entire company. Slow build times not only impact engineering efficiency, they also affect product quality and company agility. Furthermore, diagnosing build problems is difficult or impossible: cryptic output in Make log files can be hard to decipher and is difficult to relate to the individual build steps in a large build. Electric Cloud's core products, ElectricAccelerator and ElectricInsight, solve these problems by reducing software build times dramatically and providing graphical insight into the performance and structure of builds at a level impossible with existing tools. The solution to the build speed problem is at a first glance simple: create a distributed version of industry standard build tools (such as GNU Make or Microsoft NMAKE) that distributes individual job steps in parallel to a cluster of inexpensive servers. Over the years, many attempts have been made to create "parallel" or "distributed" build systems. However, none are in widespread use due to dependency issues and distributed computing problems that lead to broken builds. Electric Cloud, Inc. has developed an automated dependency management system that makes parallel builds safe, scalable, and efficient. ElectricAccelerator is a software build accelerator that significantly reduces software build times by distributing the build over a large cluster of inexpensive servers. ElectricAccelerator uses its patent-pending dependency management system to identify and fix problems in real time that would break traditional parallel builds. ElectricAccelerator plugs seamlessly into existing Make-based infrastructures, and includes Web-based reporting and management tools. ElectricAccelerator has been proven in some of the most demanding software organizations and against large open-source projects. In one organization, a product took four and a half hours to build on a single system. It now builds 20x faster on a 30-node ElectricAccelerator cluster, finishing in less than 13 minutes.
In another organization, a product took 3 hours and 12 minutes to build on a single system. It now builds 16x faster on a 30-node ElectricAccelerator cluster, finishing in less than 12 minutes.
The open-source Samba file and print server takes 16 minutes to build on a single processor, but builds 16x faster, in 58 seconds, on a 20-node ElectricAccelerator cluster.
The open-source MySQL database takes over 23 minutes to build on a single processor. It builds 12x faster, in 1 minute 54 seconds, on a 20-node ElectricAccelerator cluster.
ElectricAccelerator improves the software development process by reducing build times, so development teams can reduce costs, shorten time-to-market, and improve quality and customer satisfaction. ElectricInsight is Electric Cloud's new build visualization tool. It takes advantage of structural information recorded by ElectricAccelerator and provides a graphical display of this information that makes it easy to understand the behavior of builds, tune performance, and quickly debug broken builds.
A Scalable Software Build Accelerator Page 2 © Electric Cloud, Inc. All rights reserved. V2006.02 2.0 The Real Impact of Slow Builds Every software engineer has experienced the frustration and delay caused by slow software builds. Every VP of Engineering has dreamed of rapid, accurate builds that meet the needs of overextended QA teams, enable rapid innovation to acquire new customers and allow timely turnaround for critical bug fixes. For most organizations such builds are quite simply dreams. Although the widespread Make-based build infrastructure used by many large software organizations has been around for over 20 years, it has lagged behind advanced... [download for more]