Thursday, August 7, 2008 at 4:10 PM
For a while now, Google has been using distcc, a distributed C/C++ compilation system, to speed up building software made of millions of lines of code. With distcc, we can build code an order of magnitude faster than we could if everyone had to compile on their own workstation. But even with distcc, compiles could take a long time: compiling the Google Webserver might take 20 minutes. We started looking at distcc to see if we could make it even faster.
We're proud to report that we've succeeded: we've developed an algorithm we call "pump mode", which can be added to distcc to speed it up by a factor of 3. Pump mode works by pushing even more processing onto the servers. Based on an incremental static analysis of the source code, pump mode is able to quickly identify the sets of files needed for the preprocessing phase of compiling C/C++ programs and send them to the compilation servers for preprocessing. This achieves a dramatic decrease in the CPU load of the workstation and of course much better build speed. We have tested pump mode on some open source software and seen improvements in build speed between 50% (the Linux kernel) and 200% (Samba). With simple changes to the project Makefiles, most projects we have looked at would be even faster!
The pump mode extension has been Google's main C/C++ build system for over a year now.
Distcc's pump mode was developed by a small team at Google that included myself, Manos Renieris, Fergus Henderson, and Craig Silverstein. The pump mode extension complements the recently released open source gold linker, which addresses the other basic bottleneck for fast building of C/C++ software.
Distcc's pump mode is included in release 3.0 of distcc. This is the first release since 2004 when Martin Pool, the original author of the code base, released version 2.18.3. Distcc 3.0 contains many other contributions from a variety of contributors, including Avahi Zeroconf support by Lennart Poettering, "lsdistcc" by Dan Kegel, and bug fixes and portability improvements by Nadim Khemir, Maks Verver, Niklaus Giger, Sascha Demetrio, Alex Besogonov, Ben Skeggs, Lisa Seelye, Lei Zhang, Michael Moss, Dongmin Zhang, and others. Disctcc is now maintained by Fergus Henderson.
7 comments:
All sounds good, but perhaps some (linked?) performance comparisons with the more popular alternative distributed-compilation software - rather than just the prior version of the same software? A few more facts would help support the article: frankly, the incremental preprocessing argument sounds a bit weak - preprocessing is already blindingly fast compared to even unoptimised compilation. Given the claimed performance improvement, surely there's more to this story? Or maybe my instincts are all wrong... *shrug*.
That is awesome news!
I can't wait to try the pump mode.
I have my own Avahi Zeroconf distcc (improving Lennart Poettering's version) that I'll try to port to this new version.
Hopefully, I'll submit a patch for those interested soon enough.
What a cool tool! Now if only I had a project which could take advantage of it....
Very nice. Anyone happen to test Qt 4.4? With WebKit it can take quiet a long time to compile.
tony_in_da_uk said...
All sounds good, but perhaps some (linked?) performance comparisons with the more popular alternative distributed-compilation software - rather than just the prior version of the same software?
Is there anything specific that you would like us to compare against?
An older comparison of distcc 2 against other alternatives is available at
http://distcc.googlecode.com/svn/trunk/doc/web/compared.html
A few more facts would help support the article: frankly, the incremental preprocessing argument sounds a bit weak - preprocessing is already blindingly fast compared to even unoptimised compilation. Given the claimed performance improvement, surely there's more to this story? Or maybe my instincts are all wrong... *shrug*.
It's certainly surprising that preprocessing can become the bottleneck.
But that turns out to be the case for many large builds, once you've distributed the compilation.
For Google's own builds, we found that 70% of the CPU time on the local machine was spent doing preprocessing!
The reason this happens is that compilations can be parallelized, so
if you a sufficient number of distcc servers, compilations don't contribute significantly to the overall latency. The bottleneck becomes the steps that must be done locally, such as preprocessing and linking.
For comparable systems, the only thing that springs immediately to mind is vesta (www.vestasys.org)
@sun we have been using dmake for years, and wouldn't dream of compiling opensolaris without it.
-rahul
Post a Comment