: linux : distcc
 photo gallery
 the whisp's pedigree
 script feh
 Hikaru's kernel goodies
 slackware mirrors
 slackware packages
powered by slackware linux
distributed C/C++ compiles with caching.. yummy [ 2003 Jan 03 ]
WTF's this all about? Head off to the distcc & ccache homepages & rtfm for a bit. What follows here are just the random scribbles of a madman who wants stuff to build faster.

the build farm^Wgarden plot
Hardware (powerful, huh?):
    slackware 13.37 machines
  • helium: Intel Atom D525 1.8GHz, 2GB ram, kernel
  • lithium: Intel Atom N450 1.66GHz, 2GB ram, kernel

Compiler related software:

First, some obligatory disambiguation. The "client" machine is the one where a compile is originated - it has the sources, build script (if any), & pretty much the full D series set of packages. The "server" machine(s) are simply along for the ride - no sources, no build script, & only a specific subset of D series packages are required.

On the client machine: binutils, ccache, distcc, gcc, gcc-g++, & pretty much everything else you'd expect from the D series.

On the server machine(s): binutils, distcc, gcc, & gcc-g++. That's it.

Before you start:

If you haven't done so already, I'd suggest you read distcc security design right about now. Since my setup is deployed on a trusted lan, I've opted for simplicity: tcp transport, no distcc user, no use of lzo, etc.

Keeping that thought firmly in mind, here's my setup.

distccd configuration (servers):

Simplicity has its place. This is one of them.

Depending on your distribution, distcc may not be listed in /etc/services on your machines. No worries, simply invoke your favourite editor & add the following lines:

    distcc          3632/tcp   #distributed C/C++ compiler system
    distcc          3632/udp   #distributed C/C++ compiler system
(Why is there an entry for udp? I'm feeling IANA-ish today.)

Next up, running the server. Since we live in a wonderful world of choices, you get your pick of running distccd as a standalone daemon or via inetd.

For standalone usage, here's a sample rc.distccd. You'll likely want to edit the "--allow" & "--listen" lines to suit your setup.

If you prefer inetd, add something like this to /etc/inetd.conf.

    distcc          stream  tcp     nowait  nobody  /usr/sbin/tcpd
        /usr/bin/distccd --inetd --log-file=/var/log/distccd
(Note that the above is actually one line. I've split it purely for presentational purposes. Don't forget to "killall -HUP inetd")

Lastly, /var/log/distccd

    touch /var/log/distccd
    chown nobody:nogroup /var/log/distccd

distcc configuration (client):

Starting with version 2.0, distcc now supports masquerade mode which is described quite nicely in the distcc manpage.

For the client machine, I did the following:

    mkdir -p /usr/lib/distcc/bin/
    ln -s ../../../bin/distcc /usr/lib/distcc/bin/c++
    ln -s ../../../bin/distcc /usr/lib/distcc/bin/cc
    ln -s ../../../bin/distcc /usr/lib/distcc/bin/g++
    ln -s ../../../bin/distcc /usr/lib/distcc/bin/g++-gcc-4.5.2
    ln -s ../../../bin/distcc /usr/lib/distcc/bin/gcc
    ln -s ../../../bin/distcc /usr/lib/distcc/bin/gcc-4.5.2
    ln -s ../../../bin/distcc /usr/lib/distcc/bin/i486-slackware-linux-c++
    ln -s ../../../bin/distcc /usr/lib/distcc/bin/i486-slackware-linux-g++
    ln -s ../../../bin/distcc /usr/lib/distcc/bin/i486-slackware-linux-gcc
    ln -s ../../../bin/distcc /usr/lib/distcc/bin/i486-slackware-linux-gcc-4.5.2
If you'd rather not figure out all the symlinks to create, grab the mkdistccsymlinks script & run it.
ccache configuration (client):

Conveniently, ccache also supports masquerade mode which is briefly described in the ccache manpage.

For the client machine, I did the following:

    mkdir -p /usr/lib/ccache/bin/
    ln -s ../../../bin/ccache /usr/lib/ccache/bin/c++
    ln -s ../../../bin/ccache /usr/lib/ccache/bin/cc
    ln -s ../../../bin/ccache /usr/lib/ccache/bin/g++
    ln -s ../../../bin/ccache /usr/lib/ccache/bin/g++-gcc-4.5.2
    ln -s ../../../bin/ccache /usr/lib/ccache/bin/gcc
    ln -s ../../../bin/ccache /usr/lib/ccache/bin/gcc-4.5.2
    ln -s ../../../bin/ccache /usr/lib/ccache/bin/i486-slackware-linux-c++
    ln -s ../../../bin/ccache /usr/lib/ccache/bin/i486-slackware-linux-g++
    ln -s ../../../bin/ccache /usr/lib/ccache/bin/i486-slackware-linux-gcc
    ln -s ../../../bin/ccache /usr/lib/ccache/bin/i486-slackware-linux-gcc-4.5.2
If you'd rather not figure out all the symlinks to create, grab the mkccachesymlinks script & run it.

You'll likely want to set limits on the cache at some point, so read up on the -F (maxfiles) & -M (maxsize) options.

Putting it all together:

Depending on what I'm building &/or the mood I'm in, I'll source one of the following scripts:

  • c+ (enable ccache)
  • c- (disable ccache)
  • d+ (enable distcc)
  • d- (disable distcc)
  • dc+ (enable distcc & ccache)
  • dc- (disable distcc & ccache)
To keep confusion to a minimum (ie: I edit one script, but forget to edit the others), d+ & dc+ require a separate host list called, oddly enough, "hostlist".

Assuming, for example, you keep these files in ~/.distcc/ (that's where I keep mine.. easy to find) & you want to enable both distcc & ccache, just issue:

    . ~/.distcc/dc+
(that's "<dot><space>~/.distcc/dc+", btw) then start the build as you normally would. No Makefile hacks, no manual twiddling of environment variables - just pure distcc & ccache goodness.

Big honking note:

Those of you actually paying attention will notice the use of "$MAKEJOBS" rather than "$MAKEFLAGS" in d+ & dc+. This is quite deliberate. I found through trial & error (& error & error) that some projects toss errors if all of

    ./configure ; make ; make install
are parallelized. By using an environment variable that make doesn't reference & changing the above steps to be

    ./configure ; make $MAKEJOBS ; make install
said projects behave as they should.
Copyright © 1999-2018 Erik Jan Tromp