Parallelizing compiles without parallelizing linking – using make

I have to build LLVM and Clang a lot for my research. Clang/LLVM is quite large and takes a long time to build if I don’t use -j8 or so to parallelize the build; but I also quickly discovered that parallelizing the build didn’t work either! I work on a laptop with 8gb of RAM and while this can easily handle 8 parallel compiles, 8 parallel links plus Firefox and Emacs and everything else is a one way ticket to swap town.

So I set about finding a way to parallelize the compiles but not the links. Here I am focusing on building an existing project. There are probably nicer ways that someone writing the Makefile could use to make this easier for people or the default, but I haven’t really thought about that.

My first attempt was the hacky (while ! pgrep ld.bfd.real; do sleep 1; done; killall make ld.bfd.real) & make -j8; sleep 2; make. Here we wait until a linker has run, kill make, then rerun make without parallel execution. I expanded this into a more general script:

This approach is kind of terrible. It’s really hacky, it has a concurrency bug (that I would fix if the whole thing wasn’t already so bad), and it slows things down way more than necessary; as soon as one link has started, nothing more is done in parallel.

A better approach is by using locking to make sure only one link command can run at a time. There is a handy command, flock, that does just that: it uses a file link to serialize execution of a command. We can just replace the Makefile’s linker command with a command that calls flock and everything will sort itself out. Unfortunately there is no totally standard way for Makefiles to represent how they do linking, so some Makefile source diving becomes necessary. (Many use $(LD); LLVM does not.) With LLVM, the following works: make -j8 'Link=flock /tmp/llvm-build $(Compile.Wrapper) $(CXX) $(CXXFLAGS) $(LD.Flags) $(LDFLAGS) $(TargetCommonOpts) $(Strip)'

That’s kind of nasty, and we can do a bit better. Many projects use $(CC) and/or $(CXX) as their underlying linking command; if we override that with something that uses flock then we’ll wind up serializing compiles as well as links. My hacky solution was to write a wrapper script that scans its arguments for “-c”; if it finds a “-c” it assumes it is a compile, otherwise it assumes it is a link and uses locking. We can then build LLVM with: make -j8 'CXX=lock-linking /tmp/llvm-build-lock clang++'.

Is there a better way to do this sort of thing?

3 thoughts on “Parallelizing compiles without parallelizing linking – using make

  1. maurer

    I have no swap and only 8GB of RAM.

    cmake + ninja allows me to peg 4 cores building LLVM without trouble (at least as far as I can tell), and the build completes.
    I haven’t monitored it to make sure it keeps them always pegged, but it seems to work well for me.

  2. Steve Fink

    If you don’t mind messing up your binutils installation, you could replace your ld.bfd with a shell script that does the locking. And given that your ld is named ld.bfd.real, maybe you’re already doing something funky there? ld.bfd -> ld.bfd.real -> ld.bfd.really.real? (Or is ld.bfd.real an osx thing?)

    All of these solutions allow multiple compiles alongside a link, but I guess that’s not a huge deal.

  3. sully Post author

    Yeah, I considered replacing the ld but didn’t want to *always* restrict to one link, just when building certain projects. Not on OS X; Ubuntu 14.10. Apparently ld.bfd is a symlink to hardened-fd, which is a perl script that “enforces hardening toolchain improvements”. Looks like a Debian thing. I’m not really sure.

Leave a Reply

Your email address will not be published. Required fields are marked *