{"id":32,"date":"2015-02-25T20:21:46","date_gmt":"2015-02-25T20:21:46","guid":{"rendered":"http:\/\/www.msully.net\/blog\/?p=32"},"modified":"2015-02-25T20:21:46","modified_gmt":"2015-02-25T20:21:46","slug":"parallelizing-compiles-without-parallelizing-linking-using-make","status":"publish","type":"post","link":"https:\/\/www.msully.net\/blog\/2015\/02\/25\/parallelizing-compiles-without-parallelizing-linking-using-make\/","title":{"rendered":"Parallelizing compiles without parallelizing linking &#8211; using make"},"content":{"rendered":"<p>I have to build LLVM and Clang a lot for my research. Clang\/LLVM is quite large and takes a long time to build if I don&#8217;t use -j8 or so to parallelize the build; but I also quickly discovered that parallelizing the build didn&#8217;t work either! I work on a laptop with 8gb of RAM and while this can easily handle 8 parallel compiles, 8 parallel links plus Firefox and Emacs and everything else is a one way ticket to swap town.<\/p>\n<p>So I set about finding a way to parallelize the compiles but not the links. Here I am focusing on building an existing project. There are probably nicer ways that someone writing the Makefile could use to make this easier for people or the default, but I haven&#8217;t really thought about that.<\/p>\n<p>My first attempt was the hacky <code>(while ! pgrep ld.bfd.real; do sleep 1; done; killall make ld.bfd.real) &amp; make -j8; sleep 2; make. <\/code> Here we wait until a linker has run, kill make, then rerun make without parallel execution. I expanded this into a more general script:<\/p>\n<div class=\"oembed-gist\"><script src=\"https:\/\/gist.github.com\/3290e688670c54f8d1a2.js\"><\/script><noscript>View the code on <a href=\"https:\/\/gist.github.com\/3290e688670c54f8d1a2\">Gist<\/a>.<\/noscript><\/div>\n<p>This approach is kind of terrible. It&#8217;s really hacky, it has a concurrency bug (that I would fix if the whole thing wasn&#8217;t already so bad), and it slows things down way more than necessary; as soon as one link has started, nothing more is done in parallel.<\/p>\n<p>A better approach is by using locking to make sure only one link command can run at a time. There is a handy command, <code>flock<\/code>, that does just that: it uses a file link to serialize execution of a command. We can just replace the Makefile&#8217;s linker command with a command that calls <code>flock<\/code> and everything will sort itself out. Unfortunately there is no totally standard way for Makefiles to represent how they do linking, so some Makefile source diving becomes necessary. (Many use $(LD); LLVM does not.) With LLVM, the following works: <code>make -j8 'Link=flock \/tmp\/llvm-build $(Compile.Wrapper) $(CXX) $(CXXFLAGS) $(LD.Flags) $(LDFLAGS) $(TargetCommonOpts) $(Strip)'<\/code><\/p>\n<p>That&#8217;s kind of nasty, and we can do a bit better. Many projects use $(CC) and\/or $(CXX) as their underlying linking command; if we override that with something that uses flock then we&#8217;ll wind up serializing compiles as well as links. My hacky solution was to write a wrapper script that scans its arguments for &#8220;-c&#8221;; if it finds a &#8220;-c&#8221; it assumes it is a compile, otherwise it assumes it is a link and uses locking. We can then build LLVM with: <code>make -j8 'CXX=lock-linking \/tmp\/llvm-build-lock clang++'<\/code>.<\/p>\n<div class=\"oembed-gist\"><script src=\"https:\/\/gist.github.com\/d33029fcda6889b7d097.js\"><\/script><noscript>View the code on <a href=\"https:\/\/gist.github.com\/d33029fcda6889b7d097\">Gist<\/a>.<\/noscript><\/div>\n<p>Is there a better way to do this sort of thing?<!--more--><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I have to build LLVM and Clang a lot for my research. Clang\/LLVM is quite large and takes a long time to build if I don&#8217;t use -j8 or so to parallelize the build; but I also quickly discovered that parallelizing the build didn&#8217;t work either! I work on a laptop with 8gb of RAM [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17],"tags":[19,18],"class_list":["post-32","post","type-post","status-publish","format-standard","hentry","category-hacks","tag-llvm","tag-make"],"_links":{"self":[{"href":"https:\/\/www.msully.net\/blog\/wp-json\/wp\/v2\/posts\/32","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.msully.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.msully.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.msully.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.msully.net\/blog\/wp-json\/wp\/v2\/comments?post=32"}],"version-history":[{"count":4,"href":"https:\/\/www.msully.net\/blog\/wp-json\/wp\/v2\/posts\/32\/revisions"}],"predecessor-version":[{"id":36,"href":"https:\/\/www.msully.net\/blog\/wp-json\/wp\/v2\/posts\/32\/revisions\/36"}],"wp:attachment":[{"href":"https:\/\/www.msully.net\/blog\/wp-json\/wp\/v2\/media?parent=32"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.msully.net\/blog\/wp-json\/wp\/v2\/categories?post=32"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.msully.net\/blog\/wp-json\/wp\/v2\/tags?post=32"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}