Redesigning GHC’s build system

Over recent months we’ve been migrating more and more of GHC’s build system from make (+GNU extensions) into Cabal.  It seemed like the obvious thing to do: Cabal is a generic build and packaging system for Haskell libraries, and GHC’s build system was duplicating much of the functionality of Cabal.  What’s more, many libraries needed two sets of metadata to describe them: one set for GHC’s build system, and another set for Cabal, so that users could build and install updated versions of these libraries outside of the GHC build.

So in changing GHC’s build system to use Cabal under the hood, we eliminated some duplicated effort, which is always a good thing.  We got to piggyback off the Cabal project, which is ticking along nice thanks to the efforts of Duncan Coutts and many other contributors.

But it wasn’t perfect, even from the outset.  We knew we wouldn’t be able to use Cabal as it stands.  For one thing, Cabal uses ghc --make,  which doesn’t build in parallel, and we really want parallel builds (building GHC can take a while).  Secondly, when working on GHC we often want to build individual modules, sometimes with extra flags, just to try something out.  Cabal doesn’t support this, it can only build the whole package.  Thirdly, the GHC build system has a lot of configuration information that we need to propagate into Cabal somehow.  So the solution we came up with for all this was to have Cabal generate a Makefile from the package metadata, so that we could build the package using make.  Make knows how to build in parallel, it knows how to make just a single target, and it can import configuration information from other makefiles.

This was a cute hack (or an ugly one, depending on your point of view), but it certainly has its downsides: for one thing, it’s really hard to modify the code that generates the Makefile, as it’s part of Cabal.  For someone new to GHC, it’s even hard to find the code that generates the Makefile.  The build system is getting less maleable.  We hadn’t really fixed the duplication problem either, since we’d had to write another build system, in the form of an auto-generated Makefile.  Users began to complain.

Using Cabal is still morally the right thing to do.  It’s just that the marriage of Cabal with our existing Makefile-based build system isn’t a happy one.  If Cabal becomes a better make than make, we will be able to use it more effectively, but for now we have decided to back off and redesign the build system in the light of the recent problems we’ve had.

GHC’s build system has always been something of a struggle to use.  Building GHC is a 2-stage bootstrap process, where we first build the “stage 1” compiler, use that to build a bunch of libraries, and then use stage 1 + libraries to build stage 2, which is the compiler that we actually install.  There are a lot of dependencies between different parts of the tree, most of which are not expressed explicitly in the build system.  Partly this is because we use recursive make, which as many will know is considered to be harmful.  The result is that we often need to “make clean” to fix inconsistencies caused by changing parts of the tree without rebuilding enough, or simply by pulling the latest changes into your tree.  We actually recommend doing “make distclean” before pulling new changes, which is obviously not ideal.

So we’re going to try the non-recursive make approach, coupled with a very light use of Cabal for preprocessing the library metadata into Makefile bindings (that way we don’t have to keep the metadata in two forms).  There’s a design sketch for the new build system on the wiki.  Over the next few weeks we’ll be working on the new build system in a repository branch.  Watch this space!

This entry was posted in Uncategorized. Bookmark the permalink.

One Response to Redesigning GHC’s build system

  1. Pingback: magnus » Blog Archive » Re: Redesigning GHC’s build system

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s