SCons Inside Out

Introduction

SCons follows the model established by Ellen Borison in 1986 in the paper “A Model of Software Manufacture”. There are two phases: defining a DAG of the desired build, and using the DAG to bring the build up to date. In fact, as of SCons 0.9.6, you can run scons --interactive to separate the two phases, and potentially do multiple builds from the same DAG.

In phase 1, the SConstruct and SConscript files are read and parsed, generating a DAG of the entire build. The developer uses SCons API calls in SConstruct and SConscript files to define the pieces of the dependency graph. At the end of phase 1, you have a DAG of your build that is then consumed by phase 2.

In phase 2, the DAG is evaluated, and any out-of-date nodes result in the generation of jobs that are scheduled and executed. SCons can schedule them in a deterministic order or a semi-random order (a random order that does not violate the partial order as defined by the DAG). At the end of phase 2, you have an up-to-date build.

Phase 1 looks like functional programming, because you are declaring what should happen, not when. With the exception of Python code itself, all the SCons APIs do is to allow you to declare the parts of the dependency graph, including the actions that will be taken when parts of the graph are out of date. The actual actions happen in the engine, which starts operation once the dependency graph has been assembled. And, if the entire build is up to date, no actions take place.

Quick start

We’re going to follow the canonical naming that SCons introduced. You will have a top-level file called SConstruct that is used to invoke builds, you’ll have a folder next to it named site_scons that contains all your custom code, and then you’ll have one or more SConscript files that list the various physical targets and dependencies.

A prototypical big-project multi-platform SCons layout looks like this.

my_big_project/
  SConstruct
  site_scons/
    site_tools/
      my_big_project.py
  [source files]

A skeleton for a big-project SConstruct

#--------------------------------------------------------------------
# My big project
#--------------------------------------------------------------------

import SCons

#--------------------------------------------------------------------

# Create base environment. The host-platform option is largely for
# script debugging, you really can't get anywhere with cross-platform
# builds with current SCons. This has to be done before we create the
# Environment.

# allow host platform to be switched
SCons.Script.AddOption(
    '--host-platform',
    dest='host-platform',
    action='store',
    default=None,
    help='Specify a host platform: darwin|posix|win32.'
         'The only use of this currently in debugging build scripts')

# Create our base project-building environment
host_platform = SCons.Script.GetOption('host-platform')
env = Environment(platform=host_platform, tools=['my_big_project'])

#--------------------------------------------------------------------

# invoke subsidiary build scripts
SConscript('my_path_to/SConscript', 'env')

In current SCons, platform is either auto-detected, or supplied in an Environment() call. It cannot be changed afterwords. Since typically we create a single base Environment call per build, this means that if we want to set the platform to something specific, we have to do it in the SConstruct.

Note that setting platform in current SCons (as of 2.3.0) won’t let you do cross-platform builds. However, it will let you do a fair amount of sanity checking as you are working on a single platform in a multi-platform build.

Environment

The construction environment, created with Environment(), is a dictionary holding state information needed by builders and other parts of the SCons system. It contains construction variables, builder objects, scanner objects, and even custom information as needed by builders or scanners. While it has something in common with the idea of an environment as first made popular by Unix, there is also not one global environment, but as many as the build needs to declare.

Environments are built up by actions in the SConstruct and SConscript files, and by tools. There is no history, changes to environment variables replace the previous values. But, more importantly, environments are used by builders in phase 2, which means that if you have environment variables that need specific and conflicting values for multiple parts of the build, you need multiple environments, one for each subset of the build.

To this aim, you can do lightweight copies of environments through a clone() method. A typical pattern in a large build is to inherit a base environment, update it as needed, and then use that for the commands for that section of the build.

SCons Cookbook

Caching with CacheDir

SCons has a caching mechanism for objects. Environment has a method called CacheDir() that you call to set a cache directory for your build. If there is a cache, then SCons will check to see if a derived file already exists in the cache (by looking for its MD5 build signature); if so, the derived file will just be copied out of the cache, skipping the need for that specific build step. You’d have a line like this in your SConstruct:

env.CacheDir('path/to/cache')

and then your builds will populate and use this cache. This can be very useful if you often switch between branches where large parts of the code need to be rebuilt as part of that switching.

The current cache uses the filesystem as a lookup mechanism – individual derived files are stored with their MD5 signature names. This becomes impractical once you get to tens of thousands of files or up, and the cache actually becomes a bottleneck, at least on magnetic media.

SCons Tools

SCons has core tools implemented as Python modules. The suggestion is that contributed tools are distributed as packages (a directory with an __init__.py file in it); this is so that these directories can each be under a DVCS system—it’s far easier to have a directory under source control than to have individual files under source control.

http://www.scons.org/wiki/ToolsIndex

Reference

Chapter 4 of Empirical Comparison of SCons and GNU Make contain a brief description of the design and implementation of SCons. It is one of the few third-party write-ups on SCons.

SCons really is Cons written in Python, or it least it started out that way. Cons had a construction environment, a top-level Construct file, and subsidiary Conscript files; it had the notion of using cryptographic signatures instead of timestamps to detect out-of-date dependencies; it even had a .consign file to cache information. However, SCons is very object-oriented compared to Cons, and has diverged over the past 10 years from the Cons design in some areas.

SCons started out with the name sccons, in early drafts of the proposal for the Software Carpentry Build contest, and after dropping out of the Software Carpentry contest became called just SCons. Parts of the design remain from the first drafts (Environment, Scanner objects, Builder objects), whereas other components were either dropped (Intercessor as the interface to external objects, the idea of an embeddable Build Engine, the idea of a wrapper to allow backwards compatibility with Makefiles) or added (***), or mutated (Signature class).

SCons actually owes its interpolation syntax to Perl – the $var or ${var} are pure Perl syntax for variable interpolation, something that Python is lacking but that SCons brought along from Cons along with the overall design.

Documentation

SCons man page – current released version is 2.3.1. The man page is probably the single best piece of documentation on SCons.

SCons user guide. This is somewhat complementary to the man page.

SCons Wiki. This has useful snippets of code on it. It also has a FAQ that’s worth reading.

Bibliography

SCons: cleaning variant_dir

Some Things Are Obvious