Using SCons effectively

(This is currently a work in progress)

There are no great cross-platform build systems. Of all the general-purpose ones, SCons is merely the least bad. I don’t want to re-hash those arguments here. Just trust me.

What do you need to know in order to use SCons effectively? You need to understand a little about the architecture of SCons, and you need to set up and follow some conventions.

In general, you create an Environment (or multiple Environments), populate it with config and tools, and then operate it with Builders and Actions and Aliases. SCons is a functional/declarative system; you declare what you want. It can be somewhat disconcerting at first to work this way, although many other build systems do follow this model, starting with the venerable Make.

Environment

SCons does its best to decouple the build system from the environment of a build machine. This is generally a good thing, otherwise you end up with build environments that are hard to replicate. You can import things from your machine’s environment, but you generally have to do so either explicitly, or through using a Tool.

The SCons build process has the idea of an Environment, which is the collection of things that builders operate on. The subset of the local machine environment that is useful is in the ENV field of the environment. Its structure is somewhat Unix-centric, since SCons started in an Unix environment, but enough of this maps to Windows such that you can almost write generic code.

Platforms

By default, environments are initialized with builders and variables appropriate to the host platform. It is possible to some degree to do cross-platform builds, and create an Environment with a specific platform. This does imply that builders are available on the host platform for that target platform.

SCons knows about the following platforms (as well as  lot of other old or niche platforms that you can find in the source code).

  • ‘cygwin’
  • ‘darwin’
  • ‘posix’
  • ‘win32′

Some cross-platform building is simple – for example, building ‘cygwin’ on a Windows OS, or ‘posix’ on a Mac OS. Other cross-platform building would be extremely challenging, but getting simpler. For example, at some point it might be possible to generate Windows binaries on a Posix platform with Clang, and the vice-versa is nearly true today.

Options and Variables

There are two main ways for a user to pass information to an SCons build operation.

Command-line options are declared with SCons.Script.AddOption, and used with two dashes – e.g., –verbose or –arch=win32. This is essentially the Python optparse module, with a few additions. Command-line options always have default values, even if the default value is None. SCons itself uses AddOption for its options. A handful of options can be set with SetOption() in code, but most options can only be supplied on the command-line.

Variables, or arguments, are passed on the command line as key/value pairs. These are stored in the SCons.Script.ARGUMENTS dictionary, and are accessed as a normal Python dictionary. They are also in the ARGLIST list, in the order they were given on the command line (but without the names, just the values). The ARGLIST approach is also the only way to access multiple occurrences of a variable. You can also use a Variables object to allow these to have help and default values.

Which is the correct approach? Generally, it seems like build scripts use Option instead of Variable. However, Variables are attached to environments, and Option is global, so there is room for both.

Construction variables

While these may be initialized by both arguments and options, and tools and platforms, construction variables are how information is passed to tool invocations when Builders and Actions actually create and run tasks. There is a generalized expansion operation that instantiates construction variables as needed. This also can be the source of some confusion at first, since expansion happens at the point it is needed, and usually not at the point where you declare Builders and Actions.

Tools

The way to use and extend SCons is through tools. Almost every bit of build action is done as a tool. When you create a new environment and do not specific any tools, this is the same as specifying the ‘default’ tool, which automatically adds a set of tools based on the host platform.

Most of your build actions should be expressed as tools, and your SConstruct and SConscripts should simply be declaring all the dependencies and targets.

Your tools typically go into the site_scons/site_tools directory. However, there is a toolpath argument to Environment that can point to additional tools directories, and the toolpath argument is stored in the environment for use by further Clone or Tool calls.

Tools can even be functions, as opposed to files. Tool entries can also have arguments attached to them, which can be useful for tools that need information in order to be initialized properly. For example, the code in a tool might be unable to locate tool binaries, whereas an argument (perhaps from the user on the command-line) might contain that information.

Builders

Builders are somewhat orthogonal to tools. A tool may introduce custom builders, or it may refine builders. Builders turn sources into targets, and through inference (usually with rules about file extensions) either targets or sources can be inferred from the other.

Builders do two things. At parse time, builders are the major factor in building the dependency tree. At build time, builders are invoked when dependencies declare that a target needs to be generated or regenerated. The classic builder is exemplified by StaticObject, which builds a static object file from one or more C, C++, D, or Fortran source files.

If you have one-off operations, then you can use Command to create an ad-hoc builder; this specifies the sources, the targets, and a command-line or Python object used to turn the sources into targets.

Aliases

Closely related to Builders are Aliases. An Alias creates one or more abstract targets that expand to one or more other targets. As well, an action or list of actions can be specified when any of the alias targets are not up-to-date.

The major use of Alias is to create meta targets. For example, a set of related programs that are all used together could be connected by an Alias, and then the user builds the Alias target, instead of each individual target. Quite often, there is no one single top-level target, and electing one of the lower-level targets arbitrarily as “the target” is confusing and prone to long-term error.

Actions

Actions are not often directly used. However, some Actions find great use. PreActions are run before a specified target is build, and PostActions are run after a specified target is built. Both of these let you customize Build actions without needing to create a new Builder, or modify an existing one.

Performance

SCons is slow. The major sources of slowness are the reading and comprehending of the SCons build file (the SConstruct, SConscripts, and related build files), and then the detection of dependencies by looking at every individual file in the build. Even a moderate build with a few tens of thousands of files can take 5-10 seconds to get through the parsing of the scripts, and another 30 seconds in dependency checking.

For the former, scons –interactive might come in handy. This reads and parses all the scripts, then drops the user into a very simple shell. The most common action is “build”, and the user can build specific targets, or the default targets. if multiple builds are done back-to-back without changing the build scripts themselves, this could save some time.

For the latter, the only way to speed up SCons is to delegate the file scanning to some other system, or to rewrite its system. SCons has a cache, but it still hits the filesystem to validate that the cache is good, and this makes the cache less useful. It might be possible to integrate a system like ibb to be an oracle for the portion of the filesystem referenced by SCons. Systems like ibb have code that watches for filesystem updates; they basically cache a subset of the filesystem in RAM, and updates to that cache are proportional to the number of changes to that filesystem.

The actual answer is to rewrite a large part of SCons to allow for a filesystem-watching approach. Systems like ibb and tup are bottom-up, driven by the changes from the filesystem. Systems like make and scons are top-down, driven by the declaration of file dependencies. One answer that would preserve the general power of the top-down approach would be to have filesystem changes result in repair actions to the dependency tree.

And the true answer would be to integrate a build system, a filesystem watcher, and a content-addressable revision control system like git. Files that are committed no longer need watching, and don’t need dependency rediscovery.

To-do

Experiment with custom env.Decider functions to see where the time is being spent. If an scons –interactive build with a custom Decider that just returns false takes no time, then an ibb approach would be of huge benefit.

Use EnsureSConsVersion and EnsurePythonVersion to make sure that users have appropriate versions of both.

Python’s atexit.register() function can be used to cause functions to run at exit.

Investigate Repository() to see if it would be practical to mix scons and git together.

Appendix

Scons documentation

Threads about command-line arguments

Threads about precompiled headers

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>