Category Archives: Build systems

SCons documentation links

Unlike for other build systems, there is no single source of truth for SCons design and architecture, and thus for use.

SCons documentation

Documentation page at scons.org points to the pieces of the SCons documentation.

man page, Steven Knight and Anthony Roach, last updated March 2013. While technically a man page, this is also large and comprehensive, and is probably the single best existing piece of documentation on SCons.

User Guide, 2.3.0. This has useful and complementary information, but is not as complete as the man page.

API, 2.3.0. This is quickly becoming dangerous to use. This is generated automatically from comments in the source, and the comments in the source are often old. This is more useful as a quick-start before working on the SCons source than as a source of information about SCons. And even then, it’s probably more useful to just dive into the source code.

SCons source on bitbucket, and SCons Development practices on scons.org. Sometimes you just have to go to the source.

FAQ, from the SCons Wiki.

Papers

sccons Design Overview, Steven Knight, June 2000 (rescued via the Internet Archive). This was the second round draft of the Software Carpentry competition to create a modern build tool.

SCons Design version 0.91, Steven Knight, 2001. This was the last draft of the original SCons design, but is quickly getting out of date.

SCons Design and Implementation, Steven Knight, 2002 (paper presented at the Python 10 conference in 2002).

Empirical Comparison of SCons and GNU Make, Ludwig Hähne, 2008. This is a re-examination of theory and practice for make and SCons.

Related

Cons – A Software Construction System, FSF, 1996-2000. Cons is the predecessor to SCons, written by Bob Siddenbotham in 1991 in Perl, and first released in 1996.

Popular, Fast, or Usable: Pick One, 2010, an article somewhat unhappy with SCons performance.

Make Alternatives, Adrian Neagu, July 2005. A great overview to what was the state of the art in 2005. SCons comes out favorably by his criteria.

The Quest for the Perfect Build System, Noel Llopsis, 2005. Another great overview from 2005 (what was it about that year?). Here’s a working link to his generate_libs program: https://github.com/greatwolf/generate_libs.

Four Interesting Build Tools, Neil Mitchell, 2012. This is less comprehensive but far more up to date, and covers Redo, Ninja, Tup and Fabricate.

Wikipedia articles on SCons, Waf (an SCons derivative used by KDE for a few years until they switched to CMake), CMake (somewhat different direction than SCons).

waf is a rewrite of SCons by Thomas Nagy, obtained initial purchase from its use by KDE, who has since abandoned it in favor of CMake.

Tup, written by Mike Shal in C, is interesting and its techniques should be merged with SCons. Build System Rules and Algorithms is a 2009 paper describing the techniques used to make Tup. If we graded build systems by the combination of how funny and how analytical their authors are, Mike Shal would win.

ibb, written by Chad Austin in Python, is something similar, and its author even wrote Scalable Build Systems: An Analysis of Tup to compare it to ibb. See Your Version Control and Build Systems Don’t Scale for the main post about ibb.

redo is a new competitor to make, inspired by D.J. Bernstein. Purely top-down software rebuilding is a software thesis written by Alan Grosskurth in 2007 on redo.

Tundra is also something to look at – source is on Github at deplinenoise/tundra.

Shake, written by Neil Mitchell in Haskell, had several papers and videos released in 2012 describing it.

GBS, git-build-system, is a Tizen package build tool that builds from Git repositories.

gyp was written by Google to be Chromium’s build system. It takes an approach similar to CMake in that it generates project files for popular build environments instead of being the build environment as well as the build tool – e.g. it can generate Visual Studio or Xcode projects, as well as SCons or Make build files. The authors are Mark Mentovai and Steven Knight, although Steven Knight is not in the Members list for the project.

According to a Reddit thread on Tup, Steven Knight was hired to Google, the Chromium team used SCons for a bit, then Google switched to gyp (written by Steven Knight?) and subsequently there has been almost no progress on SCons itself.

SCons Environment in depth, part 2

This is part 2 of an exploration of the SCons Environment() system. We’re going to talk about ENV (the subset of environment variables for our Environment), Tools and Builders. For other parts in the series, see:

ENV

Continuing the exploration of Environment() – so, what about the ENV portion of an SCons Environment? What relationship does that have with the platform settings, discussed at length in part 1?

It appears that it has no relationship, or at least no visible one, when restricted to just defining platform (mandatory) and importing from the external environment. Let’s extend our test SConstruct as follows.

import SCons

def log(name, env, prompt):
  logfile = open(name, 'w')
  print >> logfile, prompt
  print >> logfile, env.Dump()
  logfile.close

for plat in ['win32', 'darwin', 'posix', 'cygwin']:
  env = Environment(platform=plat, tools=[], BUILDERS={}, ENV={})
  log('no_env_%s.txt' % plat, env,
      "env = Environment(ENV = {}, platform=%s...)" % plat)

for plat in ['win32', 'darwin', 'posix', 'cygwin']:
  env = Environment(platform=plat, tools=[], BUILDERS={})
  log('env_%s.txt' % plat, env,
  "env = Environment(platform=%s...)" % plat)

When we run this and compare files, the only thing that allowing the environment to populate the ENV field does is – we populate the ENV field. Seems like a letdown.

platform = ‘win32′ when run on a Windows machine.

  'ENV': { 'COMSPEC': 'C:\\windows\\system32\\cmd.exe',
           'PATH': u'C:\\windows\\System32',
           'PATHEXT': '.COM;.EXE;.BAT;.CMD',
           'SystemDrive': 'C:',
           'SystemRoot': 'C:\\windows',
           'TEMP': 'C:\\Users\\BFitz\\AppData\\Local\\Temp',
           'TMP': 'C:\\Users\\BFitz\\AppData\\Local\\Temp'},

PATH and PATHEXT are artificial, and do not contain the contents of the native environment. COMSPEC (listed as ComSpec in the Windows environment output), SystemDrive, SystemRoot, TEMP and TMP are copies of their respective environment values, and are somewhat required for proper operation of binaries.

platform = ‘darwin’ when run on a Windows machine is completely artificial.

  'ENV': { 'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin'},

platform = ‘posix’ when run on a Linux machine is spartan, and actually artificial (this is not what PATH is set to on this machine).

  'ENV': { 'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin'},

platform = ‘darwin’ when run on a Mac OS X machine is still pretty spartan.

  'ENV': { 'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin',
           'PATHOSX': '/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin'},

PATH is artificial, whereas PATHOSX is a copy of the PATH environment variable. Controlling PATH is important, otherwise build consistency can be hard to achieve between machines. But it’s tricky, because you might fail to find things.

So, for some reason, SCons sets up a PATH environment variable that is made up.

All said, there’s little need or effect for environment settings from the machine.

Unanswered questions to date – I see items being added to ENV as part of platform setup, but don’t see them showing up in the Environment itself.

Interaction between ENV and builders

We’re going to jump ahead a little bit. When we start adding builders, those builders cause items to be added to ENV.

‘win32′ platform with default BUILDERS adds a lot of entries to PATH, and also adds INCLUDE, LIB and LIBPATH environment variables.

  'ENV': { 'COMSPEC': 'C:\\windows\\system32\\cmd.exe',
           'INCLUDE': ..., # 5 entries
           'LIB': ..., # 3 entries for amd64 and x64
           'LIBPATH': ..., # 6 entries
           'PATH': ..., # 16 entries
           'PATHEXT': '.COM;.EXE;.BAT;.CMD',
           'SystemDrive': 'C:',
           'SystemRoot': 'C:\\windows',
           'TEMP': 'C:\\Users\\BFitz\\AppData\\Local\\Temp',
           'TMP': 'C:\\Users\\BFitz\\AppData\\Local\\Temp'},

The various builders in the default BUILDERS set find installed tools, and insert their appropriate settings. It feels a bit haphazard, and I’m hoping to find a way to tame it for big builds.

We’ll get back to this when we dive more deeply into tools and builders.

Tools and Builders

SCons is based around the concept of a Tool, which you can think of as a module. Each Tool is the interface to some piece of functionality, and often installs one or more Builders. Tools are also the preferred way for a user to extend SCons – any files inside the user’s site_tools directory are available to be installed as Tools, in one of three ways

  • env = Environment(tools=[…])
  • env.Tool(…)
  • Tool(…)

Let’s look at the default set of tools.  When we run this

env = Environment()
env.Dump()

we see this in the output on a Windows machine

  'TOOLS': [ 'default',
             'mslink',
             'msvc',
             'gfortran',
             'masm',
             'mslib',
             'msvs',
             'midl',
             'filesystem',
             'jar',
             'javac',
             'rmic',
             'zip'],

and we see this in the output on a Mac OS X machine

  'TOOLS': [ 'default',
             'applelink',
             'gcc',
             'g++',
             'gfortran',
             'as',
             'ar',
             'filesystem',
             'm4',
             'lex',
             'yacc',
             'rpcgen',
             'jar',
             'javac',
             'javah',
             'rmic',
             'tar',
             'zip',
             'CVS',
             'RCS'],

and we see this in the output on a Linux machine.

  'TOOLS': [ 'default',
             'gnulink',
             'gcc',
             'g++',
             'gfortran',
             'gas',
             'ar',
             'filesystem',
             'm4',
             'rpcgen',
             'jar',
             'javac',
             'rmic',
             'tar',
             'zip',
             'rpm'],

Each of these corresponds to a file of the same name in a directory in the toolpath. SCons’ built-in tools are in the toolpath, of course, and are located in the <scons_dir>/SCons/Tool directory (typically you will have installed SCons into the Python/Lib/site_packages directory). If you look in this directory, you’ll see default.py, mslink.py and etc.

A user’s build hierarchy can have a site_scons folder located next to its top-level SConstruct file, and a site_tools folder inside this directory will automatically be added to the toolpath. There are other global locations for site_tools folders, but those tend to add to build fragility (tying builds to specific machine setups) and are best avoided.

The ‘default’ tool is interesting. It has a large pre-canned list of tools. Each tool found on the machine is added to the Environment. Note that the ‘default’ tool is left on the Tools list, but the Tools list is nothing more than a manifest of what was found and added. This produces a system that works out of the box for many different build targets, but also increases build time, since much of the work to find tools is repeated on each build. For example, if you don’t build with Fortran, looking for and installing Fortran tools is wasted effort.

Let’s take a quick look at each of the default tools. Or rather, let’s look at some of the default tools, since there are a lot of them (I’m skipping the Java and Fortran tools for now). Our goal is two-fold – what effect does each tool have, and what are the relationships between tools? We’re going to go from simple tools to complex tools.

filesystem

This tool installs very little. The source for this is SCons/Tool/filesystem.py.

  'BUILDERS':
  {
    'CopyAs': <SCons.Builder.BuilderBase object at ...>,
    'CopyTo': <SCons.Builder.BuilderBase object at ...>
  },
  'COPYSTR': 'Copy file(s): "$SOURCES" to "$TARGETS"',
  'TOOLS': ['filesystem'],

It installs two Builders, CopyAs and CopyTo, and adds one construction variable to the Environment, COPYSTR, which is used in progress messages. The CopyAs Builder is used by the ‘packaging’ Tool to copy files from one directory to another. The distinction between CopyAs and CopyTo is that CopyTo copies from a source to a target directory (thus leaf names stay the same) whereas CopyAs copies from a source to a target (and leaf names can change).

These Builders are not mentioned in the SCons man page nor in the SCons User’s guide.

zip

This tool installs a little more than the filesystem tool, but not much more. The source for this is SCons/Tool/zip.py.

  'BUILDERS':
  {
    'Zip': <SCons.Builder.BuilderBase object at ...>
  },
  'TOOLS': ['zip'],
  'ZIP': 'zip',
  'ZIPCOM': ,
  'ZIPCOMPRESSION': 8,
  'ZIPFLAGS': [],
  'ZIPSUFFIX': '.zip',

It installs one Builder, Zip. The tool tries to use the Python zipfile module, making a wrapper for it that can be called from an Action. If it can’t be found, it assumes there is a zip program named “zip” on the path, and creates a command line to use for the Action. The Builder then uses that Action.

This is a good module to look at in order to understand how to create Actions and Builders, because it is fairly small, but not too simple. It even has default behavior that can be overridden by adding a construction variable (ZIPCOMSTR, although I would have documented that in code a little better).

This points out an interesting weakness in the SCons man page – it lists every variable in one flat list, but most of the variables are of interest to only one Tool or one Builder.

midl

This wraps a Microsoft program named midl.exe that is used to create type libraries. The source for this is SCons/Tool/midl.py.

  'BUILDERS':
  {
    'TypeLibrary': <SCons.Builder.BuilderBase object at ...>
  },
  'MIDL': 'MIDL.EXE',
  'MIDLCOM': ('$MIDL $MIDLFLAGS /tlb ${TARGETS[0]}'
             '/h ${TARGETS[1]} /iid ${TARGETS[2]}'
             '/proxy ${TARGETS[3]} /dlldata ${TARGETS[4]}'
             '$SOURCE 2> NUL'),
  'MIDLFLAGS': ['/nologo'],
  'TOOLS': ['midl'],

This adds one Builder, TypeLibrary. It also adds several configuration variables that are used in the invocation of TypeLibrary, one of which allows for user extension (MIDLFLAGS).

Again, this is a simple builder. However, when we look at the source code, we’ll see the first hint of entanglement between tools. The midl tool’s exists() function calls msvc_exists(), which is also used by the mslib, mslink, msvc and msvs tools. The msvc_exists() function memoizes its answer, so it only has cost the first time it’s called. the downside is that anything needed by any of these tools has to be configured before the first tool is installed. Essentially, all of these are part of a suite, even if they aren’t installed as such.

It’s also interesting that msvc_exists() can detect a specific VC version, and not just that one is installed. Nothing in the code currently uses this, however.

And.. as we’ll see later on, there are environment variables missing. We need a path to midl.exe, and we probably need some other environment variables as well. This is a bug, but not a very big one, because it’s hard to imagine a build that only uses TypeLibrary. Still, it’s wrong. See mslib for what the environment should look like at this point.

SCons needs tool suites. This wouldn’t be that hard to do, you could have a ‘vc’ tool that installs all the related tools by default, and could have one or more parameters to control what is installed.

masm

This wraps the Microsoft assembler masm.exe. The source for this is SCons/Tool/masm.py.

  'AS': 'ml',
  'ASCOM': '$AS $ASFLAGS /c /Fo$TARGET $SOURCES',
  'ASFLAGS': ['/nologo'],
  'ASPPCOM': ('$CC $ASPPFLAGS $CPPFLAGS $_CPPDEFFLAGS'
              '$_CPPINCFLAGS /c /Fo$TARGET $SOURCES'),
  'ASPPFLAGS': '$ASFLAGS',
  'BUILDERS':
  {
    'Object': <SCons.Builder.CompositeBuilder object at ...>,
    'SharedObject': <SCons.Builder.CompositeBuilder object at ...>,
    'StaticObject': <SCons.Builder.CompositeBuilder object at ...>
  },
  'STATIC_AND_SHARED_OBJECTS_ARE_THE_SAME': 1,
  'TOOLS': ['masm'],

While slightly more complicated, it’s still on par with the previous tools. We add three Builders: Object, SharedObject and StaticObject. We add command-line strings and flags to control the command-line invocation.

This tool doesn’t call msvc_exists(), and does not add anything to my path (unlike midl, which dumps all the Visual Studio locations into my path). However, this is wrong, because ml.exe (which is what it’s using to implement the masm functionality) is part of Visual C++. I’d say this is a bug, that msvc_exists needs to be called(). Otherwise, a build that just does assembly with the masm tool won’t actually work. I’ll file a bug or a pull request.

Again, I’d like to see documentation that shows each construction variable referenced by any given Tool, not just a flat list.

mslib

This wraps the Microsoft librarian lib.exe. The source for this is SCons/Tool/mslib.py.

{ 'AR': 'lib',
  'ARCOM': "${TEMPFILE('$AR $ARFLAGS /OUT:$TARGET $SOURCES')}",
  'ARFLAGS': ['/nologo'],
  'BUILDERS':
  {
    'Library': <SCons.Builder.BuilderBase object at ...>,
    'StaticLibrary': <SCons.Builder.BuilderBase object at ...>
  },
  'ENV':
   {
     ...
     'INCLUDE': ..., # a bunch
     'LIB': ..., # a bunch
     'LIBPATH': ..., # a bunch
     'PATH': ..., # a bunch
  }
  'MSVC_SETUP_RUN': True,
  'MSVC_VERSION': '11.0',
  'MSVS': { },
  'MSVS_VERSION': '11.0',
  'TARGET_ARCH': 'amd64',
  'TOOLS': ['mslib'],

This adds two Builders: Library and StaticLibrary. It also adds a number of construction variables. I left out enumerating all the paths, includes, libs and libpath entries added. If you want to see them, just run the SConstruct lines I listed above.

This tool ran msvc_exists(), and it also ran msvc_default_version(), and msvc_setup(). This is one-time setup (it won’t do anything in the future since MSVC_SETUP_RUN is set to True), so your chance to influence the specific toolchain is before you install any of these tools. Now, I suppose you could remove MSVC_SETUP_RUN and install again, but that sounds a bit dodgy. We see that it picked Visual Studio 2012 (VC11). I don’t know why MSVS is an empty array, I thought this was supposed to contain all the found versions. Maybe when it selects a specific version, it throws away the other information without recording it.

Note that this builder decided to change the TARGET_ARCH from x86_64 to amd64. That seems odd. I’ll have to look at the code to see why. It’s not a substantial change, but it’s a change I didn’t expect.

msvs

This is one of the bigger builders, to drive Microsoft Visual Studio. The source for this is SCons/Tool/msvs.py.

  'BUILDERS':
  {
    'MSVSSolution': <SCons.Builder.BuilderBase object at ...>,
    'MSVSProject': <SCons.Builder.BuilderBase object at ...>
  },
  'ENV': { 'INCLUDE': ... }
  'GET_MSVSPROJECTSUFFIX': <function GetMSVSProjectSuffix at ...>,
  'GET_MSVSSOLUTIONSUFFIX': <function GetMSVSSolutionSuffix at ...>,
  'MSVC_SETUP_RUN': True,
  'MSVC_VERSION': '11.0',
  'MSVS': { 'PROJECTSUFFIX': '.vcxproj', 'SOLUTIONSUFFIX': '.sln'},
  'MSVSBUILDCOM': '$MSVSSCONSCOM "$MSVSBUILDTARGET"',
  'MSVSCLEANCOM': '$MSVSSCONSCOM -c "$MSVSBUILDTARGET"',
  'MSVSENCODING': 'utf-8',
  'MSVSPROJECTCOM': <SCons.Action.FunctionAction object at ...>,
  'MSVSPROJECTSUFFIX': '${GET_MSVSPROJECTSUFFIX}',
  'MSVSREBUILDCOM': '$MSVSSCONSCOM "$MSVSBUILDTARGET"',
  'MSVSSCONS': ('"C:\\Python27\\Scripts\\..\\python.exe" '
                '-c "from os.path import join; import sys; '
                'sys.path = [ join(sys.prefix, \'Lib\', '
                '\'site-packages\', \'scons-2.2.0\'), join('
                'sys.prefix, \'scons-2.2.0\'), join(sys.prefix, '
                '\'Lib\', \'site-packages\', \'scons\'), '
                'join(sys.prefix, \'scons\') ] + sys.path; '
                'import SCons.Script; SCons.Script.main()"'),
  'MSVSSCONSCOM': '$MSVSSCONS $MSVSSCONSFLAGS',
  'MSVSSCONSCRIPT': <SCons.Node.FS.File object at ...>,
  'MSVSSCONSFLAGS': ('-C "${MSVSSCONSCRIPT.dir.abspath}" '
                     '-f ${MSVSSCONSCRIPT.name}'),
  'MSVSSOLUTIONCOM': <SCons.Action.FunctionAction object at ...>,
  'MSVSSOLUTIONSUFFIX': '${GET_MSVSSOLUTIONSUFFIX}',
  'MSVS_VERSION': '11.0',
  'SCONS_HOME': None,
  'TARGET_ARCH': 'amd64',
  'TOOLS': ['msvs'],

This tool adds two Builders: MSVSSolution, which generates a Visual Studio solution file, and MSVSProject, which generates a Visual Studio project file (and by default, the owning solution file). These do not run msbuild or devenv; these create project files from the set of sources and headers and target and variants that are presented to it.

The built project files just turn around and invoke SCons to do any actual building – see the rather large MSVSSCONS construction variable, which is the heart of the build operation that the project file is told.

There is an optional string SCONS_HOME that, if set before the msvs tool is constructed, is used to create a shorter path for MSVSSCONS. Or, that’s what the documentation says, but I’m not sure why that would be needed in order to avoid the mess of join calls in the MSVSSCONS variable. I also don’t know why this would be in a long command-line rather than in a file that’s invoked.

And most importantly, there’s a crying need for actual real project file generation, so that code builds could be done inside the generated project. Of course, this would be more complicated, because the builders would have to figure out what parts of the build would still need to be done by SCons. But that information is all there…

msvc

This drives the Visual Studio compiler, and it’s fair to say that it is moderately complicated. The source for this is SCons/Tool/msvs.py.

  'BUILDERS':
  {
    'Object': <SCons.Builder.CompositeBuilder object at ...>,
    'SharedObject': <SCons.Builder.CompositeBuilder object at ...>,
    'StaticObject': <SCons.Builder.CompositeBuilder object at ...>,
    'PCH': <SCons.Builder.BuilderBase object at ...>,
    'RES': <SCons.Builder.BuilderBase object at ...>
  },
  'CC': 'cl',
  'CCCOM': ('${TEMPFILE("$CC $_MSVC_OUTPUT_FLAG /c $CHANGED_SOURCES '
            '$CFLAGS $CCFLAGS $_CCCOMCOM")}'),
  'CCFLAGS': ['/nologo'],
  'CCPCHFLAGS': [('${(PCH and "/Yu%s \\"/Fp%s\\""%(PCHSTOP or "",'
                  'File(PCH))) or ""}')],
  'CCPDBFLAGS': ['${(PDB and "/Z7") or ""}'],
  'CFILESUFFIX': '.c',
  'CFLAGS': [],
  'CPPDEFPREFIX': '/D',
  'CPPDEFSUFFIX': '',
  'CXX': '$CC',
  'CXXCOM': ('${TEMPFILE("$CXX $_MSVC_OUTPUT_FLAG '
             '/c $CHANGED_SOURCES $CXXFLAGS $CCFLAGS $_CCCOMCOM")}'),
  'CXXFILESUFFIX': '.cc',
  'CXXFLAGS': ['$(', '/TP', '$)'],
  'ENV': { 'INCLUDE': ... }
  'INCPREFIX': '/I',
  'INCSUFFIX': '',
  'MSVC_SETUP_RUN': True,
  'MSVC_VERSION': '11.0',
  'MSVS': { },
  'MSVS_VERSION': '11.0',
  'PCHCOM': ('$CXX /Fo${TARGETS[1]} $CXXFLAGS $CCFLAGS '
             '$CPPFLAGS $_CPPDEFFLAGS $_CPPINCFLAGS /c $SOURCES '
             '/Yc$PCHSTOP /Fp${TARGETS[0]} $CCPDBFLAGS $PCHPDBFLAGS'),
  'PCHPDBFLAGS': ['${(PDB and "/Yd") or ""}'],
  'RC': 'rc',
  'RCCOM': ('$RC $_CPPDEFFLAGS $_CPPINCFLAGS '
            '$RCFLAGS /fo$TARGET $SOURCES'),
  'RCFLAGS': [],
  'RCSUFFIXES': ['.rc', '.rc2'],
  'SHCC': '$CC',
  'SHCCCOM': ('${TEMPFILE("$SHCC $_MSVC_OUTPUT_FLAG '
              '/c $CHANGED_SOURCES '
              '$SHCFLAGS $SHCCFLAGS $_CCCOMCOM")}'),
  'SHCCFLAGS': ['$CCFLAGS'],
  'SHCFLAGS': ['$CFLAGS'],
  'SHCXX': '$CXX',
  'SHCXXCOM': ('${TEMPFILE("$SHCXX $_MSVC_OUTPUT_FLAG '
               '/c $CHANGED_SOURCES $SHCXXFLAGS '
               '$SHCCFLAGS $_CCCOMCOM")}'),
  'SHCXXFLAGS': ['$CXXFLAGS'],
  'STATIC_AND_SHARED_OBJECTS_ARE_THE_SAME': 1,
  'TARGET_ARCH': 'amd64',
  'TOOLS': ['msvc'],
  '_CCCOMCOM': ('$CPPFLAGS $_CPPDEFFLAGS $_CPPINCFLAGS '
                '$CCPCHFLAGS $CCPDBFLAGS'),
  '_MSVC_OUTPUT_FLAG': <function msvc_output_flag at ...>,

By the way, in case it hasn’t been obvious, I’ve been hand-prettifying the output. Hopefully I haven’t introduced any typos. The … are where I have elided unimportant data, usually addresses, but sometimes long strings like paths or includes where the data isn’t really relevant to the discussion at hand.

This adds five builders, a few of which should be familiar. Object, SharedObject and StaticObject are builders that are shared between the masm tool and the msvc tool. The other two builders added here are PCH and RES, which generate precompiled headers and resource files, respectively.

Also note the use of TEMPFILE, which we talked about earlier in the discussion about environments; TEMPFILE is used to handle overly long command-lines. Specifically for us, Windows only gives us a 2048 character command line, so if we end up generating a longer one (and this happens quite easily), we need to wrap it in a file. Note that Visual Studio does this itself, and in fact a lot of the compile from devenv is driven through *.RSP files for that reason, but also that it makes debugging the build process a little easier (you have a record of what was going on).

Let’s talk a little about Python syntax you may be unfamiliar with (hopefully not, since you need to know Python to use SCons).

'PCHPDBFLAGS': ['${(PDB and "/Yd") or ""}']

In Python, the use of ‘and’ and ‘or’ gives the last value encountered, whereas && and || return True or False. So ‘PDB and “/Yd”‘ will either give the value of PDB or the value “/Yd”, and ‘(PDB and “/Yd”) or “”‘ will evaluate to “/Yd” if PBD is a True value, or “” if PDB is a False value.

Let’s break down a more complicated expression:

  'CCPCHFLAGS': [('${(PCH and "/Yu%s \\"/Fp%s\\""%(PCHSTOP or "",'
                  'File(PCH))) or ""}')],

This is a little hard to read because of the escaped backslash characters which only exist to make sure the double-quote characters make it into the final output, so I’m going to turn them into characters that preserve meaning even if not quite kosher to the compiler (because we’re smarter than compilers, even still).

${(PCH and '/Yu%s "/Fp%s"'%(PCHSTOP or "",File(PCH))) or ""}

This either turns into a string, or the empty string “”. If PCH is true, then we have the format string ‘/Yu%s “/Fp%s”‘ and the two values ‘PCHSTOP or “”‘ and File(PCH). So we either have ‘/Yu “/Fp”‘, or ‘/Yu “/Fp”‘, or ”, as the result.

I’m assuming there’s a good reason why /Fp needs to be quoted. Perhaps this is drive-by collateral damage from use of the ESCAPE function (which, if you remember, is double-quoting its argument).

Also, this is a good reason why you want to have more white-space in the right places, so it’s slightly easier to read complex expressions, and even (or especially when) your complex expressions are generated by code.

mslink

This drives the Microsoft linker. The source for this is SCons/Tool/msvs.py.

  'BUILDERS':
  {
    'LoadableModule': <SCons.Builder.BuilderBase object at ...>,
    'Program': <SCons.Builder.BuilderBase object at ...>,
    'SharedLibrary': <SCons.Builder.BuilderBase object at ...>
  },
  'ENV': { 'INCLUDE': ... }
  'LDMODULE': '$SHLINK',
  'LDMODULECOM': <SCons.Action.ListAction object at ...>,
  'LDMODULEEMITTER': [<function ldmodEmitter at ...>],
  'LDMODULEFLAGS': '$SHLINKFLAGS',
  'LDMODULEPREFIX': '$SHLIBPREFIX',
  'LDMODULESUFFIX': '$SHLIBSUFFIX',
  'LIBDIRPREFIX': '/LIBPATH:',
  'LIBDIRSUFFIX': '',
  'LIBLINKPREFIX': '',
  'LIBLINKSUFFIX': '$LIBSUFFIX',
  'LINK': 'link',
  'LINKCOM': <SCons.Action.ListAction object at ...>,
  'LINKFLAGS': ['/nologo'],
  'MSVC_SETUP_RUN': True,
  'MSVC_VERSION': '11.0',
  'MSVS': { },
  'MSVS_VERSION': '11.0',
  'MT': 'mt',
  'MTEXECOM': ('-$MT $MTFLAGS -manifest ${TARGET}.manifest '
               '$_MANIFEST_SOURCES -outputresource:$TARGET;1'),
  'MTFLAGS': ['/nologo'],
  'MTSHLIBCOM': ('-$MT $MTFLAGS '
                 '-manifest ${TARGET}.manifest $_MANIFEST_SOURCES '
                 '-outputresource:$TARGET;2'),
  'PROGEMITTER': [<function prog_emitter at ...>],
  'REGSVR': u'C:\\windows\\System32\\regsvr32',
  'REGSVRACTION': <SCons.Action.FunctionAction object at ...>,
  'REGSVRCOM': '$REGSVR $REGSVRFLAGS ${TARGET.windows}',
  'REGSVRFLAGS': '/s ',
  'SHLIBEMITTER': [<function windowsLibEmitter at ...>],
  'SHLINK': '$LINK',
  'SHLINKCOM': <SCons.Action.ListAction object at ...>,
  'SHLINKFLAGS': ['$LINKFLAGS', '/dll'],
  'TARGET_ARCH': 'amd64',
  'TOOLS': ['mslink'],
  'WIN32DEFPREFIX': '',
  'WIN32DEFSUFFIX': '.def',
  'WIN32EXPPREFIX': '',
  'WIN32EXPSUFFIX': '.exp',
  'WIN32_INSERT_DEF': 0,
  'WINDOWSDEFPREFIX': '${WIN32DEFPREFIX}',
  'WINDOWSDEFSUFFIX': '${WIN32DEFSUFFIX}',
  'WINDOWSEXPPREFIX': '${WIN32EXPPREFIX}',
  'WINDOWSEXPSUFFIX': '${WIN32EXPSUFFIX}',
  'WINDOWSPROGMANIFESTPREFIX': '',
  'WINDOWSPROGMANIFESTSUFFIX': '${PROGSUFFIX}.manifest',
  'WINDOWSSHLIBMANIFESTPREFIX': '',
  'WINDOWSSHLIBMANIFESTSUFFIX': '${SHLIBSUFFIX}.manifest',
  'WINDOWS_EMBED_MANIFEST': 0,
  'WINDOWS_INSERT_DEF': '${WIN32_INSERT_DEF}',
  '_LDMODULE_SOURCES': <function _windowsLdmodSources at ...>,
  '_LDMODULE_TARGETS': <function _windowsLdmodTargets at ...>,
  '_MANIFEST_SOURCES': None,
  '_PDB': <function pdbGenerator at ...>,
  '_SHLINK_SOURCES': <function windowsShlinkSources at ...>,
  '_SHLINK_TARGETS': <function windowsShlinkTargets at ...>,

Yeah. That’s a lot of stuff.

This introduces three builders: LoadableModule, Program, and SharedLibrary. This is actually a bit of an omnibus tool, since it comprises the linker proper (link.exe), the manifest generation tool (mt.exe), and Regsvr32 which registers COM DLLs (regsrvr.exe).

Yes, SCons will register your built shared library if you want it to. This requires you to set the ‘register’ construction variable to 1, and then when a DLL is built, it will be registered with Regsrvr32; this is needed if you’re working with COM or ActiveX.

SCons Environment in depth, part 1

Let’s tear down Environment(), so we know how to use it properly. As much as possible, I’m looking at things as a user, so I’ve been trying things without looking at the source code of SCons. I want to see how things actually work, as opposed to what the SCons developers intended.

For other parts in the series, see:

Minimal environment

Looking at a somewhat minimal Environment in SCons (we have to supply a platform, we’ll get to that in a moment) like so:

env = Environment(tools=[], platform='win32', BUILDERS={}, ENV={})
env.Dump()

and re-arranging for prettiness, we have

env = Environment(tools=[], platform='win32', builders={}), ENV={}
{
  'BUILDERS': {},
  'ENV': { },
  'SCANNERS': [],

  'CONFIGUREDIR': '#/.sconf_temp',
  'CONFIGURELOG': '#/config.log',
  'MAXLINELENGTH': 2048,
  'TEMPFILE': <class 'SCons.Platform.TempFileMunge'>,
  'TEMPFILEPREFIX': '@',

  'HOST_ARCH': 'x86_64',
  'HOST_OS': 'win32',
  'PLATFORM': 'win32',
  'TARGET_ARCH': 'x86_64',
  'TARGET_OS': 'win32',

  'LIBPREFIX': '',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.lib',
  'LIBSUFFIXES': ['$LIBSUFFIX'],
  'OBJPREFIX': '',
  'OBJSUFFIX': '.obj',
  'PROGPREFIX': '',
  'PROGSUFFIX': '.exe',
  'SHLIBPREFIX': '',
  'SHLIBSUFFIX': '.dll',
  'SHOBJPREFIX': '$OBJPREFIX',
  'SHOBJSUFFIX': '$OBJSUFFIX',

  'CPPSUFFIXES': [ '.c',
                   '.C',
                   '.cxx',
                   '.cpp',
                   '.c++',
                   '.cc',
                   '.h',
                   '.H',
                   '.hxx',
                   '.hpp',
                   '.hh',
                   '.F',
                   '.fpp',
                   '.FPP',
                   '.m',
                   '.mm',
                   '.S',
                   '.spp',
                   '.SPP',
                   '.sx'],
  'DSUFFIXES': ['.d'],
  'IDLSUFFIXES': ['.idl', '.IDL'],

  '_CPPDEFFLAGS': '${_defines(CPPDEFPREFIX, CPPDEFINES,
                              CPPDEFSUFFIX, __env__)}',
  '_CPPINCFLAGS': '$( ${_concat(INCPREFIX, CPPPATH, INCSUFFIX,
                                __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBDIRFLAGS': '$( ${_concat(LIBDIRPREFIX, LIBPATH, LIBDIRSUFFIX,
                                __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBFLAGS': '${_concat(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX,
                          __env__)}',

  'Dir': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'Dirs': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'ESCAPE': <function escape at ...>,
  'File': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'PSPAWN': <function piped_spawn at 0x025E64F0>,
  'RDirs': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'SHELL': u'C:\\windows\\System32\\cmd.exe',
  'SPAWN': <function spawn at ...>,
  '_concat': <function _concat at ...>,
  '_defines': <function _defines at ...>,
  '_stripixes': <function _stripixes at ...>
}

The only difference you’ll see if you run this on a specific platform is SHELL – this will be None if you run this in a posix environment, and ‘command’ if you run it in a darwin environment.

Platform

You must specify a platform when creating an environment, and it’s not easy to change after that. By “you must”, I mean that a platform will be specified, whether you do so explicitly or let SCons do it for you. By “not easy to change after that”, I mean that you can’t do it through normal SCons means. For example

env = Environment(platform='win32')
denv = env.Clone(platform='darwin')
log('win32_to_darwin.txt', cenv, "env.Clone(platform='darwin')")

still gives us a ‘win32′ platform, as we can see from our environment dump.

...
  'AS': 'ml',
  'ENV': { 'COMSPEC': 'C:\\windows\\system32\\cmd.exe', ...
  'HOST_ARCH': 'x86_64',
  'HOST_OS': 'win32',
  'PLATFORM': 'win32',
...

Since we’re not going to reach inside an Environment and fiddle with it, this means that we have to specify the platform up front, in our Environment call. This also means that anything that is platform-specific is going to be configured in that Environment call, which means if we have anything we need to declare for the platform setup, we have to declare it before the Environment call.

ENV, the SCons Environment’s copy of the host environment variables, is an interesting case. Are there things that need to be in ENV for platform setup to work? What exactly does platform setup encompass? Well, one way to see is to look at what changes in Environment calls. Here’s our test suite, saved to an SConstruct file.

import SCons

def log(name, env, prompt):
  logfile = open(name, 'w')
  print >> logfile, prompt
  print >> logfile, env.Dump()
  logfile.close

for plat in ['win32', 'darwin', 'posix', 'cygwin']:
  env = Environment(platform=plat, tools=[], BUILDERS={}, ENV={})
  log('raw_%s.txt' % plat, env,
      "env = Environment(platform=%s...)" % plat)

We’ll run this on a Windows machine with Visual Studio, a Mac OS X machine with XCode, and a Linux box with GCC 4.7.4, and we’ll look at the differences on same and different host OS machines. If something below is not shown, assume it is the same on all platforms and machines.

Note: I wrote the following before I really understood SCons. I’m going to rip out some of the text, because my comment about SCons not being a cross-platform tool is the key. It needs a HOST_PLATFORM/HOST_ARCH config for finding toolchains that it can run, and it needs a TARGET_PLATFORM/TARGET_ARCH for finding or configuring toolchains that will produce the desired output.

platform=’win32′

‘win32′ on Windows

  'HOST_ARCH': 'x86_64',
  'HOST_OS': 'win32',
  'MAXLINELENGTH': 2048,
  'PLATFORM': 'win32',
  'SHELL': u'C:\\windows\\System32\\cmd.exe',
  'TARGET_ARCH': 'x86_64',
  'TARGET_OS': 'win32',

  'LIBPREFIX': '',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.lib',
  'LIBSUFFIXES': ['$LIBSUFFIX'],
  'OBJSUFFIX': '.obj',
  'PROGSUFFIX': '.exe',
  'SHLIBPREFIX': '',
  'SHLIBSUFFIX': '.dll',
  'PSPAWN': <function piped_spawn at ...> 
  'SPAWN': <function spawn at ...>

‘win32′ on Mac OS X is almost the same as ‘win32′ on Windows with these differences (and the differences would prevent actual cross-compiling, assuming there were Win32 tools that ran on the Mac). Among other things, SPAWN and PSPAWN are probably not set correctly for Mac OS X.

  'HOST_ARCH': '',
  'SHELL': 'command',
  'TARGET_ARCH': '',

‘win32′ on Linux is almost the same, except with these differences

  'HOST_ARCH': '',
  'SHELL': None,
  'TARGET_ARCH': None,
  'TARGET_OS': None,

platform=’darwin’

‘darwin’ on Mac OS X

  'HOST_ARCH': None,
  'HOST_OS': None,
  'MAXLINELENGTH': 128072,
  'PLATFORM': 'darwin',
  'TARGET_ARCH': None,
  'TARGET_OS': None,

  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'OBJSUFFIX': '.o',
  'PROGSUFFIX': '',
  'SHELL': 'sh',
  'SHLIBPREFIX': '$LIBPREFIX',
  'SHLIBSUFFIX': '.dylib',
  '__RPATH': '$_RPATH',

  'PSPAWN': <function piped_fork_spawn at ...>,
  'SPAWN': <function fork_spawn at ...>,

‘darwin’ on Windows is identical to ‘darwin’ on Mac OS X. This also means it’s not actually used for cross-compiling, since there is likely no sh shell on Windows, but there could be a Clang cross-compiler running on Windows that targets Darwin (and that probably does exist somewhere).

‘darwin’ on Linux is identical to both, and actually this means that the darwin environment is just setting variables, it’s not actually looking for anything.

platform = ‘posix’

‘posix’ on Linux

  'HOST_ARCH': None,
  'HOST_OS': None,
  'MAXLINELENGTH': 128072,
  'PLATFORM': 'posix',
  'TARGET_ARCH': None,
  'TARGET_OS': None,

  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'OBJSUFFIX': '.o',
  'PROGSUFFIX': '',
  'SHELL': 'sh',
  'SHLIBPREFIX': '$LIBPREFIX',
  'SHLIBSUFFIX': '.so',
  '__RPATH': '$_RPATH',

‘posix’ on Mac OS X is almost identical to ‘darwin’ on Mac OS X. The only differences are platform and SHLIBSUFFIX, as well as some of the tools

  'HOST_ARCH': None,
  'HOST_OS': None,
  'MAXLINELENGTH': 128072,
  'PLATFORM': 'posix',
  'TARGET_ARCH': None,
  'TARGET_OS': None,

  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'OBJSUFFIX': '.o',
  'PROGSUFFIX': '',
  'SHELL': 'sh',
  'SHLIBPREFIX': '$LIBPREFIX',
  'SHLIBSUFFIX': '.so',
  '__RPATH': '$_RPATH','

  'PSPAWN': <function piped_env_spawn at ...>,
  'SPAWN': <function spawnvpe_spawn at ...>,

‘posix’ on Windows is identical to ‘posix’ on Mac OS X, with the exception of PSPAWN and SPAWN (pointing to piped_fork_spawn). So there’s a tiny bit of accomodation to Windows, but not enough to be a valid platform foundation. Still, maybe there are tools that would build posix on Windows.

platform=’cygwin’

We’re starting to see a pattern. It makes sense that ‘cygwin’ is only really valid when used on a Windows machine, because Cygwin is a Posix-emulation for Windows, and you don’t need Posix-emulation on a machine that is natively Posix (like with Mac OS or Linux).

‘cygwin’ on Windows has some oddities due to its nature, because while the internals of the toolchain are Posix, it emits Windows DLLS and EXEs. It also supports both Posix and Windows-style libraries, where posix libraries typically are libSOMETHING, whereas Windows libraries are just SOMETHING.

  'HOST_ARCH': 'None',
  'HOST_OS': 'None',
  'MAXLINELENGTH': 2048,
  'PLATFORM': 'cygwin',
  'SHELL': 'sh',
  'TARGET_ARCH': 'None',
  'TARGET_OS': 'None',

  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX', '$SHLIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'OBJSUFFIX': '.o',
  'PROGSUFFIX': '.exe',
  'SHLIBPREFIX': '',
  'SHLIBSUFFIX': '.dll',
  'PSPAWN': <function piped_fork_spawn at ...> 
  'SPAWN': <function fork_spawn at ...>
  '__RPATH': '$_RPATH',

platform tips

Platform in SCons is mixing up host platform support (e.g. SPAWN, RSPAWN and COMMAND) with target platform and toolchain settings (LIPBPREFIX etc).

The only cross-platform compiling that SCons is likely to support is ‘win32′ versus ‘cygwin’, or ‘darwin’ versus ‘posix’. I would think a more sane approach would be to declare a build error if an untenable platform were asked for.

It’s interesting and depressing that we are mixing tools, builders and platforms together. The LIBPREFIX and LIBSUFFIX are really paired with specific toolchains. It turns out that this works with current systems I know about, but platform in the context of SCons is actually more about “toolchain family”. This view is inherited from make, I believe.

However, this is all easily fixable. I suppose no one to date has really cared. SCons is probably not being used for embedded development, which is where this would really matter.

I think I’ll send in some pull requests so that HOST_ARCH and HOST_OS aren’t Windows-specific. The documentation mentions that they currently are, but won’t be in the future, so – let’s make the future today. Also, we really should have a HOST_PLATFORM and TARGET_PLATFORM. This is especially relevant with Mac OS versus iOS builds. There’s also the Xamarin toolchain, which lets you build for iOS on Windows (to some degree), and there’s also Android targets built on Posix hosts.

What does this mean for us, the large project build developers? We will only allow for specific combinations of host operating system and SCons platform. And more specifically, at the moment, we have to use the native platform unless we are using a Cygwin toolchain.

Other settings

'CONFIGUREDIR': '#/.sconf_temp'

This is used by SConf, the Autoconf-like configuration support in SCONS. It defaults to a directory named .sconf_temp in the root of your build (wherever your top-level SConstruct file is located). It’s a dot-name so it’s invisible by default on Posix systems. This is where Configure test files are written, and the entire directory is removed after Configure runs (I think).

'CONFIGURELOG': '#/config.log

This is also used by SConf and holds status of Configure operations. I presume it’s largely used for debugging the Configure system. The SCons manual says “The name of the Configure context log”.  It looks like it contains any output from tools run as part of Configure operations (test compiles and so on).

'MAXLINELENGTH': 2048,
'TEMPFILE': <class 'SCons.Platform.TempFileMunge'>,
'TEMPFILEPREFIX': '@',

Systems have a max length for a command that can be issued. For example, Windows has a limit of 2048 characters (or probably bytes) for a command-line. The TempFileMunge function can be used to get around this limit – command-lines longer than MAXLINELENGTH are actually executed as a temporary command file. If you want to override this with your own system, you can replace ‘TEMPFILE’.

Naming abstractions

SCons tries to have one name for things at the SConstruct level. These need to be translated into platform-specific and tool-specific names to get actual work done.

  'CPPSUFFIXES': [ '.c', ...
  'DSUFFIXES': ['.d'],
  'IDLSUFFIXES': ['.idl', '.IDL'],

CPPSUFFIXES is the list of suffixes of files that will be scanned for implicit dependencies as caused by #include directives. DSUFFIXES is the list of suffixes of files that will be scanned for imported D packages, again for implicit dependencies. IDLSUFFIXES is the list of suffixes of files that will be scanned for IDL implicit dependencies as indicated by either #include directives or import lines.

There’s probably some architectural reason why these are in the platform section. That said, these really should be tied to tools, not the platform. If I’m not building C/C++ files, I shouldn’t try to scan my source files for implicit dependencies, even if they end in some C-like suffix. And this is especially true for not-yet-mainstream languages like D. Adding spurious dependencies is in some respects “harmless” – it won’t cause a bad build. But it can cause your build to take longer than it should.

The reverse is definitely not true – if you miss some implicit dependencies, you can end up not rebuilding something and have a bad build. Implicit dependencies are always problematic, and perhaps the actual answer is what other build systems do, monitoring all file I/O while a full build is running, and capturing dependencies that way. Maybe do both?

  '_defines': <function _defines at ...>,
  '_CPPDEFFLAGS': '${_defines(CPPDEFPREFIX, CPPDEFINES,
                              CPPDEFSUFFIX, __env__)}',

  '_concat': <function _concat at ...>,
  'RDirs': <SCons.Defaults.Variable_Method_Caller object at ...>,
  '_CPPINCFLAGS': '$( ${_concat(INCPREFIX, CPPPATH, INCSUFFIX,
                                __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBDIRFLAGS': '$( ${_concat(LIBDIRPREFIX, LIBPATH, LIBDIRSUFFIX,
                                __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBFLAGS': '${_concat(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX,
                          __env__)}',

These are helper functions that glue together C preprocessor defines, C includes, library directories, and librarian flags.

_CPPDEFFLAGS is a construction variable that turns a list of preprocessor symbols to define and turns them into compiler command-line arguments. For example, GCC likes -D and Visual C++ likes /D. The _defines function is used to do the actual concatenation, handling both /Dsym and /Dsym=val forms.

_CPPINCFLAGS is a construction variable that turns a list of directories into compiler command-line arguments for include search paths. _LIBDIRFLAGS is a construction variable to turn a list of directories into compiler command-line arguments for library search paths. _LIBFLAGS is a construction variable to turn a list of libraries into compiler command-line arguments for library linking. The _concat function takes a prefix, suffix, list of elements, an environment, and optional post-processing function, target and source.

Again, why are these defined in the platform as opposed to being tied to tools? I suppose it’s because there’s not really the idea of tool suites?

  'LIBPREFIX': '',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.lib',
  'LIBSUFFIXES': ['$LIBSUFFIX'],
  'OBJPREFIX': '',
  'OBJSUFFIX': '.obj',
  'PROGPREFIX': '',
  'PROGSUFFIX': '.exe',
  'SHLIBPREFIX': '',
  'SHLIBSUFFIX': '.dll',
  'SHOBJPREFIX': '$OBJPREFIX',
  'SHOBJSUFFIX': '$OBJSUFFIX',

These are abstractions for naming of intermediate files.

LIBPREFIX, LIBPREFIXES, LIBSUFFIX and LIBSUFFIXES are prefixes and suffixes for static library files. On Windows, these are typically NAME.lib, and on Posix, these are typically libNAME.a. Your code would refer to the library as NAME. The reason for the plural for is to handle cases where multiple naming schemes can co-exist.

OBJPREFIX and OBJSUFFIX are prefixes and suffixes for object files that are linked into libraries. On windows, these are typically NAME.obj, and on Posix, these are NAME.o. I’m guessing no one has run across a case where multiple naming schemes need to be handled on a single platform.

PROGPREFIX and PROGSUFFIX are for executable names.

SHLIBPREFIX and SHLIBSUFFIX are for shared library names.

SHOBJPREFIX and SHOBJSUFFIX are for naming of objects that go into shared libraries. I’m guessing there is some platform or toolchain that wants these to be distinguished from “normal” object files.

  'Dir': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'Dirs': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'ESCAPE': <function escape at ...>,
  'File': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'PSPAWN': <function piped_spawn at 0x025E64F0>,
  'SHELL': u'C:\\windows\\System32\\cmd.exe',
  'SPAWN': <function spawn at ...>,
  '_stripixes': <function _stripixes at ...>

ESCAPE is a helper function that escapes strings used on command-lines so that special characters in strings don’t cause problems. For example, spaces in paths are problematic, and those strings need to be quoted. The default escape function is pretty draconian, it just quotes all strings on Windows, for example.

SHELL is the name of the shell program passed to the SPAWN function.

SPAWN is the command interpreter function called to execute command lines. PSPAWN is a piped variant of SPAWN used in some cases (currently only in the Configure module).

_stripixes is a helper function that removes prefixes and suffixes in certain cases. Specifically, the GNU linker wants to turn libfoo.a into -lfoo.

Dir, Dirs, File and RDirs are all functions that convert a list of strings into something relative to the target being built. This is done by looking up the call chain for the first TARGET variable and then using it as the base for the string. TARGET will be set by a builder before the build actions are started. So, for example, a env[‘Dir’] call will end up calling TARGET.Dir() for the first TARGET that is found while going up the stack. This saves a lot of copying of environment variables at the expense of a little speed and clarity in the code.

scons –interactive

scons –interactive is one way to speed up repeated builds.

SCons spends most of its time at three tasks

  1. Reading and parsing the SConstruct/SConscript/site_scons files. This takes about 6-10 seconds at the moment in one of our moderate-sized projects.
  2. Examining dependencies. This takes 1-20 seconds in that same project.
  3. Building targets. This can take 0-1000 seconds, depending on how much is being rebuilt.

The first one can be separated from the other two quite easily. If you do this

scons --interactive

then SCons will read all the build files, and drop you into a little shell

scons>>> build

will simply build everything. Or

scons>>> build game_client

will just build the game_client target.

Type exit to leave the shell

scons>>> exit

As long as you are not updating any build scripts themselves, you can leave a command-prompt in the scons shell, and you’ll shave seconds off each rebuild (for the project I’m working on, this is 5-10 seconds). This can be significant if you are working on a subset of the files and know it.

If you have command-line options that affect the build, you’d typically do them on the call that starts the interactive shell. For example, we have some typical options like

scons --interactive --fast --skiptests

I’ve just started doing this, so I don’t know yet if there are hidden gotchas.

The commands you can issue in interactive mode are

  • build
  • clean
  • exit
  • help
  • shell
  • version

You can use shell to run another scons, but I suggest against doing “shell scons –interactive” from inside an scons shell :)

For slightly more information, see the scons man page at http://www.scons.org/doc/production/HTML/scons-man.html.

The real goal is that naive builds are fast (that’s what I call just typing “scons”), but right now naive builds have a fairly large overhead on non-trivial projects.

Using SCons effectively

(This is currently a work in progress)

There are no great cross-platform build systems. Of all the general-purpose ones, SCons is merely the least bad. I don’t want to re-hash those arguments here. Just trust me.

What do you need to know in order to use SCons effectively? You need to understand a little about the architecture of SCons, and you need to set up and follow some conventions.

In general, you create an Environment (or multiple Environments), populate it with config and tools, and then operate it with Builders and Actions and Aliases. SCons is a functional/declarative system; you declare what you want. It can be somewhat disconcerting at first to work this way, although many other build systems do follow this model, starting with the venerable Make.

Environment

SCons does its best to decouple the build system from the environment of a build machine. This is generally a good thing, otherwise you end up with build environments that are hard to replicate. You can import things from your machine’s environment, but you generally have to do so either explicitly, or through using a Tool.

The SCons build process has the idea of an Environment, which is the collection of things that builders operate on. The subset of the local machine environment that is useful is in the ENV field of the environment. Its structure is somewhat Unix-centric, since SCons started in an Unix environment, but enough of this maps to Windows such that you can almost write generic code.

Platforms

By default, environments are initialized with builders and variables appropriate to the host platform. It is possible to some degree to do cross-platform builds, and create an Environment with a specific platform. This does imply that builders are available on the host platform for that target platform.

SCons knows about the following platforms (as well as  lot of other old or niche platforms that you can find in the source code).

  • ‘cygwin’
  • ‘darwin’
  • ‘posix’
  • ‘win32′

Some cross-platform building is simple – for example, building ‘cygwin’ on a Windows OS, or ‘posix’ on a Mac OS. Other cross-platform building would be extremely challenging, but getting simpler. For example, at some point it might be possible to generate Windows binaries on a Posix platform with Clang, and the vice-versa is nearly true today.

Options and Variables

There are two main ways for a user to pass information to an SCons build operation.

Command-line options are declared with SCons.Script.AddOption, and used with two dashes – e.g., –verbose or –arch=win32. This is essentially the Python optparse module, with a few additions. Command-line options always have default values, even if the default value is None. SCons itself uses AddOption for its options. A handful of options can be set with SetOption() in code, but most options can only be supplied on the command-line.

Variables, or arguments, are passed on the command line as key/value pairs. These are stored in the SCons.Script.ARGUMENTS dictionary, and are accessed as a normal Python dictionary. They are also in the ARGLIST list, in the order they were given on the command line (but without the names, just the values). The ARGLIST approach is also the only way to access multiple occurrences of a variable. You can also use a Variables object to allow these to have help and default values.

Which is the correct approach? Generally, it seems like build scripts use Option instead of Variable. However, Variables are attached to environments, and Option is global, so there is room for both.

Construction variables

While these may be initialized by both arguments and options, and tools and platforms, construction variables are how information is passed to tool invocations when Builders and Actions actually create and run tasks. There is a generalized expansion operation that instantiates construction variables as needed. This also can be the source of some confusion at first, since expansion happens at the point it is needed, and usually not at the point where you declare Builders and Actions.

Tools

The way to use and extend SCons is through tools. Almost every bit of build action is done as a tool. When you create a new environment and do not specific any tools, this is the same as specifying the ‘default’ tool, which automatically adds a set of tools based on the host platform.

Most of your build actions should be expressed as tools, and your SConstruct and SConscripts should simply be declaring all the dependencies and targets.

Your tools typically go into the site_scons/site_tools directory. However, there is a toolpath argument to Environment that can point to additional tools directories, and the toolpath argument is stored in the environment for use by further Clone or Tool calls.

Tools can even be functions, as opposed to files. Tool entries can also have arguments attached to them, which can be useful for tools that need information in order to be initialized properly. For example, the code in a tool might be unable to locate tool binaries, whereas an argument (perhaps from the user on the command-line) might contain that information.

Builders

Builders are somewhat orthogonal to tools. A tool may introduce custom builders, or it may refine builders. Builders turn sources into targets, and through inference (usually with rules about file extensions) either targets or sources can be inferred from the other.

Builders do two things. At parse time, builders are the major factor in building the dependency tree. At build time, builders are invoked when dependencies declare that a target needs to be generated or regenerated. The classic builder is exemplified by StaticObject, which builds a static object file from one or more C, C++, D, or Fortran source files.

If you have one-off operations, then you can use Command to create an ad-hoc builder; this specifies the sources, the targets, and a command-line or Python object used to turn the sources into targets.

Aliases

Closely related to Builders are Aliases. An Alias creates one or more abstract targets that expand to one or more other targets. As well, an action or list of actions can be specified when any of the alias targets are not up-to-date.

The major use of Alias is to create meta targets. For example, a set of related programs that are all used together could be connected by an Alias, and then the user builds the Alias target, instead of each individual target. Quite often, there is no one single top-level target, and electing one of the lower-level targets arbitrarily as “the target” is confusing and prone to long-term error.

Actions

Actions are not often directly used. However, some Actions find great use. PreActions are run before a specified target is build, and PostActions are run after a specified target is built. Both of these let you customize Build actions without needing to create a new Builder, or modify an existing one.

Performance

SCons is slow. The major sources of slowness are the reading and comprehending of the SCons build file (the SConstruct, SConscripts, and related build files), and then the detection of dependencies by looking at every individual file in the build. Even a moderate build with a few tens of thousands of files can take 5-10 seconds to get through the parsing of the scripts, and another 30 seconds in dependency checking.

For the former, scons –interactive might come in handy. This reads and parses all the scripts, then drops the user into a very simple shell. The most common action is “build”, and the user can build specific targets, or the default targets. if multiple builds are done back-to-back without changing the build scripts themselves, this could save some time.

For the latter, the only way to speed up SCons is to delegate the file scanning to some other system, or to rewrite its system. SCons has a cache, but it still hits the filesystem to validate that the cache is good, and this makes the cache less useful. It might be possible to integrate a system like ibb to be an oracle for the portion of the filesystem referenced by SCons. Systems like ibb have code that watches for filesystem updates; they basically cache a subset of the filesystem in RAM, and updates to that cache are proportional to the number of changes to that filesystem.

The actual answer is to rewrite a large part of SCons to allow for a filesystem-watching approach. Systems like ibb and tup are bottom-up, driven by the changes from the filesystem. Systems like make and scons are top-down, driven by the declaration of file dependencies. One answer that would preserve the general power of the top-down approach would be to have filesystem changes result in repair actions to the dependency tree.

And the true answer would be to integrate a build system, a filesystem watcher, and a content-addressable revision control system like git. Files that are committed no longer need watching, and don’t need dependency rediscovery.

To-do

Experiment with custom env.Decider functions to see where the time is being spent. If an scons –interactive build with a custom Decider that just returns false takes no time, then an ibb approach would be of huge benefit.

Use EnsureSConsVersion and EnsurePythonVersion to make sure that users have appropriate versions of both.

Python’s atexit.register() function can be used to cause functions to run at exit.

Investigate Repository() to see if it would be practical to mix scons and git together.

Appendix

Scons documentation

Threads about command-line arguments

Threads about precompiled headers

Scons reading

http://blog.nelhage.com/2010/11/why-scons-is-cool/

http://mu2e.fnal.gov/public/hep/computing/scons.shtml

http://cournape.wordpress.com/?s=scons

http://www.m5sim.org/SCons_build_system

https://code.google.com/p/gyp/

Blender has two build toolchains – scons and cmake. But Google switched away from scons in pepper and are using makefiles (which is sad because they had a pretty disciplined way of using scons).

https://groups.google.com/forum/#!msg/native-client-discuss/O3kRuaYm5bE/wnpyYOP5MHkJ

http://0install.net/package-scons.html

http://aligorith.blogspot.com/2010/08/build-systems-in-rush-quick-recipe-for.html

http://two.pairlist.net/pipermail/scons-dev/2013-February/000597.html

V8 for Chromium uses gyp instead of scons now.

Libjingle (also Google) uses both gyp and scons.

Native Client might be supporting both? But comment above about pepper contradicts. https://code.google.com/p/nativeclient/issues/detail?id=2731

http://cloud9.epfl.ch/testing-programs/producing-llvm-binaries

Several people have extended Scons in one way or another.

Ninja is yet another build system, and someone has integrated it with Scons. Or, rather, generate a Ninja script from an Scons dry run.

SCons tools

SCons has a plug-in-like architecture that they call Tools. There are a large number of built-in tools for compiling, linking, copying, etc. You can add your own, they are Python scripts that go into your site_scons/site_tools directory.

You can see what the default tools are for a particular platform by looking in the environment you create.

base_env = Environment()
log('base_env.txt', base_env, "base_env = Environment()")

where log is a simple function I made

def log(name, env, prompt):
  logfile = open(name, 'w')
  print >> logfile, prompt
  print >> logfile, env.Dump()
  logfile.close

When I run this on a Windows machine, among all the output, I see:

  'TOOLS': [ 'default',
             'mslink',
             'msvc',
             'gfortran',
             'masm',
             'mslib',
             'msvs',
             'midl',
             'filesystem',
             'jar',
             'javac',
             'rmic',
             'zip'],

When I run this on a Mac with XCode 4.4.1 (clang, not gcc), I see:

  'TOOLS': [ 'default',
             'applelink',
             'gcc',
             'g++',
             'gfortran',
             'as',
             'ar',
             'filesystem',
             'm4',
             'lex',
             'yacc',
             'rpcgen',
             'jar',
             'javac',
             'javah',
             'rmic',
             'tar',
             'zip',
             'CVS',
             'RCS'],

I can’t quite figure out how the ‘default’ keyword expands to this list, but it has to be in the SCons/Tool/__init__.py script. It’s finding one of compiler, C++ compiler, linker, assembler, fortran_compiler and ar, as well as some other tools like filesystem, m4, wix and etc. It’s only adding files that actually exist, hence not showing yacc on Windows.

The names of the SCons tools are set in stone, so perhaps we should specifically add names and not use ‘default’? Except… well, it’s probably ok, because if SCons change that much, other parts of our build will break too.

It’s also crazy that SCons on Windows is so slow at finding things – at least, I assume that’s what the difference in performance is on Mac vs Windows. The Mac is about 3x faster per Environment() call versus Windows.

All the built-in tools for SCons are listed here: http://www.scons.org/doc/2.2.0/HTML/scons-user.html#app-tools.

It’s interesting that SCons thinks the D language compiler is on my Mac OS X 10.8 box (the dmd tool). I didn’t think it was. SCons is looking in my path for dmd or gdmd, and I don’t see any such thing. A bug?

Timing SCons

Note: if you run across this, I’ve been updating it as I go. I know, bad form for blog posts. Oh well.

I’m trying to figure out what SCons spends all its time doing. In all the below, I wrote synthetic tests, then try to calculate the cost of individual actions. Here’s the summary

  • Python: 60 milliseconds
  • SCons: 490 milliseconds
  • AddOption(): 0.05 milliseconds
  • Environment(): 350 milliseconds

Starting up the Python interpreter on my machine has an overhead of about 60 milliseconds.

SCons appears to have an overhead of just under 0.5 seconds. Or, at least, running scons with an empty SConstruct takes about 0.55 seconds on my machine, putting SCons overhead itself at .49 seconds.

import SCons

AddOption is pretty fast too. It’s hard to measure the time accurately, but a file with 10,000 AddOption calls runs in just under 1.0 seconds, so that means AddOption takes just under 50 microseconds per invocation. This means you can pretty much ignore it.

import SCons
AddOption('--opt0', dest='opt0', action='store_true', default=False, help='help for opt0')
AddOption('--opt1', dest='opt1', action='store_true', default=False, help='help for opt1')
...
AddOption('--opt9999', dest='opt9999', action='store_true', default=False, help='help for opt9999')

Creating an environment takes time, around 0.35 seconds on my machine. Tossing an SConstruct with 10 Environment() calls in it runs in 4.01 seconds.

import SCons
env = Environment()
...
env = Environment() # 10 times

The first time you call File() or Dir(), it takes .357 seconds (at least on my machine). Subsequent calls to either are cheap (as in not really measurable). I guess there’s some lazy initialization going on. It made my timing wonky, I kept thinking this was an improvement, but I just shifted the time.