SCons Environment in depth, part 2

This is part 2 of an exploration of the SCons Environment() system. We’re going to talk about ENV (the subset of environment variables for our Environment), Tools and Builders. For other parts in the series, see:


Continuing the exploration of Environment() – so, what about the ENV portion of an SCons Environment? What relationship does that have with the platform settings, discussed at length in part 1?

It appears that it has no relationship, or at least no visible one, when restricted to just defining platform (mandatory) and importing from the external environment. Let’s extend our test SConstruct as follows.

import SCons

def log(name, env, prompt):
  logfile = open(name, 'w')
  print >> logfile, prompt
  print >> logfile, env.Dump()

for plat in ['win32', 'darwin', 'posix', 'cygwin']:
  env = Environment(platform=plat, tools=[], BUILDERS={}, ENV={})
  log('no_env_%s.txt' % plat, env,
      "env = Environment(ENV = {}, platform=%s...)" % plat)

for plat in ['win32', 'darwin', 'posix', 'cygwin']:
  env = Environment(platform=plat, tools=[], BUILDERS={})
  log('env_%s.txt' % plat, env,
  "env = Environment(platform=%s...)" % plat)

When we run this and compare files, the only thing that allowing the environment to populate the ENV field does is – we populate the ENV field. Seems like a letdown.

platform = ‘win32′ when run on a Windows machine.

  'ENV': { 'COMSPEC': 'C:\\windows\\system32\\cmd.exe',
           'PATH': u'C:\\windows\\System32',
           'PATHEXT': '.COM;.EXE;.BAT;.CMD',
           'SystemDrive': 'C:',
           'SystemRoot': 'C:\\windows',
           'TEMP': 'C:\\Users\\BFitz\\AppData\\Local\\Temp',
           'TMP': 'C:\\Users\\BFitz\\AppData\\Local\\Temp'},

PATH and PATHEXT are artificial, and do not contain the contents of the native environment. COMSPEC (listed as ComSpec in the Windows environment output), SystemDrive, SystemRoot, TEMP and TMP are copies of their respective environment values, and are somewhat required for proper operation of binaries.

platform = ‘darwin’ when run on a Windows machine is completely artificial.

  'ENV': { 'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin'},

platform = ‘posix’ when run on a Linux machine is spartan, and actually artificial (this is not what PATH is set to on this machine).

  'ENV': { 'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin'},

platform = ‘darwin’ when run on a Mac OS X machine is still pretty spartan.

  'ENV': { 'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin',
           'PATHOSX': '/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin'},

PATH is artificial, whereas PATHOSX is a copy of the PATH environment variable. Controlling PATH is important, otherwise build consistency can be hard to achieve between machines. But it’s tricky, because you might fail to find things.

So, for some reason, SCons sets up a PATH environment variable that is made up.

All said, there’s little need or effect for environment settings from the machine.

Unanswered questions to date – I see items being added to ENV as part of platform setup, but don’t see them showing up in the Environment itself.

Interaction between ENV and builders

We’re going to jump ahead a little bit. When we start adding builders, those builders cause items to be added to ENV.

‘win32′ platform with default BUILDERS adds a lot of entries to PATH, and also adds INCLUDE, LIB and LIBPATH environment variables.

  'ENV': { 'COMSPEC': 'C:\\windows\\system32\\cmd.exe',
           'INCLUDE': ..., # 5 entries
           'LIB': ..., # 3 entries for amd64 and x64
           'LIBPATH': ..., # 6 entries
           'PATH': ..., # 16 entries
           'PATHEXT': '.COM;.EXE;.BAT;.CMD',
           'SystemDrive': 'C:',
           'SystemRoot': 'C:\\windows',
           'TEMP': 'C:\\Users\\BFitz\\AppData\\Local\\Temp',
           'TMP': 'C:\\Users\\BFitz\\AppData\\Local\\Temp'},

The various builders in the default BUILDERS set find installed tools, and insert their appropriate settings. It feels a bit haphazard, and I’m hoping to find a way to tame it for big builds.

We’ll get back to this when we dive more deeply into tools and builders.

Tools and Builders

SCons is based around the concept of a Tool, which you can think of as a module. Each Tool is the interface to some piece of functionality, and often installs one or more Builders. Tools are also the preferred way for a user to extend SCons – any files inside the user’s site_tools directory are available to be installed as Tools, in one of three ways

  • env = Environment(tools=[…])
  • env.Tool(…)
  • Tool(…)

Let’s look at the default set of tools.  When we run this

env = Environment()

we see this in the output on a Windows machine

  'TOOLS': [ 'default',

and we see this in the output on a Mac OS X machine

  'TOOLS': [ 'default',

and we see this in the output on a Linux machine.

  'TOOLS': [ 'default',

Each of these corresponds to a file of the same name in a directory in the toolpath. SCons’ built-in tools are in the toolpath, of course, and are located in the <scons_dir>/SCons/Tool directory (typically you will have installed SCons into the Python/Lib/site_packages directory). If you look in this directory, you’ll see, and etc.

A user’s build hierarchy can have a site_scons folder located next to its top-level SConstruct file, and a site_tools folder inside this directory will automatically be added to the toolpath. There are other global locations for site_tools folders, but those tend to add to build fragility (tying builds to specific machine setups) and are best avoided.

The ‘default’ tool is interesting. It has a large pre-canned list of tools. Each tool found on the machine is added to the Environment. Note that the ‘default’ tool is left on the Tools list, but the Tools list is nothing more than a manifest of what was found and added. This produces a system that works out of the box for many different build targets, but also increases build time, since much of the work to find tools is repeated on each build. For example, if you don’t build with Fortran, looking for and installing Fortran tools is wasted effort.

Let’s take a quick look at each of the default tools. Or rather, let’s look at some of the default tools, since there are a lot of them (I’m skipping the Java and Fortran tools for now). Our goal is two-fold – what effect does each tool have, and what are the relationships between tools? We’re going to go from simple tools to complex tools.


This tool installs very little. The source for this is SCons/Tool/

    'CopyAs': <SCons.Builder.BuilderBase object at ...>,
    'CopyTo': <SCons.Builder.BuilderBase object at ...>
  'COPYSTR': 'Copy file(s): "$SOURCES" to "$TARGETS"',
  'TOOLS': ['filesystem'],

It installs two Builders, CopyAs and CopyTo, and adds one construction variable to the Environment, COPYSTR, which is used in progress messages. The CopyAs Builder is used by the ‘packaging’ Tool to copy files from one directory to another. The distinction between CopyAs and CopyTo is that CopyTo copies from a source to a target directory (thus leaf names stay the same) whereas CopyAs copies from a source to a target (and leaf names can change).

These Builders are not mentioned in the SCons man page nor in the SCons User’s guide.


This tool installs a little more than the filesystem tool, but not much more. The source for this is SCons/Tool/

    'Zip': <SCons.Builder.BuilderBase object at ...>
  'TOOLS': ['zip'],
  'ZIP': 'zip',
  'ZIPCOM': ,
  'ZIPFLAGS': [],
  'ZIPSUFFIX': '.zip',

It installs one Builder, Zip. The tool tries to use the Python zipfile module, making a wrapper for it that can be called from an Action. If it can’t be found, it assumes there is a zip program named “zip” on the path, and creates a command line to use for the Action. The Builder then uses that Action.

This is a good module to look at in order to understand how to create Actions and Builders, because it is fairly small, but not too simple. It even has default behavior that can be overridden by adding a construction variable (ZIPCOMSTR, although I would have documented that in code a little better).

This points out an interesting weakness in the SCons man page – it lists every variable in one flat list, but most of the variables are of interest to only one Tool or one Builder.


This wraps a Microsoft program named midl.exe that is used to create type libraries. The source for this is SCons/Tool/

    'TypeLibrary': <SCons.Builder.BuilderBase object at ...>
             '/h ${TARGETS[1]} /iid ${TARGETS[2]}'
             '/proxy ${TARGETS[3]} /dlldata ${TARGETS[4]}'
             '$SOURCE 2> NUL'),
  'MIDLFLAGS': ['/nologo'],
  'TOOLS': ['midl'],

This adds one Builder, TypeLibrary. It also adds several configuration variables that are used in the invocation of TypeLibrary, one of which allows for user extension (MIDLFLAGS).

Again, this is a simple builder. However, when we look at the source code, we’ll see the first hint of entanglement between tools. The midl tool’s exists() function calls msvc_exists(), which is also used by the mslib, mslink, msvc and msvs tools. The msvc_exists() function memoizes its answer, so it only has cost the first time it’s called. the downside is that anything needed by any of these tools has to be configured before the first tool is installed. Essentially, all of these are part of a suite, even if they aren’t installed as such.

It’s also interesting that msvc_exists() can detect a specific VC version, and not just that one is installed. Nothing in the code currently uses this, however.

And.. as we’ll see later on, there are environment variables missing. We need a path to midl.exe, and we probably need some other environment variables as well. This is a bug, but not a very big one, because it’s hard to imagine a build that only uses TypeLibrary. Still, it’s wrong. See mslib for what the environment should look like at this point.

SCons needs tool suites. This wouldn’t be that hard to do, you could have a ‘vc’ tool that installs all the related tools by default, and could have one or more parameters to control what is installed.


This wraps the Microsoft assembler masm.exe. The source for this is SCons/Tool/

  'AS': 'ml',
  'ASFLAGS': ['/nologo'],
              '$_CPPINCFLAGS /c /Fo$TARGET $SOURCES'),
    'Object': <SCons.Builder.CompositeBuilder object at ...>,
    'SharedObject': <SCons.Builder.CompositeBuilder object at ...>,
    'StaticObject': <SCons.Builder.CompositeBuilder object at ...>
  'TOOLS': ['masm'],

While slightly more complicated, it’s still on par with the previous tools. We add three Builders: Object, SharedObject and StaticObject. We add command-line strings and flags to control the command-line invocation.

This tool doesn’t call msvc_exists(), and does not add anything to my path (unlike midl, which dumps all the Visual Studio locations into my path). However, this is wrong, because ml.exe (which is what it’s using to implement the masm functionality) is part of Visual C++. I’d say this is a bug, that msvc_exists needs to be called(). Otherwise, a build that just does assembly with the masm tool won’t actually work. I’ll file a bug or a pull request.

Again, I’d like to see documentation that shows each construction variable referenced by any given Tool, not just a flat list.


This wraps the Microsoft librarian lib.exe. The source for this is SCons/Tool/

{ 'AR': 'lib',
  'ARFLAGS': ['/nologo'],
    'Library': <SCons.Builder.BuilderBase object at ...>,
    'StaticLibrary': <SCons.Builder.BuilderBase object at ...>
     'INCLUDE': ..., # a bunch
     'LIB': ..., # a bunch
     'LIBPATH': ..., # a bunch
     'PATH': ..., # a bunch
  'MSVC_VERSION': '11.0',
  'MSVS': { },
  'MSVS_VERSION': '11.0',
  'TARGET_ARCH': 'amd64',
  'TOOLS': ['mslib'],

This adds two Builders: Library and StaticLibrary. It also adds a number of construction variables. I left out enumerating all the paths, includes, libs and libpath entries added. If you want to see them, just run the SConstruct lines I listed above.

This tool ran msvc_exists(), and it also ran msvc_default_version(), and msvc_setup(). This is one-time setup (it won’t do anything in the future since MSVC_SETUP_RUN is set to True), so your chance to influence the specific toolchain is before you install any of these tools. Now, I suppose you could remove MSVC_SETUP_RUN and install again, but that sounds a bit dodgy. We see that it picked Visual Studio 2012 (VC11). I don’t know why MSVS is an empty array, I thought this was supposed to contain all the found versions. Maybe when it selects a specific version, it throws away the other information without recording it.

Note that this builder decided to change the TARGET_ARCH from x86_64 to amd64. That seems odd. I’ll have to look at the code to see why. It’s not a substantial change, but it’s a change I didn’t expect.


This is one of the bigger builders, to drive Microsoft Visual Studio. The source for this is SCons/Tool/

    'MSVSSolution': <SCons.Builder.BuilderBase object at ...>,
    'MSVSProject': <SCons.Builder.BuilderBase object at ...>
  'ENV': { 'INCLUDE': ... }
  'GET_MSVSPROJECTSUFFIX': <function GetMSVSProjectSuffix at ...>,
  'GET_MSVSSOLUTIONSUFFIX': <function GetMSVSSolutionSuffix at ...>,
  'MSVC_VERSION': '11.0',
  'MSVS': { 'PROJECTSUFFIX': '.vcxproj', 'SOLUTIONSUFFIX': '.sln'},
  'MSVSENCODING': 'utf-8',
  'MSVSPROJECTCOM': <SCons.Action.FunctionAction object at ...>,
  'MSVSSCONS': ('"C:\\Python27\\Scripts\\..\\python.exe" '
                '-c "from os.path import join; import sys; '
                'sys.path = [ join(sys.prefix, \'Lib\', '
                '\'site-packages\', \'scons-2.2.0\'), join('
                'sys.prefix, \'scons-2.2.0\'), join(sys.prefix, '
                '\'Lib\', \'site-packages\', \'scons\'), '
                'join(sys.prefix, \'scons\') ] + sys.path; '
                'import SCons.Script; SCons.Script.main()"'),
  'MSVSSCONSCRIPT': <SCons.Node.FS.File object at ...>,
  'MSVSSCONSFLAGS': ('-C "${MSVSSCONSCRIPT.dir.abspath}" '
                     '-f ${}'),
  'MSVSSOLUTIONCOM': <SCons.Action.FunctionAction object at ...>,
  'MSVS_VERSION': '11.0',
  'SCONS_HOME': None,
  'TARGET_ARCH': 'amd64',
  'TOOLS': ['msvs'],

This tool adds two Builders: MSVSSolution, which generates a Visual Studio solution file, and MSVSProject, which generates a Visual Studio project file (and by default, the owning solution file). These do not run msbuild or devenv; these create project files from the set of sources and headers and target and variants that are presented to it.

The built project files just turn around and invoke SCons to do any actual building – see the rather large MSVSSCONS construction variable, which is the heart of the build operation that the project file is told.

There is an optional string SCONS_HOME that, if set before the msvs tool is constructed, is used to create a shorter path for MSVSSCONS. Or, that’s what the documentation says, but I’m not sure why that would be needed in order to avoid the mess of join calls in the MSVSSCONS variable. I also don’t know why this would be in a long command-line rather than in a file that’s invoked.

And most importantly, there’s a crying need for actual real project file generation, so that code builds could be done inside the generated project. Of course, this would be more complicated, because the builders would have to figure out what parts of the build would still need to be done by SCons. But that information is all there…


This drives the Visual Studio compiler, and it’s fair to say that it is moderately complicated. The source for this is SCons/Tool/

    'Object': <SCons.Builder.CompositeBuilder object at ...>,
    'SharedObject': <SCons.Builder.CompositeBuilder object at ...>,
    'StaticObject': <SCons.Builder.CompositeBuilder object at ...>,
    'PCH': <SCons.Builder.BuilderBase object at ...>,
    'RES': <SCons.Builder.BuilderBase object at ...>
  'CC': 'cl',
            '$CFLAGS $CCFLAGS $_CCCOMCOM")}'),
  'CCFLAGS': ['/nologo'],
  'CCPCHFLAGS': [('${(PCH and "/Yu%s \\"/Fp%s\\""%(PCHSTOP or "",'
                  'File(PCH))) or ""}')],
  'CCPDBFLAGS': ['${(PDB and "/Z7") or ""}'],
  'CFILESUFFIX': '.c',
  'CFLAGS': [],
  'CXX': '$CC',
  'CXXFLAGS': ['$(', '/TP', '$)'],
  'ENV': { 'INCLUDE': ... }
  'INCPREFIX': '/I',
  'INCSUFFIX': '',
  'MSVC_VERSION': '11.0',
  'MSVS': { },
  'MSVS_VERSION': '11.0',
  'PCHPDBFLAGS': ['${(PDB and "/Yd") or ""}'],
  'RC': 'rc',
            '$RCFLAGS /fo$TARGET $SOURCES'),
  'RCFLAGS': [],
  'RCSUFFIXES': ['.rc', '.rc2'],
  'SHCC': '$CC',
              '/c $CHANGED_SOURCES '
              '$SHCFLAGS $SHCCFLAGS $_CCCOMCOM")}'),
  'SHCXX': '$CXX',
               '/c $CHANGED_SOURCES $SHCXXFLAGS '
               '$SHCCFLAGS $_CCCOMCOM")}'),
  'TARGET_ARCH': 'amd64',
  'TOOLS': ['msvc'],
                '$CCPCHFLAGS $CCPDBFLAGS'),
  '_MSVC_OUTPUT_FLAG': <function msvc_output_flag at ...>,

By the way, in case it hasn’t been obvious, I’ve been hand-prettifying the output. Hopefully I haven’t introduced any typos. The … are where I have elided unimportant data, usually addresses, but sometimes long strings like paths or includes where the data isn’t really relevant to the discussion at hand.

This adds five builders, a few of which should be familiar. Object, SharedObject and StaticObject are builders that are shared between the masm tool and the msvc tool. The other two builders added here are PCH and RES, which generate precompiled headers and resource files, respectively.

Also note the use of TEMPFILE, which we talked about earlier in the discussion about environments; TEMPFILE is used to handle overly long command-lines. Specifically for us, Windows only gives us a 2048 character command line, so if we end up generating a longer one (and this happens quite easily), we need to wrap it in a file. Note that Visual Studio does this itself, and in fact a lot of the compile from devenv is driven through *.RSP files for that reason, but also that it makes debugging the build process a little easier (you have a record of what was going on).

Let’s talk a little about Python syntax you may be unfamiliar with (hopefully not, since you need to know Python to use SCons).

'PCHPDBFLAGS': ['${(PDB and "/Yd") or ""}']

In Python, the use of ‘and’ and ‘or’ gives the last value encountered, whereas && and || return True or False. So ‘PDB and “/Yd”‘ will either give the value of PDB or the value “/Yd”, and ‘(PDB and “/Yd”) or “”‘ will evaluate to “/Yd” if PBD is a True value, or “” if PDB is a False value.

Let’s break down a more complicated expression:

  'CCPCHFLAGS': [('${(PCH and "/Yu%s \\"/Fp%s\\""%(PCHSTOP or "",'
                  'File(PCH))) or ""}')],

This is a little hard to read because of the escaped backslash characters which only exist to make sure the double-quote characters make it into the final output, so I’m going to turn them into characters that preserve meaning even if not quite kosher to the compiler (because we’re smarter than compilers, even still).

${(PCH and '/Yu%s "/Fp%s"'%(PCHSTOP or "",File(PCH))) or ""}

This either turns into a string, or the empty string “”. If PCH is true, then we have the format string ‘/Yu%s “/Fp%s”‘ and the two values ‘PCHSTOP or “”‘ and File(PCH). So we either have ‘/Yu “/Fp”‘, or ‘/Yu “/Fp”‘, or ”, as the result.

I’m assuming there’s a good reason why /Fp needs to be quoted. Perhaps this is drive-by collateral damage from use of the ESCAPE function (which, if you remember, is double-quoting its argument).

Also, this is a good reason why you want to have more white-space in the right places, so it’s slightly easier to read complex expressions, and even (or especially when) your complex expressions are generated by code.


This drives the Microsoft linker. The source for this is SCons/Tool/

    'LoadableModule': <SCons.Builder.BuilderBase object at ...>,
    'Program': <SCons.Builder.BuilderBase object at ...>,
    'SharedLibrary': <SCons.Builder.BuilderBase object at ...>
  'ENV': { 'INCLUDE': ... }
  'LDMODULECOM': <SCons.Action.ListAction object at ...>,
  'LDMODULEEMITTER': [<function ldmodEmitter at ...>],
  'LINK': 'link',
  'LINKCOM': <SCons.Action.ListAction object at ...>,
  'LINKFLAGS': ['/nologo'],
  'MSVC_VERSION': '11.0',
  'MSVS': { },
  'MSVS_VERSION': '11.0',
  'MT': 'mt',
  'MTEXECOM': ('-$MT $MTFLAGS -manifest ${TARGET}.manifest '
               '$_MANIFEST_SOURCES -outputresource:$TARGET;1'),
  'MTFLAGS': ['/nologo'],
                 '-manifest ${TARGET}.manifest $_MANIFEST_SOURCES '
  'PROGEMITTER': [<function prog_emitter at ...>],
  'REGSVR': u'C:\\windows\\System32\\regsvr32',
  'REGSVRACTION': <SCons.Action.FunctionAction object at ...>,
  'REGSVRFLAGS': '/s ',
  'SHLIBEMITTER': [<function windowsLibEmitter at ...>],
  'SHLINK': '$LINK',
  'SHLINKCOM': <SCons.Action.ListAction object at ...>,
  'SHLINKFLAGS': ['$LINKFLAGS', '/dll'],
  'TARGET_ARCH': 'amd64',
  'TOOLS': ['mslink'],
  'WIN32DEFSUFFIX': '.def',
  'WIN32EXPSUFFIX': '.exp',
  'WIN32_INSERT_DEF': 0,
  '_LDMODULE_SOURCES': <function _windowsLdmodSources at ...>,
  '_LDMODULE_TARGETS': <function _windowsLdmodTargets at ...>,
  '_PDB': <function pdbGenerator at ...>,
  '_SHLINK_SOURCES': <function windowsShlinkSources at ...>,
  '_SHLINK_TARGETS': <function windowsShlinkTargets at ...>,

Yeah. That’s a lot of stuff.

This introduces three builders: LoadableModule, Program, and SharedLibrary. This is actually a bit of an omnibus tool, since it comprises the linker proper (link.exe), the manifest generation tool (mt.exe), and Regsvr32 which registers COM DLLs (regsrvr.exe).

Yes, SCons will register your built shared library if you want it to. This requires you to set the ‘register’ construction variable to 1, and then when a DLL is built, it will be registered with Regsrvr32; this is needed if you’re working with COM or ActiveX.

One thought on “SCons Environment in depth, part 2”

  1. Hi Brian,

    you seem to have analyzed SCons’ internals a lot, and from what I can see you did a good job. Would you be willing to contribute some of your descriptions to our Wiki? Then please reply either to me (PM), or contact the project via the SCons user mailing list.

    Regarding SCons’ speed performance you might want to have a look at .

    Best regards,


Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>