Monthly Archives: August 2013

Mac OS X

Show hidden files/folders in Mac OS X Finder

Leave a comment

I always forget how to do this, and fortunately Google and Stack Exchange have my back

defaults write com.apple.Finder AppleShowAllFiles YES; killall -HUP Finder

If by some reason you want the default behavior back, set AppleshowAllFiles back to NO (or False, either works).

http://apple.stackexchange.com/questions/58709/how-can-i-view-hidden-files-in-the-finder

 

Python

Deconstructing Deferred

Leave a comment

This is a great thread about Twisted’s Deferred object, used to manage callbacks.

https://groups.google.com/forum/#!topic/python-tulip/ut4vTG-08k8/discussion

Patterns for clear and yet robust async programming are very important. This was a great write-up by Guido van Rossum explaining Deferred, and inferring its design decisions.

The actual Deferred documentation is here: http://twistedmatrix.com/documents/12.3.0/core/howto/defer.html

Build systems

SCons Environment in depth, part 2

1 Comment

This is part 2 of an exploration of the SCons Environment() system. We’re going to talk about ENV (the subset of environment variables for our Environment), Tools and Builders. For other parts in the series, see:

ENV

Continuing the exploration of Environment() – so, what about the ENV portion of an SCons Environment? What relationship does that have with the platform settings, discussed at length in part 1?

It appears that it has no relationship, or at least no visible one, when restricted to just defining platform (mandatory) and importing from the external environment. Let’s extend our test SConstruct as follows.

import SCons

def log(name, env, prompt):
  logfile = open(name, 'w')
  print >> logfile, prompt
  print >> logfile, env.Dump()
  logfile.close

for plat in ['win32', 'darwin', 'posix', 'cygwin']:
  env = Environment(platform=plat, tools=[], BUILDERS={}, ENV={})
  log('no_env_%s.txt' % plat, env,
      "env = Environment(ENV = {}, platform=%s...)" % plat)

for plat in ['win32', 'darwin', 'posix', 'cygwin']:
  env = Environment(platform=plat, tools=[], BUILDERS={})
  log('env_%s.txt' % plat, env,
  "env = Environment(platform=%s...)" % plat)

When we run this and compare files, the only thing that allowing the environment to populate the ENV field does is – we populate the ENV field. Seems like a letdown.

platform = ‘win32′ when run on a Windows machine.

  'ENV': { 'COMSPEC': 'C:\\windows\\system32\\cmd.exe',
           'PATH': u'C:\\windows\\System32',
           'PATHEXT': '.COM;.EXE;.BAT;.CMD',
           'SystemDrive': 'C:',
           'SystemRoot': 'C:\\windows',
           'TEMP': 'C:\\Users\\BFitz\\AppData\\Local\\Temp',
           'TMP': 'C:\\Users\\BFitz\\AppData\\Local\\Temp'},

PATH and PATHEXT are artificial, and do not contain the contents of the native environment. COMSPEC (listed as ComSpec in the Windows environment output), SystemDrive, SystemRoot, TEMP and TMP are copies of their respective environment values, and are somewhat required for proper operation of binaries.

platform = ‘darwin’ when run on a Windows machine is completely artificial.

  'ENV': { 'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin'},

platform = ‘posix’ when run on a Linux machine is spartan, and actually artificial (this is not what PATH is set to on this machine).

  'ENV': { 'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin'},

platform = ‘darwin’ when run on a Mac OS X machine is still pretty spartan.

  'ENV': { 'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin',
           'PATHOSX': '/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin'},

PATH is artificial, whereas PATHOSX is a copy of the PATH environment variable. Controlling PATH is important, otherwise build consistency can be hard to achieve between machines. But it’s tricky, because you might fail to find things.

So, for some reason, SCons sets up a PATH environment variable that is made up.

All said, there’s little need or effect for environment settings from the machine.

Unanswered questions to date – I see items being added to ENV as part of platform setup, but don’t see them showing up in the Environment itself.

Interaction between ENV and builders

We’re going to jump ahead a little bit. When we start adding builders, those builders cause items to be added to ENV.

‘win32′ platform with default BUILDERS adds a lot of entries to PATH, and also adds INCLUDE, LIB and LIBPATH environment variables.

  'ENV': { 'COMSPEC': 'C:\\windows\\system32\\cmd.exe',
           'INCLUDE': ..., # 5 entries
           'LIB': ..., # 3 entries for amd64 and x64
           'LIBPATH': ..., # 6 entries
           'PATH': ..., # 16 entries
           'PATHEXT': '.COM;.EXE;.BAT;.CMD',
           'SystemDrive': 'C:',
           'SystemRoot': 'C:\\windows',
           'TEMP': 'C:\\Users\\BFitz\\AppData\\Local\\Temp',
           'TMP': 'C:\\Users\\BFitz\\AppData\\Local\\Temp'},

The various builders in the default BUILDERS set find installed tools, and insert their appropriate settings. It feels a bit haphazard, and I’m hoping to find a way to tame it for big builds.

We’ll get back to this when we dive more deeply into tools and builders.

Tools and Builders

SCons is based around the concept of a Tool, which you can think of as a module. Each Tool is the interface to some piece of functionality, and often installs one or more Builders. Tools are also the preferred way for a user to extend SCons – any files inside the user’s site_tools directory are available to be installed as Tools, in one of three ways

  • env = Environment(tools=[…])
  • env.Tool(…)
  • Tool(…)

Let’s look at the default set of tools.  When we run this

env = Environment()
env.Dump()

we see this in the output on a Windows machine

  'TOOLS': [ 'default',
             'mslink',
             'msvc',
             'gfortran',
             'masm',
             'mslib',
             'msvs',
             'midl',
             'filesystem',
             'jar',
             'javac',
             'rmic',
             'zip'],

and we see this in the output on a Mac OS X machine

  'TOOLS': [ 'default',
             'applelink',
             'gcc',
             'g++',
             'gfortran',
             'as',
             'ar',
             'filesystem',
             'm4',
             'lex',
             'yacc',
             'rpcgen',
             'jar',
             'javac',
             'javah',
             'rmic',
             'tar',
             'zip',
             'CVS',
             'RCS'],

and we see this in the output on a Linux machine.

  'TOOLS': [ 'default',
             'gnulink',
             'gcc',
             'g++',
             'gfortran',
             'gas',
             'ar',
             'filesystem',
             'm4',
             'rpcgen',
             'jar',
             'javac',
             'rmic',
             'tar',
             'zip',
             'rpm'],

Each of these corresponds to a file of the same name in a directory in the toolpath. SCons’ built-in tools are in the toolpath, of course, and are located in the <scons_dir>/SCons/Tool directory (typically you will have installed SCons into the Python/Lib/site_packages directory). If you look in this directory, you’ll see default.py, mslink.py and etc.

A user’s build hierarchy can have a site_scons folder located next to its top-level SConstruct file, and a site_tools folder inside this directory will automatically be added to the toolpath. There are other global locations for site_tools folders, but those tend to add to build fragility (tying builds to specific machine setups) and are best avoided.

The ‘default’ tool is interesting. It has a large pre-canned list of tools. Each tool found on the machine is added to the Environment. Note that the ‘default’ tool is left on the Tools list, but the Tools list is nothing more than a manifest of what was found and added. This produces a system that works out of the box for many different build targets, but also increases build time, since much of the work to find tools is repeated on each build. For example, if you don’t build with Fortran, looking for and installing Fortran tools is wasted effort.

Let’s take a quick look at each of the default tools. Or rather, let’s look at some of the default tools, since there are a lot of them (I’m skipping the Java and Fortran tools for now). Our goal is two-fold – what effect does each tool have, and what are the relationships between tools? We’re going to go from simple tools to complex tools.

filesystem

This tool installs very little. The source for this is SCons/Tool/filesystem.py.

  'BUILDERS':
  {
    'CopyAs': <SCons.Builder.BuilderBase object at ...>,
    'CopyTo': <SCons.Builder.BuilderBase object at ...>
  },
  'COPYSTR': 'Copy file(s): "$SOURCES" to "$TARGETS"',
  'TOOLS': ['filesystem'],

It installs two Builders, CopyAs and CopyTo, and adds one construction variable to the Environment, COPYSTR, which is used in progress messages. The CopyAs Builder is used by the ‘packaging’ Tool to copy files from one directory to another. The distinction between CopyAs and CopyTo is that CopyTo copies from a source to a target directory (thus leaf names stay the same) whereas CopyAs copies from a source to a target (and leaf names can change).

These Builders are not mentioned in the SCons man page nor in the SCons User’s guide.

zip

This tool installs a little more than the filesystem tool, but not much more. The source for this is SCons/Tool/zip.py.

  'BUILDERS':
  {
    'Zip': <SCons.Builder.BuilderBase object at ...>
  },
  'TOOLS': ['zip'],
  'ZIP': 'zip',
  'ZIPCOM': ,
  'ZIPCOMPRESSION': 8,
  'ZIPFLAGS': [],
  'ZIPSUFFIX': '.zip',

It installs one Builder, Zip. The tool tries to use the Python zipfile module, making a wrapper for it that can be called from an Action. If it can’t be found, it assumes there is a zip program named “zip” on the path, and creates a command line to use for the Action. The Builder then uses that Action.

This is a good module to look at in order to understand how to create Actions and Builders, because it is fairly small, but not too simple. It even has default behavior that can be overridden by adding a construction variable (ZIPCOMSTR, although I would have documented that in code a little better).

This points out an interesting weakness in the SCons man page – it lists every variable in one flat list, but most of the variables are of interest to only one Tool or one Builder.

midl

This wraps a Microsoft program named midl.exe that is used to create type libraries. The source for this is SCons/Tool/midl.py.

  'BUILDERS':
  {
    'TypeLibrary': <SCons.Builder.BuilderBase object at ...>
  },
  'MIDL': 'MIDL.EXE',
  'MIDLCOM': ('$MIDL $MIDLFLAGS /tlb ${TARGETS[0]}'
             '/h ${TARGETS[1]} /iid ${TARGETS[2]}'
             '/proxy ${TARGETS[3]} /dlldata ${TARGETS[4]}'
             '$SOURCE 2> NUL'),
  'MIDLFLAGS': ['/nologo'],
  'TOOLS': ['midl'],

This adds one Builder, TypeLibrary. It also adds several configuration variables that are used in the invocation of TypeLibrary, one of which allows for user extension (MIDLFLAGS).

Again, this is a simple builder. However, when we look at the source code, we’ll see the first hint of entanglement between tools. The midl tool’s exists() function calls msvc_exists(), which is also used by the mslib, mslink, msvc and msvs tools. The msvc_exists() function memoizes its answer, so it only has cost the first time it’s called. the downside is that anything needed by any of these tools has to be configured before the first tool is installed. Essentially, all of these are part of a suite, even if they aren’t installed as such.

It’s also interesting that msvc_exists() can detect a specific VC version, and not just that one is installed. Nothing in the code currently uses this, however.

And.. as we’ll see later on, there are environment variables missing. We need a path to midl.exe, and we probably need some other environment variables as well. This is a bug, but not a very big one, because it’s hard to imagine a build that only uses TypeLibrary. Still, it’s wrong. See mslib for what the environment should look like at this point.

SCons needs tool suites. This wouldn’t be that hard to do, you could have a ‘vc’ tool that installs all the related tools by default, and could have one or more parameters to control what is installed.

masm

This wraps the Microsoft assembler masm.exe. The source for this is SCons/Tool/masm.py.

  'AS': 'ml',
  'ASCOM': '$AS $ASFLAGS /c /Fo$TARGET $SOURCES',
  'ASFLAGS': ['/nologo'],
  'ASPPCOM': ('$CC $ASPPFLAGS $CPPFLAGS $_CPPDEFFLAGS'
              '$_CPPINCFLAGS /c /Fo$TARGET $SOURCES'),
  'ASPPFLAGS': '$ASFLAGS',
  'BUILDERS':
  {
    'Object': <SCons.Builder.CompositeBuilder object at ...>,
    'SharedObject': <SCons.Builder.CompositeBuilder object at ...>,
    'StaticObject': <SCons.Builder.CompositeBuilder object at ...>
  },
  'STATIC_AND_SHARED_OBJECTS_ARE_THE_SAME': 1,
  'TOOLS': ['masm'],

While slightly more complicated, it’s still on par with the previous tools. We add three Builders: Object, SharedObject and StaticObject. We add command-line strings and flags to control the command-line invocation.

This tool doesn’t call msvc_exists(), and does not add anything to my path (unlike midl, which dumps all the Visual Studio locations into my path). However, this is wrong, because ml.exe (which is what it’s using to implement the masm functionality) is part of Visual C++. I’d say this is a bug, that msvc_exists needs to be called(). Otherwise, a build that just does assembly with the masm tool won’t actually work. I’ll file a bug or a pull request.

Again, I’d like to see documentation that shows each construction variable referenced by any given Tool, not just a flat list.

mslib

This wraps the Microsoft librarian lib.exe. The source for this is SCons/Tool/mslib.py.

{ 'AR': 'lib',
  'ARCOM': "${TEMPFILE('$AR $ARFLAGS /OUT:$TARGET $SOURCES')}",
  'ARFLAGS': ['/nologo'],
  'BUILDERS':
  {
    'Library': <SCons.Builder.BuilderBase object at ...>,
    'StaticLibrary': <SCons.Builder.BuilderBase object at ...>
  },
  'ENV':
   {
     ...
     'INCLUDE': ..., # a bunch
     'LIB': ..., # a bunch
     'LIBPATH': ..., # a bunch
     'PATH': ..., # a bunch
  }
  'MSVC_SETUP_RUN': True,
  'MSVC_VERSION': '11.0',
  'MSVS': { },
  'MSVS_VERSION': '11.0',
  'TARGET_ARCH': 'amd64',
  'TOOLS': ['mslib'],

This adds two Builders: Library and StaticLibrary. It also adds a number of construction variables. I left out enumerating all the paths, includes, libs and libpath entries added. If you want to see them, just run the SConstruct lines I listed above.

This tool ran msvc_exists(), and it also ran msvc_default_version(), and msvc_setup(). This is one-time setup (it won’t do anything in the future since MSVC_SETUP_RUN is set to True), so your chance to influence the specific toolchain is before you install any of these tools. Now, I suppose you could remove MSVC_SETUP_RUN and install again, but that sounds a bit dodgy. We see that it picked Visual Studio 2012 (VC11). I don’t know why MSVS is an empty array, I thought this was supposed to contain all the found versions. Maybe when it selects a specific version, it throws away the other information without recording it.

Note that this builder decided to change the TARGET_ARCH from x86_64 to amd64. That seems odd. I’ll have to look at the code to see why. It’s not a substantial change, but it’s a change I didn’t expect.

msvs

This is one of the bigger builders, to drive Microsoft Visual Studio. The source for this is SCons/Tool/msvs.py.

  'BUILDERS':
  {
    'MSVSSolution': <SCons.Builder.BuilderBase object at ...>,
    'MSVSProject': <SCons.Builder.BuilderBase object at ...>
  },
  'ENV': { 'INCLUDE': ... }
  'GET_MSVSPROJECTSUFFIX': <function GetMSVSProjectSuffix at ...>,
  'GET_MSVSSOLUTIONSUFFIX': <function GetMSVSSolutionSuffix at ...>,
  'MSVC_SETUP_RUN': True,
  'MSVC_VERSION': '11.0',
  'MSVS': { 'PROJECTSUFFIX': '.vcxproj', 'SOLUTIONSUFFIX': '.sln'},
  'MSVSBUILDCOM': '$MSVSSCONSCOM "$MSVSBUILDTARGET"',
  'MSVSCLEANCOM': '$MSVSSCONSCOM -c "$MSVSBUILDTARGET"',
  'MSVSENCODING': 'utf-8',
  'MSVSPROJECTCOM': <SCons.Action.FunctionAction object at ...>,
  'MSVSPROJECTSUFFIX': '${GET_MSVSPROJECTSUFFIX}',
  'MSVSREBUILDCOM': '$MSVSSCONSCOM "$MSVSBUILDTARGET"',
  'MSVSSCONS': ('"C:\\Python27\\Scripts\\..\\python.exe" '
                '-c "from os.path import join; import sys; '
                'sys.path = [ join(sys.prefix, \'Lib\', '
                '\'site-packages\', \'scons-2.2.0\'), join('
                'sys.prefix, \'scons-2.2.0\'), join(sys.prefix, '
                '\'Lib\', \'site-packages\', \'scons\'), '
                'join(sys.prefix, \'scons\') ] + sys.path; '
                'import SCons.Script; SCons.Script.main()"'),
  'MSVSSCONSCOM': '$MSVSSCONS $MSVSSCONSFLAGS',
  'MSVSSCONSCRIPT': <SCons.Node.FS.File object at ...>,
  'MSVSSCONSFLAGS': ('-C "${MSVSSCONSCRIPT.dir.abspath}" '
                     '-f ${MSVSSCONSCRIPT.name}'),
  'MSVSSOLUTIONCOM': <SCons.Action.FunctionAction object at ...>,
  'MSVSSOLUTIONSUFFIX': '${GET_MSVSSOLUTIONSUFFIX}',
  'MSVS_VERSION': '11.0',
  'SCONS_HOME': None,
  'TARGET_ARCH': 'amd64',
  'TOOLS': ['msvs'],

This tool adds two Builders: MSVSSolution, which generates a Visual Studio solution file, and MSVSProject, which generates a Visual Studio project file (and by default, the owning solution file). These do not run msbuild or devenv; these create project files from the set of sources and headers and target and variants that are presented to it.

The built project files just turn around and invoke SCons to do any actual building – see the rather large MSVSSCONS construction variable, which is the heart of the build operation that the project file is told.

There is an optional string SCONS_HOME that, if set before the msvs tool is constructed, is used to create a shorter path for MSVSSCONS. Or, that’s what the documentation says, but I’m not sure why that would be needed in order to avoid the mess of join calls in the MSVSSCONS variable. I also don’t know why this would be in a long command-line rather than in a file that’s invoked.

And most importantly, there’s a crying need for actual real project file generation, so that code builds could be done inside the generated project. Of course, this would be more complicated, because the builders would have to figure out what parts of the build would still need to be done by SCons. But that information is all there…

msvc

This drives the Visual Studio compiler, and it’s fair to say that it is moderately complicated. The source for this is SCons/Tool/msvs.py.

  'BUILDERS':
  {
    'Object': <SCons.Builder.CompositeBuilder object at ...>,
    'SharedObject': <SCons.Builder.CompositeBuilder object at ...>,
    'StaticObject': <SCons.Builder.CompositeBuilder object at ...>,
    'PCH': <SCons.Builder.BuilderBase object at ...>,
    'RES': <SCons.Builder.BuilderBase object at ...>
  },
  'CC': 'cl',
  'CCCOM': ('${TEMPFILE("$CC $_MSVC_OUTPUT_FLAG /c $CHANGED_SOURCES '
            '$CFLAGS $CCFLAGS $_CCCOMCOM")}'),
  'CCFLAGS': ['/nologo'],
  'CCPCHFLAGS': [('${(PCH and "/Yu%s \\"/Fp%s\\""%(PCHSTOP or "",'
                  'File(PCH))) or ""}')],
  'CCPDBFLAGS': ['${(PDB and "/Z7") or ""}'],
  'CFILESUFFIX': '.c',
  'CFLAGS': [],
  'CPPDEFPREFIX': '/D',
  'CPPDEFSUFFIX': '',
  'CXX': '$CC',
  'CXXCOM': ('${TEMPFILE("$CXX $_MSVC_OUTPUT_FLAG '
             '/c $CHANGED_SOURCES $CXXFLAGS $CCFLAGS $_CCCOMCOM")}'),
  'CXXFILESUFFIX': '.cc',
  'CXXFLAGS': ['$(', '/TP', '$)'],
  'ENV': { 'INCLUDE': ... }
  'INCPREFIX': '/I',
  'INCSUFFIX': '',
  'MSVC_SETUP_RUN': True,
  'MSVC_VERSION': '11.0',
  'MSVS': { },
  'MSVS_VERSION': '11.0',
  'PCHCOM': ('$CXX /Fo${TARGETS[1]} $CXXFLAGS $CCFLAGS '
             '$CPPFLAGS $_CPPDEFFLAGS $_CPPINCFLAGS /c $SOURCES '
             '/Yc$PCHSTOP /Fp${TARGETS[0]} $CCPDBFLAGS $PCHPDBFLAGS'),
  'PCHPDBFLAGS': ['${(PDB and "/Yd") or ""}'],
  'RC': 'rc',
  'RCCOM': ('$RC $_CPPDEFFLAGS $_CPPINCFLAGS '
            '$RCFLAGS /fo$TARGET $SOURCES'),
  'RCFLAGS': [],
  'RCSUFFIXES': ['.rc', '.rc2'],
  'SHCC': '$CC',
  'SHCCCOM': ('${TEMPFILE("$SHCC $_MSVC_OUTPUT_FLAG '
              '/c $CHANGED_SOURCES '
              '$SHCFLAGS $SHCCFLAGS $_CCCOMCOM")}'),
  'SHCCFLAGS': ['$CCFLAGS'],
  'SHCFLAGS': ['$CFLAGS'],
  'SHCXX': '$CXX',
  'SHCXXCOM': ('${TEMPFILE("$SHCXX $_MSVC_OUTPUT_FLAG '
               '/c $CHANGED_SOURCES $SHCXXFLAGS '
               '$SHCCFLAGS $_CCCOMCOM")}'),
  'SHCXXFLAGS': ['$CXXFLAGS'],
  'STATIC_AND_SHARED_OBJECTS_ARE_THE_SAME': 1,
  'TARGET_ARCH': 'amd64',
  'TOOLS': ['msvc'],
  '_CCCOMCOM': ('$CPPFLAGS $_CPPDEFFLAGS $_CPPINCFLAGS '
                '$CCPCHFLAGS $CCPDBFLAGS'),
  '_MSVC_OUTPUT_FLAG': <function msvc_output_flag at ...>,

By the way, in case it hasn’t been obvious, I’ve been hand-prettifying the output. Hopefully I haven’t introduced any typos. The … are where I have elided unimportant data, usually addresses, but sometimes long strings like paths or includes where the data isn’t really relevant to the discussion at hand.

This adds five builders, a few of which should be familiar. Object, SharedObject and StaticObject are builders that are shared between the masm tool and the msvc tool. The other two builders added here are PCH and RES, which generate precompiled headers and resource files, respectively.

Also note the use of TEMPFILE, which we talked about earlier in the discussion about environments; TEMPFILE is used to handle overly long command-lines. Specifically for us, Windows only gives us a 2048 character command line, so if we end up generating a longer one (and this happens quite easily), we need to wrap it in a file. Note that Visual Studio does this itself, and in fact a lot of the compile from devenv is driven through *.RSP files for that reason, but also that it makes debugging the build process a little easier (you have a record of what was going on).

Let’s talk a little about Python syntax you may be unfamiliar with (hopefully not, since you need to know Python to use SCons).

'PCHPDBFLAGS': ['${(PDB and "/Yd") or ""}']

In Python, the use of ‘and’ and ‘or’ gives the last value encountered, whereas && and || return True or False. So ‘PDB and “/Yd”‘ will either give the value of PDB or the value “/Yd”, and ‘(PDB and “/Yd”) or “”‘ will evaluate to “/Yd” if PBD is a True value, or “” if PDB is a False value.

Let’s break down a more complicated expression:

  'CCPCHFLAGS': [('${(PCH and "/Yu%s \\"/Fp%s\\""%(PCHSTOP or "",'
                  'File(PCH))) or ""}')],

This is a little hard to read because of the escaped backslash characters which only exist to make sure the double-quote characters make it into the final output, so I’m going to turn them into characters that preserve meaning even if not quite kosher to the compiler (because we’re smarter than compilers, even still).

${(PCH and '/Yu%s "/Fp%s"'%(PCHSTOP or "",File(PCH))) or ""}

This either turns into a string, or the empty string “”. If PCH is true, then we have the format string ‘/Yu%s “/Fp%s”‘ and the two values ‘PCHSTOP or “”‘ and File(PCH). So we either have ‘/Yu “/Fp”‘, or ‘/Yu “/Fp”‘, or ”, as the result.

I’m assuming there’s a good reason why /Fp needs to be quoted. Perhaps this is drive-by collateral damage from use of the ESCAPE function (which, if you remember, is double-quoting its argument).

Also, this is a good reason why you want to have more white-space in the right places, so it’s slightly easier to read complex expressions, and even (or especially when) your complex expressions are generated by code.

mslink

This drives the Microsoft linker. The source for this is SCons/Tool/msvs.py.

  'BUILDERS':
  {
    'LoadableModule': <SCons.Builder.BuilderBase object at ...>,
    'Program': <SCons.Builder.BuilderBase object at ...>,
    'SharedLibrary': <SCons.Builder.BuilderBase object at ...>
  },
  'ENV': { 'INCLUDE': ... }
  'LDMODULE': '$SHLINK',
  'LDMODULECOM': <SCons.Action.ListAction object at ...>,
  'LDMODULEEMITTER': [<function ldmodEmitter at ...>],
  'LDMODULEFLAGS': '$SHLINKFLAGS',
  'LDMODULEPREFIX': '$SHLIBPREFIX',
  'LDMODULESUFFIX': '$SHLIBSUFFIX',
  'LIBDIRPREFIX': '/LIBPATH:',
  'LIBDIRSUFFIX': '',
  'LIBLINKPREFIX': '',
  'LIBLINKSUFFIX': '$LIBSUFFIX',
  'LINK': 'link',
  'LINKCOM': <SCons.Action.ListAction object at ...>,
  'LINKFLAGS': ['/nologo'],
  'MSVC_SETUP_RUN': True,
  'MSVC_VERSION': '11.0',
  'MSVS': { },
  'MSVS_VERSION': '11.0',
  'MT': 'mt',
  'MTEXECOM': ('-$MT $MTFLAGS -manifest ${TARGET}.manifest '
               '$_MANIFEST_SOURCES -outputresource:$TARGET;1'),
  'MTFLAGS': ['/nologo'],
  'MTSHLIBCOM': ('-$MT $MTFLAGS '
                 '-manifest ${TARGET}.manifest $_MANIFEST_SOURCES '
                 '-outputresource:$TARGET;2'),
  'PROGEMITTER': [<function prog_emitter at ...>],
  'REGSVR': u'C:\\windows\\System32\\regsvr32',
  'REGSVRACTION': <SCons.Action.FunctionAction object at ...>,
  'REGSVRCOM': '$REGSVR $REGSVRFLAGS ${TARGET.windows}',
  'REGSVRFLAGS': '/s ',
  'SHLIBEMITTER': [<function windowsLibEmitter at ...>],
  'SHLINK': '$LINK',
  'SHLINKCOM': <SCons.Action.ListAction object at ...>,
  'SHLINKFLAGS': ['$LINKFLAGS', '/dll'],
  'TARGET_ARCH': 'amd64',
  'TOOLS': ['mslink'],
  'WIN32DEFPREFIX': '',
  'WIN32DEFSUFFIX': '.def',
  'WIN32EXPPREFIX': '',
  'WIN32EXPSUFFIX': '.exp',
  'WIN32_INSERT_DEF': 0,
  'WINDOWSDEFPREFIX': '${WIN32DEFPREFIX}',
  'WINDOWSDEFSUFFIX': '${WIN32DEFSUFFIX}',
  'WINDOWSEXPPREFIX': '${WIN32EXPPREFIX}',
  'WINDOWSEXPSUFFIX': '${WIN32EXPSUFFIX}',
  'WINDOWSPROGMANIFESTPREFIX': '',
  'WINDOWSPROGMANIFESTSUFFIX': '${PROGSUFFIX}.manifest',
  'WINDOWSSHLIBMANIFESTPREFIX': '',
  'WINDOWSSHLIBMANIFESTSUFFIX': '${SHLIBSUFFIX}.manifest',
  'WINDOWS_EMBED_MANIFEST': 0,
  'WINDOWS_INSERT_DEF': '${WIN32_INSERT_DEF}',
  '_LDMODULE_SOURCES': <function _windowsLdmodSources at ...>,
  '_LDMODULE_TARGETS': <function _windowsLdmodTargets at ...>,
  '_MANIFEST_SOURCES': None,
  '_PDB': <function pdbGenerator at ...>,
  '_SHLINK_SOURCES': <function windowsShlinkSources at ...>,
  '_SHLINK_TARGETS': <function windowsShlinkTargets at ...>,

Yeah. That’s a lot of stuff.

This introduces three builders: LoadableModule, Program, and SharedLibrary. This is actually a bit of an omnibus tool, since it comprises the linker proper (link.exe), the manifest generation tool (mt.exe), and Regsvr32 which registers COM DLLs (regsrvr.exe).

Yes, SCons will register your built shared library if you want it to. This requires you to set the ‘register’ construction variable to 1, and then when a DLL is built, it will be registered with Regsrvr32; this is needed if you’re working with COM or ActiveX.

Build systems

SCons Environment in depth, part 1

Leave a comment

Let’s tear down Environment(), so we know how to use it properly. As much as possible, I’m looking at things as a user, so I’ve been trying things without looking at the source code of SCons. I want to see how things actually work, as opposed to what the SCons developers intended.

For other parts in the series, see:

Minimal environment

Looking at a somewhat minimal Environment in SCons (we have to supply a platform, we’ll get to that in a moment) like so:

env = Environment(tools=[], platform='win32', BUILDERS={}, ENV={})
env.Dump()

and re-arranging for prettiness, we have

env = Environment(tools=[], platform='win32', builders={}), ENV={}
{
  'BUILDERS': {},
  'ENV': { },
  'SCANNERS': [],

  'CONFIGUREDIR': '#/.sconf_temp',
  'CONFIGURELOG': '#/config.log',
  'MAXLINELENGTH': 2048,
  'TEMPFILE': <class 'SCons.Platform.TempFileMunge'>,
  'TEMPFILEPREFIX': '@',

  'HOST_ARCH': 'x86_64',
  'HOST_OS': 'win32',
  'PLATFORM': 'win32',
  'TARGET_ARCH': 'x86_64',
  'TARGET_OS': 'win32',

  'LIBPREFIX': '',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.lib',
  'LIBSUFFIXES': ['$LIBSUFFIX'],
  'OBJPREFIX': '',
  'OBJSUFFIX': '.obj',
  'PROGPREFIX': '',
  'PROGSUFFIX': '.exe',
  'SHLIBPREFIX': '',
  'SHLIBSUFFIX': '.dll',
  'SHOBJPREFIX': '$OBJPREFIX',
  'SHOBJSUFFIX': '$OBJSUFFIX',

  'CPPSUFFIXES': [ '.c',
                   '.C',
                   '.cxx',
                   '.cpp',
                   '.c++',
                   '.cc',
                   '.h',
                   '.H',
                   '.hxx',
                   '.hpp',
                   '.hh',
                   '.F',
                   '.fpp',
                   '.FPP',
                   '.m',
                   '.mm',
                   '.S',
                   '.spp',
                   '.SPP',
                   '.sx'],
  'DSUFFIXES': ['.d'],
  'IDLSUFFIXES': ['.idl', '.IDL'],

  '_CPPDEFFLAGS': '${_defines(CPPDEFPREFIX, CPPDEFINES,
                              CPPDEFSUFFIX, __env__)}',
  '_CPPINCFLAGS': '$( ${_concat(INCPREFIX, CPPPATH, INCSUFFIX,
                                __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBDIRFLAGS': '$( ${_concat(LIBDIRPREFIX, LIBPATH, LIBDIRSUFFIX,
                                __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBFLAGS': '${_concat(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX,
                          __env__)}',

  'Dir': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'Dirs': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'ESCAPE': <function escape at ...>,
  'File': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'PSPAWN': <function piped_spawn at 0x025E64F0>,
  'RDirs': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'SHELL': u'C:\\windows\\System32\\cmd.exe',
  'SPAWN': <function spawn at ...>,
  '_concat': <function _concat at ...>,
  '_defines': <function _defines at ...>,
  '_stripixes': <function _stripixes at ...>
}

The only difference you’ll see if you run this on a specific platform is SHELL – this will be None if you run this in a posix environment, and ‘command’ if you run it in a darwin environment.

Platform

You must specify a platform when creating an environment, and it’s not easy to change after that. By “you must”, I mean that a platform will be specified, whether you do so explicitly or let SCons do it for you. By “not easy to change after that”, I mean that you can’t do it through normal SCons means. For example

env = Environment(platform='win32')
denv = env.Clone(platform='darwin')
log('win32_to_darwin.txt', cenv, "env.Clone(platform='darwin')")

still gives us a ‘win32′ platform, as we can see from our environment dump.

...
  'AS': 'ml',
  'ENV': { 'COMSPEC': 'C:\\windows\\system32\\cmd.exe', ...
  'HOST_ARCH': 'x86_64',
  'HOST_OS': 'win32',
  'PLATFORM': 'win32',
...

Since we’re not going to reach inside an Environment and fiddle with it, this means that we have to specify the platform up front, in our Environment call. This also means that anything that is platform-specific is going to be configured in that Environment call, which means if we have anything we need to declare for the platform setup, we have to declare it before the Environment call.

ENV, the SCons Environment’s copy of the host environment variables, is an interesting case. Are there things that need to be in ENV for platform setup to work? What exactly does platform setup encompass? Well, one way to see is to look at what changes in Environment calls. Here’s our test suite, saved to an SConstruct file.

import SCons

def log(name, env, prompt):
  logfile = open(name, 'w')
  print >> logfile, prompt
  print >> logfile, env.Dump()
  logfile.close

for plat in ['win32', 'darwin', 'posix', 'cygwin']:
  env = Environment(platform=plat, tools=[], BUILDERS={}, ENV={})
  log('raw_%s.txt' % plat, env,
      "env = Environment(platform=%s...)" % plat)

We’ll run this on a Windows machine with Visual Studio, a Mac OS X machine with XCode, and a Linux box with GCC 4.7.4, and we’ll look at the differences on same and different host OS machines. If something below is not shown, assume it is the same on all platforms and machines.

Note: I wrote the following before I really understood SCons. I’m going to rip out some of the text, because my comment about SCons not being a cross-platform tool is the key. It needs a HOST_PLATFORM/HOST_ARCH config for finding toolchains that it can run, and it needs a TARGET_PLATFORM/TARGET_ARCH for finding or configuring toolchains that will produce the desired output.

platform=’win32′

‘win32′ on Windows

  'HOST_ARCH': 'x86_64',
  'HOST_OS': 'win32',
  'MAXLINELENGTH': 2048,
  'PLATFORM': 'win32',
  'SHELL': u'C:\\windows\\System32\\cmd.exe',
  'TARGET_ARCH': 'x86_64',
  'TARGET_OS': 'win32',

  'LIBPREFIX': '',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.lib',
  'LIBSUFFIXES': ['$LIBSUFFIX'],
  'OBJSUFFIX': '.obj',
  'PROGSUFFIX': '.exe',
  'SHLIBPREFIX': '',
  'SHLIBSUFFIX': '.dll',
  'PSPAWN': <function piped_spawn at ...> 
  'SPAWN': <function spawn at ...>

‘win32′ on Mac OS X is almost the same as ‘win32′ on Windows with these differences (and the differences would prevent actual cross-compiling, assuming there were Win32 tools that ran on the Mac). Among other things, SPAWN and PSPAWN are probably not set correctly for Mac OS X.

  'HOST_ARCH': '',
  'SHELL': 'command',
  'TARGET_ARCH': '',

‘win32′ on Linux is almost the same, except with these differences

  'HOST_ARCH': '',
  'SHELL': None,
  'TARGET_ARCH': None,
  'TARGET_OS': None,

platform=’darwin’

‘darwin’ on Mac OS X

  'HOST_ARCH': None,
  'HOST_OS': None,
  'MAXLINELENGTH': 128072,
  'PLATFORM': 'darwin',
  'TARGET_ARCH': None,
  'TARGET_OS': None,

  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'OBJSUFFIX': '.o',
  'PROGSUFFIX': '',
  'SHELL': 'sh',
  'SHLIBPREFIX': '$LIBPREFIX',
  'SHLIBSUFFIX': '.dylib',
  '__RPATH': '$_RPATH',

  'PSPAWN': <function piped_fork_spawn at ...>,
  'SPAWN': <function fork_spawn at ...>,

‘darwin’ on Windows is identical to ‘darwin’ on Mac OS X. This also means it’s not actually used for cross-compiling, since there is likely no sh shell on Windows, but there could be a Clang cross-compiler running on Windows that targets Darwin (and that probably does exist somewhere).

‘darwin’ on Linux is identical to both, and actually this means that the darwin environment is just setting variables, it’s not actually looking for anything.

platform = ‘posix’

‘posix’ on Linux

  'HOST_ARCH': None,
  'HOST_OS': None,
  'MAXLINELENGTH': 128072,
  'PLATFORM': 'posix',
  'TARGET_ARCH': None,
  'TARGET_OS': None,

  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'OBJSUFFIX': '.o',
  'PROGSUFFIX': '',
  'SHELL': 'sh',
  'SHLIBPREFIX': '$LIBPREFIX',
  'SHLIBSUFFIX': '.so',
  '__RPATH': '$_RPATH',

‘posix’ on Mac OS X is almost identical to ‘darwin’ on Mac OS X. The only differences are platform and SHLIBSUFFIX, as well as some of the tools

  'HOST_ARCH': None,
  'HOST_OS': None,
  'MAXLINELENGTH': 128072,
  'PLATFORM': 'posix',
  'TARGET_ARCH': None,
  'TARGET_OS': None,

  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'OBJSUFFIX': '.o',
  'PROGSUFFIX': '',
  'SHELL': 'sh',
  'SHLIBPREFIX': '$LIBPREFIX',
  'SHLIBSUFFIX': '.so',
  '__RPATH': '$_RPATH','

  'PSPAWN': <function piped_env_spawn at ...>,
  'SPAWN': <function spawnvpe_spawn at ...>,

‘posix’ on Windows is identical to ‘posix’ on Mac OS X, with the exception of PSPAWN and SPAWN (pointing to piped_fork_spawn). So there’s a tiny bit of accomodation to Windows, but not enough to be a valid platform foundation. Still, maybe there are tools that would build posix on Windows.

platform=’cygwin’

We’re starting to see a pattern. It makes sense that ‘cygwin’ is only really valid when used on a Windows machine, because Cygwin is a Posix-emulation for Windows, and you don’t need Posix-emulation on a machine that is natively Posix (like with Mac OS or Linux).

‘cygwin’ on Windows has some oddities due to its nature, because while the internals of the toolchain are Posix, it emits Windows DLLS and EXEs. It also supports both Posix and Windows-style libraries, where posix libraries typically are libSOMETHING, whereas Windows libraries are just SOMETHING.

  'HOST_ARCH': 'None',
  'HOST_OS': 'None',
  'MAXLINELENGTH': 2048,
  'PLATFORM': 'cygwin',
  'SHELL': 'sh',
  'TARGET_ARCH': 'None',
  'TARGET_OS': 'None',

  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX', '$SHLIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'OBJSUFFIX': '.o',
  'PROGSUFFIX': '.exe',
  'SHLIBPREFIX': '',
  'SHLIBSUFFIX': '.dll',
  'PSPAWN': <function piped_fork_spawn at ...> 
  'SPAWN': <function fork_spawn at ...>
  '__RPATH': '$_RPATH',

platform tips

Platform in SCons is mixing up host platform support (e.g. SPAWN, RSPAWN and COMMAND) with target platform and toolchain settings (LIPBPREFIX etc).

The only cross-platform compiling that SCons is likely to support is ‘win32′ versus ‘cygwin’, or ‘darwin’ versus ‘posix’. I would think a more sane approach would be to declare a build error if an untenable platform were asked for.

It’s interesting and depressing that we are mixing tools, builders and platforms together. The LIBPREFIX and LIBSUFFIX are really paired with specific toolchains. It turns out that this works with current systems I know about, but platform in the context of SCons is actually more about “toolchain family”. This view is inherited from make, I believe.

However, this is all easily fixable. I suppose no one to date has really cared. SCons is probably not being used for embedded development, which is where this would really matter.

I think I’ll send in some pull requests so that HOST_ARCH and HOST_OS aren’t Windows-specific. The documentation mentions that they currently are, but won’t be in the future, so – let’s make the future today. Also, we really should have a HOST_PLATFORM and TARGET_PLATFORM. This is especially relevant with Mac OS versus iOS builds. There’s also the Xamarin toolchain, which lets you build for iOS on Windows (to some degree), and there’s also Android targets built on Posix hosts.

What does this mean for us, the large project build developers? We will only allow for specific combinations of host operating system and SCons platform. And more specifically, at the moment, we have to use the native platform unless we are using a Cygwin toolchain.

Other settings

'CONFIGUREDIR': '#/.sconf_temp'

This is used by SConf, the Autoconf-like configuration support in SCONS. It defaults to a directory named .sconf_temp in the root of your build (wherever your top-level SConstruct file is located). It’s a dot-name so it’s invisible by default on Posix systems. This is where Configure test files are written, and the entire directory is removed after Configure runs (I think).

'CONFIGURELOG': '#/config.log

This is also used by SConf and holds status of Configure operations. I presume it’s largely used for debugging the Configure system. The SCons manual says “The name of the Configure context log”.  It looks like it contains any output from tools run as part of Configure operations (test compiles and so on).

'MAXLINELENGTH': 2048,
'TEMPFILE': <class 'SCons.Platform.TempFileMunge'>,
'TEMPFILEPREFIX': '@',

Systems have a max length for a command that can be issued. For example, Windows has a limit of 2048 characters (or probably bytes) for a command-line. The TempFileMunge function can be used to get around this limit – command-lines longer than MAXLINELENGTH are actually executed as a temporary command file. If you want to override this with your own system, you can replace ‘TEMPFILE’.

Naming abstractions

SCons tries to have one name for things at the SConstruct level. These need to be translated into platform-specific and tool-specific names to get actual work done.

  'CPPSUFFIXES': [ '.c', ...
  'DSUFFIXES': ['.d'],
  'IDLSUFFIXES': ['.idl', '.IDL'],

CPPSUFFIXES is the list of suffixes of files that will be scanned for implicit dependencies as caused by #include directives. DSUFFIXES is the list of suffixes of files that will be scanned for imported D packages, again for implicit dependencies. IDLSUFFIXES is the list of suffixes of files that will be scanned for IDL implicit dependencies as indicated by either #include directives or import lines.

There’s probably some architectural reason why these are in the platform section. That said, these really should be tied to tools, not the platform. If I’m not building C/C++ files, I shouldn’t try to scan my source files for implicit dependencies, even if they end in some C-like suffix. And this is especially true for not-yet-mainstream languages like D. Adding spurious dependencies is in some respects “harmless” – it won’t cause a bad build. But it can cause your build to take longer than it should.

The reverse is definitely not true – if you miss some implicit dependencies, you can end up not rebuilding something and have a bad build. Implicit dependencies are always problematic, and perhaps the actual answer is what other build systems do, monitoring all file I/O while a full build is running, and capturing dependencies that way. Maybe do both?

  '_defines': <function _defines at ...>,
  '_CPPDEFFLAGS': '${_defines(CPPDEFPREFIX, CPPDEFINES,
                              CPPDEFSUFFIX, __env__)}',

  '_concat': <function _concat at ...>,
  'RDirs': <SCons.Defaults.Variable_Method_Caller object at ...>,
  '_CPPINCFLAGS': '$( ${_concat(INCPREFIX, CPPPATH, INCSUFFIX,
                                __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBDIRFLAGS': '$( ${_concat(LIBDIRPREFIX, LIBPATH, LIBDIRSUFFIX,
                                __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBFLAGS': '${_concat(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX,
                          __env__)}',

These are helper functions that glue together C preprocessor defines, C includes, library directories, and librarian flags.

_CPPDEFFLAGS is a construction variable that turns a list of preprocessor symbols to define and turns them into compiler command-line arguments. For example, GCC likes -D and Visual C++ likes /D. The _defines function is used to do the actual concatenation, handling both /Dsym and /Dsym=val forms.

_CPPINCFLAGS is a construction variable that turns a list of directories into compiler command-line arguments for include search paths. _LIBDIRFLAGS is a construction variable to turn a list of directories into compiler command-line arguments for library search paths. _LIBFLAGS is a construction variable to turn a list of libraries into compiler command-line arguments for library linking. The _concat function takes a prefix, suffix, list of elements, an environment, and optional post-processing function, target and source.

Again, why are these defined in the platform as opposed to being tied to tools? I suppose it’s because there’s not really the idea of tool suites?

  'LIBPREFIX': '',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.lib',
  'LIBSUFFIXES': ['$LIBSUFFIX'],
  'OBJPREFIX': '',
  'OBJSUFFIX': '.obj',
  'PROGPREFIX': '',
  'PROGSUFFIX': '.exe',
  'SHLIBPREFIX': '',
  'SHLIBSUFFIX': '.dll',
  'SHOBJPREFIX': '$OBJPREFIX',
  'SHOBJSUFFIX': '$OBJSUFFIX',

These are abstractions for naming of intermediate files.

LIBPREFIX, LIBPREFIXES, LIBSUFFIX and LIBSUFFIXES are prefixes and suffixes for static library files. On Windows, these are typically NAME.lib, and on Posix, these are typically libNAME.a. Your code would refer to the library as NAME. The reason for the plural for is to handle cases where multiple naming schemes can co-exist.

OBJPREFIX and OBJSUFFIX are prefixes and suffixes for object files that are linked into libraries. On windows, these are typically NAME.obj, and on Posix, these are NAME.o. I’m guessing no one has run across a case where multiple naming schemes need to be handled on a single platform.

PROGPREFIX and PROGSUFFIX are for executable names.

SHLIBPREFIX and SHLIBSUFFIX are for shared library names.

SHOBJPREFIX and SHOBJSUFFIX are for naming of objects that go into shared libraries. I’m guessing there is some platform or toolchain that wants these to be distinguished from “normal” object files.

  'Dir': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'Dirs': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'ESCAPE': <function escape at ...>,
  'File': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'PSPAWN': <function piped_spawn at 0x025E64F0>,
  'SHELL': u'C:\\windows\\System32\\cmd.exe',
  'SPAWN': <function spawn at ...>,
  '_stripixes': <function _stripixes at ...>

ESCAPE is a helper function that escapes strings used on command-lines so that special characters in strings don’t cause problems. For example, spaces in paths are problematic, and those strings need to be quoted. The default escape function is pretty draconian, it just quotes all strings on Windows, for example.

SHELL is the name of the shell program passed to the SPAWN function.

SPAWN is the command interpreter function called to execute command lines. PSPAWN is a piped variant of SPAWN used in some cases (currently only in the Configure module).

_stripixes is a helper function that removes prefixes and suffixes in certain cases. Specifically, the GNU linker wants to turn libfoo.a into -lfoo.

Dir, Dirs, File and RDirs are all functions that convert a list of strings into something relative to the target being built. This is done by looking up the call chain for the first TARGET variable and then using it as the base for the string. TARGET will be set by a builder before the build actions are started. So, for example, a env[‘Dir’] call will end up calling TARGET.Dir() for the first TARGET that is found while going up the stack. This saves a lot of copying of environment variables at the expense of a little speed and clarity in the code.

Build systems

scons –interactive

Leave a comment

scons –interactive is one way to speed up repeated builds.

SCons spends most of its time at three tasks

  1. Reading and parsing the SConstruct/SConscript/site_scons files. This takes about 6-10 seconds at the moment in one of our moderate-sized projects.
  2. Examining dependencies. This takes 1-20 seconds in that same project.
  3. Building targets. This can take 0-1000 seconds, depending on how much is being rebuilt.

The first one can be separated from the other two quite easily. If you do this

scons --interactive

then SCons will read all the build files, and drop you into a little shell

scons>>> build

will simply build everything. Or

scons>>> build game_client

will just build the game_client target.

Type exit to leave the shell

scons>>> exit

As long as you are not updating any build scripts themselves, you can leave a command-prompt in the scons shell, and you’ll shave seconds off each rebuild (for the project I’m working on, this is 5-10 seconds). This can be significant if you are working on a subset of the files and know it.

If you have command-line options that affect the build, you’d typically do them on the call that starts the interactive shell. For example, we have some typical options like

scons --interactive --fast --skiptests

I’ve just started doing this, so I don’t know yet if there are hidden gotchas.

The commands you can issue in interactive mode are

  • build
  • clean
  • exit
  • help
  • shell
  • version

You can use shell to run another scons, but I suggest against doing “shell scons –interactive” from inside an scons shell :)

For slightly more information, see the scons man page at http://www.scons.org/doc/production/HTML/scons-man.html.

The real goal is that naive builds are fast (that’s what I call just typing “scons”), but right now naive builds have a fairly large overhead on non-trivial projects.

Build systems

Using SCons effectively

Leave a comment

(This is currently a work in progress)

There are no great cross-platform build systems. Of all the general-purpose ones, SCons is merely the least bad. I don’t want to re-hash those arguments here. Just trust me.

What do you need to know in order to use SCons effectively? You need to understand a little about the architecture of SCons, and you need to set up and follow some conventions.

In general, you create an Environment (or multiple Environments), populate it with config and tools, and then operate it with Builders and Actions and Aliases. SCons is a functional/declarative system; you declare what you want. It can be somewhat disconcerting at first to work this way, although many other build systems do follow this model, starting with the venerable Make.

Environment

SCons does its best to decouple the build system from the environment of a build machine. This is generally a good thing, otherwise you end up with build environments that are hard to replicate. You can import things from your machine’s environment, but you generally have to do so either explicitly, or through using a Tool.

The SCons build process has the idea of an Environment, which is the collection of things that builders operate on. The subset of the local machine environment that is useful is in the ENV field of the environment. Its structure is somewhat Unix-centric, since SCons started in an Unix environment, but enough of this maps to Windows such that you can almost write generic code.

Platforms

By default, environments are initialized with builders and variables appropriate to the host platform. It is possible to some degree to do cross-platform builds, and create an Environment with a specific platform. This does imply that builders are available on the host platform for that target platform.

SCons knows about the following platforms (as well as  lot of other old or niche platforms that you can find in the source code).

  • ‘cygwin’
  • ‘darwin’
  • ‘posix’
  • ‘win32′

Some cross-platform building is simple – for example, building ‘cygwin’ on a Windows OS, or ‘posix’ on a Mac OS. Other cross-platform building would be extremely challenging, but getting simpler. For example, at some point it might be possible to generate Windows binaries on a Posix platform with Clang, and the vice-versa is nearly true today.

Options and Variables

There are two main ways for a user to pass information to an SCons build operation.

Command-line options are declared with SCons.Script.AddOption, and used with two dashes – e.g., –verbose or –arch=win32. This is essentially the Python optparse module, with a few additions. Command-line options always have default values, even if the default value is None. SCons itself uses AddOption for its options. A handful of options can be set with SetOption() in code, but most options can only be supplied on the command-line.

Variables, or arguments, are passed on the command line as key/value pairs. These are stored in the SCons.Script.ARGUMENTS dictionary, and are accessed as a normal Python dictionary. They are also in the ARGLIST list, in the order they were given on the command line (but without the names, just the values). The ARGLIST approach is also the only way to access multiple occurrences of a variable. You can also use a Variables object to allow these to have help and default values.

Which is the correct approach? Generally, it seems like build scripts use Option instead of Variable. However, Variables are attached to environments, and Option is global, so there is room for both.

Construction variables

While these may be initialized by both arguments and options, and tools and platforms, construction variables are how information is passed to tool invocations when Builders and Actions actually create and run tasks. There is a generalized expansion operation that instantiates construction variables as needed. This also can be the source of some confusion at first, since expansion happens at the point it is needed, and usually not at the point where you declare Builders and Actions.

Tools

The way to use and extend SCons is through tools. Almost every bit of build action is done as a tool. When you create a new environment and do not specific any tools, this is the same as specifying the ‘default’ tool, which automatically adds a set of tools based on the host platform.

Most of your build actions should be expressed as tools, and your SConstruct and SConscripts should simply be declaring all the dependencies and targets.

Your tools typically go into the site_scons/site_tools directory. However, there is a toolpath argument to Environment that can point to additional tools directories, and the toolpath argument is stored in the environment for use by further Clone or Tool calls.

Tools can even be functions, as opposed to files. Tool entries can also have arguments attached to them, which can be useful for tools that need information in order to be initialized properly. For example, the code in a tool might be unable to locate tool binaries, whereas an argument (perhaps from the user on the command-line) might contain that information.

Builders

Builders are somewhat orthogonal to tools. A tool may introduce custom builders, or it may refine builders. Builders turn sources into targets, and through inference (usually with rules about file extensions) either targets or sources can be inferred from the other.

Builders do two things. At parse time, builders are the major factor in building the dependency tree. At build time, builders are invoked when dependencies declare that a target needs to be generated or regenerated. The classic builder is exemplified by StaticObject, which builds a static object file from one or more C, C++, D, or Fortran source files.

If you have one-off operations, then you can use Command to create an ad-hoc builder; this specifies the sources, the targets, and a command-line or Python object used to turn the sources into targets.

Aliases

Closely related to Builders are Aliases. An Alias creates one or more abstract targets that expand to one or more other targets. As well, an action or list of actions can be specified when any of the alias targets are not up-to-date.

The major use of Alias is to create meta targets. For example, a set of related programs that are all used together could be connected by an Alias, and then the user builds the Alias target, instead of each individual target. Quite often, there is no one single top-level target, and electing one of the lower-level targets arbitrarily as “the target” is confusing and prone to long-term error.

Actions

Actions are not often directly used. However, some Actions find great use. PreActions are run before a specified target is build, and PostActions are run after a specified target is built. Both of these let you customize Build actions without needing to create a new Builder, or modify an existing one.

Performance

SCons is slow. The major sources of slowness are the reading and comprehending of the SCons build file (the SConstruct, SConscripts, and related build files), and then the detection of dependencies by looking at every individual file in the build. Even a moderate build with a few tens of thousands of files can take 5-10 seconds to get through the parsing of the scripts, and another 30 seconds in dependency checking.

For the former, scons –interactive might come in handy. This reads and parses all the scripts, then drops the user into a very simple shell. The most common action is “build”, and the user can build specific targets, or the default targets. if multiple builds are done back-to-back without changing the build scripts themselves, this could save some time.

For the latter, the only way to speed up SCons is to delegate the file scanning to some other system, or to rewrite its system. SCons has a cache, but it still hits the filesystem to validate that the cache is good, and this makes the cache less useful. It might be possible to integrate a system like ibb to be an oracle for the portion of the filesystem referenced by SCons. Systems like ibb have code that watches for filesystem updates; they basically cache a subset of the filesystem in RAM, and updates to that cache are proportional to the number of changes to that filesystem.

The actual answer is to rewrite a large part of SCons to allow for a filesystem-watching approach. Systems like ibb and tup are bottom-up, driven by the changes from the filesystem. Systems like make and scons are top-down, driven by the declaration of file dependencies. One answer that would preserve the general power of the top-down approach would be to have filesystem changes result in repair actions to the dependency tree.

And the true answer would be to integrate a build system, a filesystem watcher, and a content-addressable revision control system like git. Files that are committed no longer need watching, and don’t need dependency rediscovery.

To-do

Experiment with custom env.Decider functions to see where the time is being spent. If an scons –interactive build with a custom Decider that just returns false takes no time, then an ibb approach would be of huge benefit.

Use EnsureSConsVersion and EnsurePythonVersion to make sure that users have appropriate versions of both.

Python’s atexit.register() function can be used to cause functions to run at exit.

Investigate Repository() to see if it would be practical to mix scons and git together.

Appendix

Scons documentation

Threads about command-line arguments

Threads about precompiled headers

Build systems

Scons reading

Leave a comment

http://blog.nelhage.com/2010/11/why-scons-is-cool/

http://mu2e.fnal.gov/public/hep/computing/scons.shtml

http://cournape.wordpress.com/?s=scons

http://www.m5sim.org/SCons_build_system

https://code.google.com/p/gyp/

Blender has two build toolchains – scons and cmake. But Google switched away from scons in pepper and are using makefiles (which is sad because they had a pretty disciplined way of using scons).

https://groups.google.com/forum/#!msg/native-client-discuss/O3kRuaYm5bE/wnpyYOP5MHkJ

http://0install.net/package-scons.html

http://aligorith.blogspot.com/2010/08/build-systems-in-rush-quick-recipe-for.html

http://two.pairlist.net/pipermail/scons-dev/2013-February/000597.html

V8 for Chromium uses gyp instead of scons now.

Libjingle (also Google) uses both gyp and scons.

Native Client might be supporting both? But comment above about pepper contradicts. https://code.google.com/p/nativeclient/issues/detail?id=2731

http://cloud9.epfl.ch/testing-programs/producing-llvm-binaries

Several people have extended Scons in one way or another.

Ninja is yet another build system, and someone has integrated it with Scons. Or, rather, generate a Ninja script from an Scons dry run.