SCons Environment in depth, part 1

Let’s tear down Environment(), so we know how to use it properly. As much as possible, I’m looking at things as a user, so I’ve been trying things without looking at the source code of SCons. I want to see how things actually work, as opposed to what the SCons developers intended.

For other parts in the series, see:

Minimal environment

Looking at a somewhat minimal Environment in SCons (we have to supply a platform, we’ll get to that in a moment) like so:

env = Environment(tools=[], platform='win32', BUILDERS={}, ENV={})
env.Dump()

and re-arranging for prettiness, we have

env = Environment(tools=[], platform='win32', builders={}), ENV={}
{
  'BUILDERS': {},
  'ENV': { },
  'SCANNERS': [],

  'CONFIGUREDIR': '#/.sconf_temp',
  'CONFIGURELOG': '#/config.log',
  'MAXLINELENGTH': 2048,
  'TEMPFILE': <class 'SCons.Platform.TempFileMunge'>,
  'TEMPFILEPREFIX': '@',

  'HOST_ARCH': 'x86_64',
  'HOST_OS': 'win32',
  'PLATFORM': 'win32',
  'TARGET_ARCH': 'x86_64',
  'TARGET_OS': 'win32',

  'LIBPREFIX': '',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.lib',
  'LIBSUFFIXES': ['$LIBSUFFIX'],
  'OBJPREFIX': '',
  'OBJSUFFIX': '.obj',
  'PROGPREFIX': '',
  'PROGSUFFIX': '.exe',
  'SHLIBPREFIX': '',
  'SHLIBSUFFIX': '.dll',
  'SHOBJPREFIX': '$OBJPREFIX',
  'SHOBJSUFFIX': '$OBJSUFFIX',

  'CPPSUFFIXES': [ '.c',
                   '.C',
                   '.cxx',
                   '.cpp',
                   '.c++',
                   '.cc',
                   '.h',
                   '.H',
                   '.hxx',
                   '.hpp',
                   '.hh',
                   '.F',
                   '.fpp',
                   '.FPP',
                   '.m',
                   '.mm',
                   '.S',
                   '.spp',
                   '.SPP',
                   '.sx'],
  'DSUFFIXES': ['.d'],
  'IDLSUFFIXES': ['.idl', '.IDL'],

  '_CPPDEFFLAGS': '${_defines(CPPDEFPREFIX, CPPDEFINES,
                              CPPDEFSUFFIX, __env__)}',
  '_CPPINCFLAGS': '$( ${_concat(INCPREFIX, CPPPATH, INCSUFFIX,
                                __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBDIRFLAGS': '$( ${_concat(LIBDIRPREFIX, LIBPATH, LIBDIRSUFFIX,
                                __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBFLAGS': '${_concat(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX,
                          __env__)}',

  'Dir': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'Dirs': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'ESCAPE': <function escape at ...>,
  'File': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'PSPAWN': <function piped_spawn at 0x025E64F0>,
  'RDirs': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'SHELL': u'C:\\windows\\System32\\cmd.exe',
  'SPAWN': <function spawn at ...>,
  '_concat': <function _concat at ...>,
  '_defines': <function _defines at ...>,
  '_stripixes': <function _stripixes at ...>
}

The only difference you’ll see if you run this on a specific platform is SHELL – this will be None if you run this in a posix environment, and ‘command’ if you run it in a darwin environment.

Platform

You must specify a platform when creating an environment, and it’s not easy to change after that. By “you must”, I mean that a platform will be specified, whether you do so explicitly or let SCons do it for you. By “not easy to change after that”, I mean that you can’t do it through normal SCons means. For example

env = Environment(platform='win32')
denv = env.Clone(platform='darwin')
log('win32_to_darwin.txt', cenv, "env.Clone(platform='darwin')")

still gives us a ‘win32′ platform, as we can see from our environment dump.

...
  'AS': 'ml',
  'ENV': { 'COMSPEC': 'C:\\windows\\system32\\cmd.exe', ...
  'HOST_ARCH': 'x86_64',
  'HOST_OS': 'win32',
  'PLATFORM': 'win32',
...

Since we’re not going to reach inside an Environment and fiddle with it, this means that we have to specify the platform up front, in our Environment call. This also means that anything that is platform-specific is going to be configured in that Environment call, which means if we have anything we need to declare for the platform setup, we have to declare it before the Environment call.

ENV, the SCons Environment’s copy of the host environment variables, is an interesting case. Are there things that need to be in ENV for platform setup to work? What exactly does platform setup encompass? Well, one way to see is to look at what changes in Environment calls. Here’s our test suite, saved to an SConstruct file.

import SCons

def log(name, env, prompt):
  logfile = open(name, 'w')
  print >> logfile, prompt
  print >> logfile, env.Dump()
  logfile.close

for plat in ['win32', 'darwin', 'posix', 'cygwin']:
  env = Environment(platform=plat, tools=[], BUILDERS={}, ENV={})
  log('raw_%s.txt' % plat, env,
      "env = Environment(platform=%s...)" % plat)

We’ll run this on a Windows machine with Visual Studio, a Mac OS X machine with XCode, and a Linux box with GCC 4.7.4, and we’ll look at the differences on same and different host OS machines. If something below is not shown, assume it is the same on all platforms and machines.

Note: I wrote the following before I really understood SCons. I’m going to rip out some of the text, because my comment about SCons not being a cross-platform tool is the key. It needs a HOST_PLATFORM/HOST_ARCH config for finding toolchains that it can run, and it needs a TARGET_PLATFORM/TARGET_ARCH for finding or configuring toolchains that will produce the desired output.

platform=’win32′

‘win32′ on Windows

  'HOST_ARCH': 'x86_64',
  'HOST_OS': 'win32',
  'MAXLINELENGTH': 2048,
  'PLATFORM': 'win32',
  'SHELL': u'C:\\windows\\System32\\cmd.exe',
  'TARGET_ARCH': 'x86_64',
  'TARGET_OS': 'win32',

  'LIBPREFIX': '',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.lib',
  'LIBSUFFIXES': ['$LIBSUFFIX'],
  'OBJSUFFIX': '.obj',
  'PROGSUFFIX': '.exe',
  'SHLIBPREFIX': '',
  'SHLIBSUFFIX': '.dll',
  'PSPAWN': <function piped_spawn at ...> 
  'SPAWN': <function spawn at ...>

‘win32′ on Mac OS X is almost the same as ‘win32′ on Windows with these differences (and the differences would prevent actual cross-compiling, assuming there were Win32 tools that ran on the Mac). Among other things, SPAWN and PSPAWN are probably not set correctly for Mac OS X.

  'HOST_ARCH': '',
  'SHELL': 'command',
  'TARGET_ARCH': '',

‘win32′ on Linux is almost the same, except with these differences

  'HOST_ARCH': '',
  'SHELL': None,
  'TARGET_ARCH': None,
  'TARGET_OS': None,

platform=’darwin’

‘darwin’ on Mac OS X

  'HOST_ARCH': None,
  'HOST_OS': None,
  'MAXLINELENGTH': 128072,
  'PLATFORM': 'darwin',
  'TARGET_ARCH': None,
  'TARGET_OS': None,

  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'OBJSUFFIX': '.o',
  'PROGSUFFIX': '',
  'SHELL': 'sh',
  'SHLIBPREFIX': '$LIBPREFIX',
  'SHLIBSUFFIX': '.dylib',
  '__RPATH': '$_RPATH',

  'PSPAWN': <function piped_fork_spawn at ...>,
  'SPAWN': <function fork_spawn at ...>,

‘darwin’ on Windows is identical to ‘darwin’ on Mac OS X. This also means it’s not actually used for cross-compiling, since there is likely no sh shell on Windows, but there could be a Clang cross-compiler running on Windows that targets Darwin (and that probably does exist somewhere).

‘darwin’ on Linux is identical to both, and actually this means that the darwin environment is just setting variables, it’s not actually looking for anything.

platform = ‘posix’

‘posix’ on Linux

  'HOST_ARCH': None,
  'HOST_OS': None,
  'MAXLINELENGTH': 128072,
  'PLATFORM': 'posix',
  'TARGET_ARCH': None,
  'TARGET_OS': None,

  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'OBJSUFFIX': '.o',
  'PROGSUFFIX': '',
  'SHELL': 'sh',
  'SHLIBPREFIX': '$LIBPREFIX',
  'SHLIBSUFFIX': '.so',
  '__RPATH': '$_RPATH',

‘posix’ on Mac OS X is almost identical to ‘darwin’ on Mac OS X. The only differences are platform and SHLIBSUFFIX, as well as some of the tools

  'HOST_ARCH': None,
  'HOST_OS': None,
  'MAXLINELENGTH': 128072,
  'PLATFORM': 'posix',
  'TARGET_ARCH': None,
  'TARGET_OS': None,

  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'OBJSUFFIX': '.o',
  'PROGSUFFIX': '',
  'SHELL': 'sh',
  'SHLIBPREFIX': '$LIBPREFIX',
  'SHLIBSUFFIX': '.so',
  '__RPATH': '$_RPATH','

  'PSPAWN': <function piped_env_spawn at ...>,
  'SPAWN': <function spawnvpe_spawn at ...>,

‘posix’ on Windows is identical to ‘posix’ on Mac OS X, with the exception of PSPAWN and SPAWN (pointing to piped_fork_spawn). So there’s a tiny bit of accomodation to Windows, but not enough to be a valid platform foundation. Still, maybe there are tools that would build posix on Windows.

platform=’cygwin’

We’re starting to see a pattern. It makes sense that ‘cygwin’ is only really valid when used on a Windows machine, because Cygwin is a Posix-emulation for Windows, and you don’t need Posix-emulation on a machine that is natively Posix (like with Mac OS or Linux).

‘cygwin’ on Windows has some oddities due to its nature, because while the internals of the toolchain are Posix, it emits Windows DLLS and EXEs. It also supports both Posix and Windows-style libraries, where posix libraries typically are libSOMETHING, whereas Windows libraries are just SOMETHING.

  'HOST_ARCH': 'None',
  'HOST_OS': 'None',
  'MAXLINELENGTH': 2048,
  'PLATFORM': 'cygwin',
  'SHELL': 'sh',
  'TARGET_ARCH': 'None',
  'TARGET_OS': 'None',

  'LIBPREFIX': 'lib',
  'LIBPREFIXES': ['$LIBPREFIX', '$SHLIBPREFIX'],
  'LIBSUFFIX': '.a',
  'LIBSUFFIXES': ['$LIBSUFFIX', '$SHLIBSUFFIX'],
  'OBJSUFFIX': '.o',
  'PROGSUFFIX': '.exe',
  'SHLIBPREFIX': '',
  'SHLIBSUFFIX': '.dll',
  'PSPAWN': <function piped_fork_spawn at ...> 
  'SPAWN': <function fork_spawn at ...>
  '__RPATH': '$_RPATH',

platform tips

Platform in SCons is mixing up host platform support (e.g. SPAWN, RSPAWN and COMMAND) with target platform and toolchain settings (LIPBPREFIX etc).

The only cross-platform compiling that SCons is likely to support is ‘win32′ versus ‘cygwin’, or ‘darwin’ versus ‘posix’. I would think a more sane approach would be to declare a build error if an untenable platform were asked for.

It’s interesting and depressing that we are mixing tools, builders and platforms together. The LIBPREFIX and LIBSUFFIX are really paired with specific toolchains. It turns out that this works with current systems I know about, but platform in the context of SCons is actually more about “toolchain family”. This view is inherited from make, I believe.

However, this is all easily fixable. I suppose no one to date has really cared. SCons is probably not being used for embedded development, which is where this would really matter.

I think I’ll send in some pull requests so that HOST_ARCH and HOST_OS aren’t Windows-specific. The documentation mentions that they currently are, but won’t be in the future, so – let’s make the future today. Also, we really should have a HOST_PLATFORM and TARGET_PLATFORM. This is especially relevant with Mac OS versus iOS builds. There’s also the Xamarin toolchain, which lets you build for iOS on Windows (to some degree), and there’s also Android targets built on Posix hosts.

What does this mean for us, the large project build developers? We will only allow for specific combinations of host operating system and SCons platform. And more specifically, at the moment, we have to use the native platform unless we are using a Cygwin toolchain.

Other settings

'CONFIGUREDIR': '#/.sconf_temp'

This is used by SConf, the Autoconf-like configuration support in SCONS. It defaults to a directory named .sconf_temp in the root of your build (wherever your top-level SConstruct file is located). It’s a dot-name so it’s invisible by default on Posix systems. This is where Configure test files are written, and the entire directory is removed after Configure runs (I think).

'CONFIGURELOG': '#/config.log

This is also used by SConf and holds status of Configure operations. I presume it’s largely used for debugging the Configure system. The SCons manual says “The name of the Configure context log”.  It looks like it contains any output from tools run as part of Configure operations (test compiles and so on).

'MAXLINELENGTH': 2048,
'TEMPFILE': <class 'SCons.Platform.TempFileMunge'>,
'TEMPFILEPREFIX': '@',

Systems have a max length for a command that can be issued. For example, Windows has a limit of 2048 characters (or probably bytes) for a command-line. The TempFileMunge function can be used to get around this limit – command-lines longer than MAXLINELENGTH are actually executed as a temporary command file. If you want to override this with your own system, you can replace ‘TEMPFILE’.

Naming abstractions

SCons tries to have one name for things at the SConstruct level. These need to be translated into platform-specific and tool-specific names to get actual work done.

  'CPPSUFFIXES': [ '.c', ...
  'DSUFFIXES': ['.d'],
  'IDLSUFFIXES': ['.idl', '.IDL'],

CPPSUFFIXES is the list of suffixes of files that will be scanned for implicit dependencies as caused by #include directives. DSUFFIXES is the list of suffixes of files that will be scanned for imported D packages, again for implicit dependencies. IDLSUFFIXES is the list of suffixes of files that will be scanned for IDL implicit dependencies as indicated by either #include directives or import lines.

There’s probably some architectural reason why these are in the platform section. That said, these really should be tied to tools, not the platform. If I’m not building C/C++ files, I shouldn’t try to scan my source files for implicit dependencies, even if they end in some C-like suffix. And this is especially true for not-yet-mainstream languages like D. Adding spurious dependencies is in some respects “harmless” – it won’t cause a bad build. But it can cause your build to take longer than it should.

The reverse is definitely not true – if you miss some implicit dependencies, you can end up not rebuilding something and have a bad build. Implicit dependencies are always problematic, and perhaps the actual answer is what other build systems do, monitoring all file I/O while a full build is running, and capturing dependencies that way. Maybe do both?

  '_defines': <function _defines at ...>,
  '_CPPDEFFLAGS': '${_defines(CPPDEFPREFIX, CPPDEFINES,
                              CPPDEFSUFFIX, __env__)}',

  '_concat': <function _concat at ...>,
  'RDirs': <SCons.Defaults.Variable_Method_Caller object at ...>,
  '_CPPINCFLAGS': '$( ${_concat(INCPREFIX, CPPPATH, INCSUFFIX,
                                __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBDIRFLAGS': '$( ${_concat(LIBDIRPREFIX, LIBPATH, LIBDIRSUFFIX,
                                __env__, RDirs, TARGET, SOURCE)} $)',
  '_LIBFLAGS': '${_concat(LIBLINKPREFIX, LIBS, LIBLINKSUFFIX,
                          __env__)}',

These are helper functions that glue together C preprocessor defines, C includes, library directories, and librarian flags.

_CPPDEFFLAGS is a construction variable that turns a list of preprocessor symbols to define and turns them into compiler command-line arguments. For example, GCC likes -D and Visual C++ likes /D. The _defines function is used to do the actual concatenation, handling both /Dsym and /Dsym=val forms.

_CPPINCFLAGS is a construction variable that turns a list of directories into compiler command-line arguments for include search paths. _LIBDIRFLAGS is a construction variable to turn a list of directories into compiler command-line arguments for library search paths. _LIBFLAGS is a construction variable to turn a list of libraries into compiler command-line arguments for library linking. The _concat function takes a prefix, suffix, list of elements, an environment, and optional post-processing function, target and source.

Again, why are these defined in the platform as opposed to being tied to tools? I suppose it’s because there’s not really the idea of tool suites?

  'LIBPREFIX': '',
  'LIBPREFIXES': ['$LIBPREFIX'],
  'LIBSUFFIX': '.lib',
  'LIBSUFFIXES': ['$LIBSUFFIX'],
  'OBJPREFIX': '',
  'OBJSUFFIX': '.obj',
  'PROGPREFIX': '',
  'PROGSUFFIX': '.exe',
  'SHLIBPREFIX': '',
  'SHLIBSUFFIX': '.dll',
  'SHOBJPREFIX': '$OBJPREFIX',
  'SHOBJSUFFIX': '$OBJSUFFIX',

These are abstractions for naming of intermediate files.

LIBPREFIX, LIBPREFIXES, LIBSUFFIX and LIBSUFFIXES are prefixes and suffixes for static library files. On Windows, these are typically NAME.lib, and on Posix, these are typically libNAME.a. Your code would refer to the library as NAME. The reason for the plural for is to handle cases where multiple naming schemes can co-exist.

OBJPREFIX and OBJSUFFIX are prefixes and suffixes for object files that are linked into libraries. On windows, these are typically NAME.obj, and on Posix, these are NAME.o. I’m guessing no one has run across a case where multiple naming schemes need to be handled on a single platform.

PROGPREFIX and PROGSUFFIX are for executable names.

SHLIBPREFIX and SHLIBSUFFIX are for shared library names.

SHOBJPREFIX and SHOBJSUFFIX are for naming of objects that go into shared libraries. I’m guessing there is some platform or toolchain that wants these to be distinguished from “normal” object files.

  'Dir': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'Dirs': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'ESCAPE': <function escape at ...>,
  'File': <SCons.Defaults.Variable_Method_Caller object at ...>,
  'PSPAWN': <function piped_spawn at 0x025E64F0>,
  'SHELL': u'C:\\windows\\System32\\cmd.exe',
  'SPAWN': <function spawn at ...>,
  '_stripixes': <function _stripixes at ...>

ESCAPE is a helper function that escapes strings used on command-lines so that special characters in strings don’t cause problems. For example, spaces in paths are problematic, and those strings need to be quoted. The default escape function is pretty draconian, it just quotes all strings on Windows, for example.

SHELL is the name of the shell program passed to the SPAWN function.

SPAWN is the command interpreter function called to execute command lines. PSPAWN is a piped variant of SPAWN used in some cases (currently only in the Configure module).

_stripixes is a helper function that removes prefixes and suffixes in certain cases. Specifically, the GNU linker wants to turn libfoo.a into -lfoo.

Dir, Dirs, File and RDirs are all functions that convert a list of strings into something relative to the target being built. This is done by looking up the call chain for the first TARGET variable and then using it as the base for the string. TARGET will be set by a builder before the build actions are started. So, for example, a env[‘Dir’] call will end up calling TARGET.Dir() for the first TARGET that is found while going up the stack. This saves a lot of copying of environment variables at the expense of a little speed and clarity in the code.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>