I had the bright idea to add SCons extras to my builds via a build helper that adds them to any build, instead of manually copying them to site_scons/site_tools folders in each project. To begin with, I was adding latest/modified Visual Studio code found in the MSCommon directory in the SCons build – rather than force everyone to have a bleeding-edge SCons install to build with Visual Studio 2013, I figured I would just wedge these in.
This sounded like a good idea…
First try – DefaultToolPath
There’s a global array called DefaultToolPath in SCons.Tool that SCons’ own startup code puts your site_scons_site_tools folder into, so I figured I would just use it:
def AddSConsTools():
# Add the appropriate SCons tools to the default toolpath. This
# contains an override MSCommon folder as well as other builders
# TBD - have a folder per scons version, in case we have incompatible
# changes happening
site_tools = os.path.join(os.path.dirname(__file__), 'scons-2.3.3', 'site_tools')
print('Adding SCons tools from %s' % site_tools)
SCons.Tool.DefaultToolpath.insert(0, os.path.abspath(site_tools))
By inserting in the front of the array, my tools come first, at least before anything that is done by SCons itself (DefaultToolPath is filled in when SCons starts up).
We have a packaging system that makes it easy for SCons projects to load dependencies. I have a package that my SConstruct loads, and that package looks like this:
build/
scons-2.3.3/
site_tools/
MSCommon/
__init__.py
arch.py
common.py
netframework.py
sdk.py
vc.py
vs.py
Clever, right? Except it didn’t work.
After some tracing through the code to find out how tools load (SCons/Tool/__init__.py, in the __init__ function for the Tool class), I found out that, while you can add extra tools this way, you can’t add sub-modules of tools this way, because the way that the Tool intercept loader works is only searching for modules, not handling sub-modules. So I could replace or add Tools themselves, but I couldn’t do this for sub-modules of Tool, e.g. the MSCommon folder.
Second try – overload tools that reference MSCommon.
Since I really wanted a replacement MSCommon folder, I decided to put some of the Visual Studio tools in my build helper as well. That makes things a little more fragile, but this way, as long as my Tool (say msvc) is loaded first, even in a dummy environment, it pulls in my MSCommon files, and then those loaded modules are used by other parts of the system. Since msvc.py is the first loaded tool (by default), I just put in msvc.py, but I could put everything in.
So, to be clear, here’s what my build helpers looks like now:
build/
scons-2.3.3/
site_tools/
msvc.py
MSCommon/
__init__.py
arch.py
common.py
netframework.py
sdk.py
vc.py
vs.py
As another quick reference, here are the tools that import modules from MSCommon:
- linkloc (Phar Lap ETS, niche)
- midl (Microsoft IDL compiler)
- mslib (Microsoft Library archiver)
- mslink (Microsoft Linker)
- mssdk (Microsoft SDK setup)
- msvc (Microsoft Visual C++)
- msvs (Microsoft Visual Studio, generates solution/project files)
Since MSCommon is a module with an __init__.py, it directly pulls in three of the submodules (SCons.Tool.MSCommon.sdk, SCons.Tool.MSCommon.vc, and SCons.Tool.MSCommon.vs). Each of those pulls in SCons.Tool.MSCommon.common. I can’t find anything that imports SCons.Tool.MSCommon.arch or SCons.Tool.MSCommon.netframework, so these appears to be dead modules.
So as long as I reference a tool in my own hierarchy first, I can get my custom MSCommon modules to be used.
But, that didn’t work either. I had added debug printing in the previous step, and, depressingly, this is what I saw:
Adding SCons tools from c:\package_cache\Battle.net Build Tools\0.75\noarch\build\scons-2.3.3\site_tools
Tool(name=default)
loading SCons.Tool.default
found 2nd at C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\default.py ('.py', 'U', 1)
Tool(name=msvc)
found 1st at c:\package_cache\Battle.net Build Tools\0.75\noarch\build\scons-2.3.3\site_tools\msvc.py ('.py', 'U', 1)
I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\__init__.pyc
I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\sdk.pyc
I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\common.pyc
I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\vc.pyc
I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\vs.pyc
I am c:\package_cache\Battle.net Build Tools\0.75\noarch\build\scons-2.3.3\site_tools\MSCommon\__init__.py
This is because __init__.py is doing absolute imports . So, while my override msvc.py was indeed loaded, my MSCommon folder was still skipped in favor of the one in the SCons 2.3.0 install folder.
Third try – put folders in the python path
DefaultToolPath isn’t a good idea for anything other than a Tool. For the MSCommon folder, I’ll just insert it into the python search path. So I rearrange my package to look like this:
build/
scons-2.3.3/
site_tools/
msvc.py
scons_patch/
SCons/
Tool/
MSCommon/
__init__.py
common.py
sdk.py
vc.py
vs.py
I didn’t actually need msvc.py in the package, I just put it there to verify that I had everything working. In my real build package, I have some custom tools that we use for builds. I also trimmed the apparently-obsolete files.
My startup code now looks like this:
def AddSConsTools():
# TBD - have a folder per scons version, in case we have incompatible
# changes happening
sconsdir = os.path.join(os.path.dirname(__file__), 'scons-2.3.3')
# Add custom SCons tools to the default toolpath
site_tools = os.path.join(sconsdir, 'site_tools')
print('Adding SCons tools from %s' % site_tools)
SCons.Tool.DefaultToolpath.insert(0, os.path.abspath(site_tools))
# Add the SCons.Tool.MSCommon folder to the python module path
MSCommon = os.path.join(sconsdir, scons_patch)
sys.path.insert(0, os.path.abspath(MSCommon))
print('Patching SCons with %s' % MSCommon)
Of course, that didn’t work either. I get an error saying it can’t find MSCommon.
EnvironmentError: No module named MSCommon:
Even worse, it is finding things inside MSCommon – in the wrong place!
I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\__init__.pyc I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\sdk.pyc I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\common.pyc I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\vc.pyc I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\vs.pyc
Fourth try – more care
It would behoove me to pay more attention to how modules inside MSCommon are being imported.
Some files are using absolute module paths
from SCons.Tool.MSCommon import msvs_exists, merge_default_version
but others are using relative paths
from MSCommon import msvc_exists
or (inside the MSCommon folder)
import common
So I can either add an MSCommon path to the python path, or add all the Microsoft-specific tools to my package?
So let’s try that:
def AddSConsTools():
# TBD - have a folder per scons version, in case we have incompatible
# changes happening
sconsdir = os.path.join(os.path.dirname(__file__), 'scons-2.3.3')
# Add custom SCons tools to the default toolpath
site_tools = os.path.join(sconsdir, 'site_tools')
print('Adding SCons tools from %s' % site_tools)
SCons.Tool.DefaultToolpath.insert(0, os.path.abspath(site_tools))
# Add the SCons.Tool.MSCommon folder to the python module path
scons_patch = os.path.join(sconsdir, 'scons_patch')
print('Patching SCons with %s' % scons_patch)
sys.path.insert(0, os.path.abspath(os.path.join(scons_patch, 'SCons', 'Tool')))
sys.path.insert(0, os.path.abspath(scons_patch))
And it didn’t work!
Tool(name=msvc) I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\__init__.pyc I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\sdk.pyc I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\common.pyc I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\vc.pyc I am C:\Python27\Scripts\..\Lib\site-packages\scons-2.3.0\SCons\Tool\MSCommon\vs.pyc I am c:\package_cache\Battle.net Build Tools\0.83\noarch\build\scons-2.3.3\scons_patch\SCons\Tool\MSCommon\__init__.py
Without being 100% sure, I believe this means that Python first searches for a .pyc, and then if it doesn’t find one, it looks for a .py. Since I’ve used SCons in the past, my SCons build has .pyc files in it. Since my newly loaded package does not, it skips right past my package when importing the stock Python way. That last use where it finds my package files is maybe due to the Tool loader code? It’s odd, because supposedly an import checks for cached modules before searching on disk. But I really don’t care that much at the moment.
So, two options
- Fiddle with the module path and directly import all the MSCommon stuff, then restore the module path
- Use sys.meta.path
The first one is probably the easiest to get working, but it’s hacky, whereas option #2 sounds cool.
To be continued…
A rant
It’s a lot of work for short-term gain, but it’s also something I’m thinking about in terms of SCons or other build systems – build tools are themselves dependencies, and having globally installed tools with a single name that could be any version is a bad idea. A lot of the Unix tools, specifically GCC, have this issue, and while there are vendor workarounds, there is no standard for doing this. Apple is a little better with XCode, because you can select the version of xcode you want used, and everything is relative to the xcode root. Visual Studio went the “every version has a unique name”, which is good and bad; they suffer from incompatibility between toolchain releases, or rather they purposefully break compatibility.
Oops, philosophy. I’ll talk more about this in a bit, but build tools should be versioned objects just like source code, and a project declares what build tools it depends on. There has to be a root, but that root would be very lightweight and have no version dependencies.
I call this “build”, but it’s really a build meta-tool that makes sure the right build tools are loaded and used.
