Rapid setup and deployment

Linux and Mac

https://www.docker.io/

http://docs.docker.io/en/latest/installation/vagrant/

https://flynn.io/

http://blog.scoutapp.com/articles/2013/08/28/docker-git-for-deployment

Windows

http://boxstarter.org/

http://chocolatey.org/

http://technet.microsoft.com/en-us/sysinternals/ee656415.aspx

http://www.hanselman.com/blog/LessVirtualMoreMachineWindows7AndTheMagicOfBootToVHD.aspx

http://www.hanselman.com/blog/StepByStepTurningAWindows7DVDOrISOIntoABootableVHDVirtualMachine.aspx

http://blogs.technet.com/b/haroldwong/archive/2012/08/18/how-to-create-windows-8-vhd-for-boot-to-vhd-using-simple-easy-to-follow-steps.aspx

http://social.technet.microsoft.com/wiki/contents/articles/516.how-to-boot-from-a-vhd.aspx

http://technet.microsoft.com/en-us/library/hh825691.aspx

http://technet.microsoft.com/en-us/library/hh825709.aspx

 

Windows developer tools

There’s some good stuff in here

http://www.hanselman.com/blog/ScottHanselmans2014UltimateDeveloperAndPowerUsersToolListForWindows.aspx

I have yet to use http://chocolatey.org/ or http://boxstarter.org/, but both sound like something I want to be using. In particular, when testing developer setup instructions, it would be nice to be able to start from a clean box each time, so that nothing gets left out of the instructions. There’s always that one magic bit of config you forget to write down.

And of course, the SysInternals tools are always a must-have on any Windows developer machine – http://technet.microsoft.com/en-us/sysinternals/bb545021.aspx. I don’t know why Microsoft hates permanent and readable URLs, though.

There’s also http://www.foldersizes.com.

Also see http://www.voidtools.com/ for Search Everything – it does an initial file/folder scan, then monitors disk changes so it stays up to date.

Something I want to try to use is Disk2vhd http://technet.microsoft.com/en-us/sysinternals/ee656415, because it can create VHD images (Microsoft’s Virtual Machine format) from mounted volumes. The main challenge is that the format only supports volumes up to 127 GB in size, which is pretty small by today’s standards.

 

A list of make systems

last updated: 2013-12-22 9:50 AM PST.

This is every make system I know of, and I’ll keep updating this. My terminology is that a make system can do dependencies and generate code (object files, binaries, arbitrary tool output); development environments typically sit on top of make systems but include editing and debugging; and build systems offer scheduling, multi-machine control, cross-platform in a single build, reporting/logging, and packaging/distribution.

This needs some sort of organization; I’ll probably do both a family tree, and alphabetical.

make

Make is written in C.

Perhaps the original make utility, written in 1976 and available with all Unix systems. There are two variants that deserve to be called make; POSIX make, the standardized version from IEEE Std 1003.1, and GNU make, which is the defacto standard. Wikipedia has links to useful information: http://en.wikipedia.org/wiki/Make_(software). It’s more likely that you have GNU make than POSIX make. The vast majority of open-source projects to date come with make files.

make is typically used only in conjunction with building Unix programs, and Mac programs that stick to POSIX features. make is also often used in conjunction with a configure program to generate a platform-specific or configuration-specific makefile, and autoconf is the most widely used such system.

There are a number of make-compatible variants.

  • Microsoft’s nmake is a subset, and sees little modern use.
  • Android uses GNU make but passes the android.mk and application.mk files to multiple phases of the build process.

Autotools

Autotools is written in C.

While listed separately (because it sits on top of make), it is used in conjunction with make. A lot of Unix software in the 1970s through 1990s was build with Autotools. However, it’s really not recommended for anyone any more, it is very arcane, very slow, and very tied to a specific set of Unix toolchains. For example, KDE switched from Autotools to CMake.

Don’t use Autotools/Autoconf. Just don’t.

msbuild

http://msdn.microsoft.com/en-us/library/0k6kkbsd.aspx. msbuild is probably written in C++.

msbuild is the build system for Visual Studio starting with Visual Studio 2005. The msbuild engine can directly consume .vcproj/.vcxproj files. Most Windows-only software is  built with msbuild.

XCode

https://developer.apple.com/technologies/tools/. XCode is (presumably) written in Objective-C++.

https://developer.apple.com/library/mac/documentation/DeveloperTools/Reference/XcodeBuildSettingRef/0-Introduction/introduction.html. XCode the IDE has an embeded make system that is available through the xcodebuild command-line tool, or directly from the IDE. Tragically, there is very little direct documentation on it, as compared to msbuild. See http://www.objc.io/issue-6/build-process.html for a simple overview.

SCons

http://www.scons.org/. SCons is written in Python.

SCons is a fairly cross-platform make system, in that a single SConstruct can be used to build on multiple platforms. It hasn’t seen much development in years, however. It would be nice if it doesn’t decay to uselessness, because it’s the only serious cross-platform make system available. It supports Windows and Mac, and a large varienty of Unix systems. SCons has configuration built in, it does not use a separate configuration system.

Some reading:

cmake

http://www.cmake.org/. CMake is written in C++.

CMake isn’t  a make system on its own. Instead, it generates projects to be used in other make systems; its main clients are msbuild, XCode and make. It supports Windows and Mac, and some Unix systems.

ninja

http://martine.github.io/ninja/. Ninja is similar to make in philosophy. It was written by a Chrome engineer, and is supposedly replacing other make systems used by Chrome. Ninja is also not usually used as a standalone make system, but in conjunction with other make systems (e.g. CMake can generate Ninja makefiles). In fact, Ninja files are usually generated by something else, since they are not particularly convenient to write by hand. Also, from the author, Ninja is based on what he thought Tup did, but without the parts he disliked.

Stuff to read:

gyp

https://code.google.com/p/gyp/. Supposedly, gyp was written in reaction to the Chromium team’s problems with SCons.

Stuff to read:

tup

http://gittup.org/tup/. Tup is a new project notable for its sense of grandiose humor, and its focus on performance. The http://gittup.org/tup/tup_vs_mordor.html page is well worth reading for anyone, not just for more Tup information. Tup is a dependency system, so it would be interesting to see what a complex code+data build looks like in Tup.

Some reading:

premake

http://industriousone.com/premake. Premake is written in C and Lua. It is like CMake in that it generates makefiles for other build systems, it’s not a make on its own.

Some reading:

waf

https://code.google.com/p/waf/. waf is written in Python.

This was originally a fork of SCons, but it can be considered to be a completely new project by this point. Also see http://docs.waf.googlecode.com/git/book_17/single.html.

fabricate

https://code.google.com/p/fabricate/. Fabricate is written in Python.

Fabricate is interesting in that it has no explicit dependency declarations; it discovers them by watching filesystem operations. It’s essentially a build script that memoizes itself. That said, it only works on a few platforms, and is somewhat fragile because filesystem watching is not perfectly reliable on any system (it’s typically a bolt-on, or a system that is allowed to drop events when overloaded).

Shake

http://hackage.haskell.org/package/shake. Shake is a Haskell library for writing build systems.

Redo

https://github.com/apenwarr/redo. Currently written in Python, although the author stated he wants to re-implement it in C. It also really only works on Unix systems.

This is a cleanroom re-implementation of Daniel J. Bernstein’s (unreleased?) redo project; this is a make successor, intended to fix make’s inherent flaws.

Crumb

https://github.com/da-x/crumb. It looks like this only runs on Unix.

This is not really a make system, but a tool that instruments filesystem calls to determine dependencies, and is intended for use in build systems, providing equivalent functionality to Fabricate.

badgerconfig

https://code.google.com/p/badgerconfig/. This is written in Python.

This looks like just a Visual Studio project generator.

Rake

http://rake.rubyforge.org/. Rake is written in Ruby.

This is “Ruby Make”, although the makefiles use Ruby syntax.

QMake

Maven

Ant

OMake

http://omake.metaprl.org/index.html. OMake is written in OCaml.

While OMake uses the syntax of make, it isn’t really a make dropin because it extends make substantially.

Some reading:

FBuild

https://github.com/felix-lang/fbuild. FBuild is written in Python 3.

This seems like yet another build system inspired by the decline of SCons.

Some reading:

mem

http://srp.github.io/mem/. Mem is written in Python.

Mem works via instrumentation and memoization.

Choco

Facile

Jam

http://www.perforce.com/resources/documentation/jam. Jam is written in C.

Jam’s makefiles are derived from the syntax of Make, although it is not make compatible. It has been around for a long time, and is fairly well documented. But it’s make, and suffers from the same limitations and has the same problems.

FTJam (FreeType Jam).

http://freetype.sourceforge.net/jam/. FTJam is written in C.

FTJam is a slight enhancement of the original Perforce Jam.

BJam (Boost Jam)

http://www.boost.org/doc/libs/1_31_0/tools/build/jam_src/index.html. Boost Jam is written in C++.

It was derived from FTJam, which in turn was derived from Perforce’s original Jam.

Boost.Build

http://www.boost.org/boost-build2/. This is written in C++.

I think this replaced the use of Jam by the Boost team. I don’t know if anyone else uses Boost.Build. It also appears to be dead – there was a plan in 2007 to reimplement most of it in Python, but it looks like that was never started.

Rant

Makepp

Wonderbuild

http://retropaganda.info/~bohan/work/psycle/branches/bohan/wonderbuild/wonderbuild/README, or http://sourceforge.net/p/psycle/code/HEAD/tree/trunk/build-systems/wonderbuild/wonderbuild/. Wonderbuild is written in Python.

Wonderbuild appears to have been written for the Psycle Modular Music Creation Studio project. A perhaps biased benchmark shows it as faster than most other make systems –  http://retropaganda.info/~bohan/work/psycle/branches/bohan/wonderbuild/benchmarks/time.xml. It is not documented except by reading through the source code.

Cook

http://miller.emu.id.au/pmiller/software/cook/.

MPC

http://ociweb.com/products/mpc.

genmake

http://www.robertnz.net/genmake.htm

bakefile

http://www.bakefile.org/index.html

Other reference

http://www.dmoz.org/Computers/Software/Build_Management/Make_Tools/

http://www.a-a-p.org/tools_build.html

http://freecode.com/articles/make-alternatives

http://neilmitchell.blogspot.com/2012/02/four-interesting-build-tools.html

https://code.google.com/p/go/issues/detail?id=1451

http://stackoverflow.com/questions/54674/any-good-building-tools-for-a-c-project-which-can-replace-make

http://ra1ndog.wordpress.com/2011/11/16/build-systems-or-make-replacements/

http://gamesfromwithin.com/the-quest-for-the-perfect-build-system

http://gamesfromwithin.com/bad-news-for-scons-fans

 

Sharing code

The issues with sharing code fall into three rough camps, at least as I see it:

  • distribution
  • build dependencies
  • runtime dependencies

It’s not quite as cut-and-dried as that, but these three areas are independent enough that we can reason about them separately. Also, we are going to talk about problems without considering solutions. We sometimes fall in love with a solution and want to use it, even if it may not be the best solution.

The problems

Distribution

I can’t use someone else’s code if I can’t get it – that’s the limiting case. But that limiting case focuses the conversation. For every iota of work I have to do to get the shared code into my project, that’s a cost I have to bear. This cost is usually very small for a small piece of code, but it goes up linearly with the number of times I want to get code. This is especially problematic when it comes to getting updates to the shared code, or needing to maintain multiple versions of it, because I have multiple versions of my own project.

Here are the sub-problems with the distribution piece.

Discovery. I have to discover the shared code in the first place. If I don’t know it exists, then everything else is moot. For discovery, I am talking not just about a name to a piece of shared code, but what it can do. I vaguely know about Drupal, but I might not consider it for my next project, because I don’t know that it actually already has the three most important things I am looking for. Or I might not know about performance, or pitfalls. This is usually a one-time cost, but it can also be an ongoing cost for large shared code – when should I get Qt 5 versus staying with Qt 4?

Acquisition. I have to get the shared code so I can use it. This could mean using a specific physical method, or network protocol. There’s also generally some kind of transcoding needed – maybe it comes in a DEB or RPM or MSI. Maybe it’s in a specific kind of source control system. Even if it’s the same kind of source control as the one I use, it can still be an effort to get it in a form that I can work with.

Merging. As discussed below, shared code often needs some adaptation. Once you’ve modified a project for your own use, there is now the problem that future versions of this code that you get will also need those edits, and this can easily get to be a lot of work. It’s the rare shared code that actually needs zero work on your part to use. It can be as simple as a handful of lines of change to your build system, so that you can link the code, or it can be your workarounds to their unfixed bugs, or a large amount of work to adapts its interfaces to your interfaces. It’s also problematic when you are evolving your code, and the shared code is evolving, and you need to maintain multiple versions of your adjustments to the shared code.

Build dependencies

Even the most homogeneous world has variety to it, and we are far from that – all but the simplest projects are a polyglot mix of technologies. Once I have the shared code, I have to build it along with my code. Note that I am not talking about functionality here, simply building.

Language. One fundamental issue is that shared code distributed in source form comes as one or more specific languages. Not all programs are written in C. For many languages, different versions of that language spec are nearly as much work as different languages themselves. C++, C# and Python all get new features over time. We can often bridge multiple languages together with some effort, and the effort depends on the languages and the design of the shared code.

Architecture. We may have a project that uses ARM processors, but the shared code was built for Intel processors. Or we may want to target specific features that are only found in the most current versions of the processor family, for performance reasons, or we use 3D hardware which has its own architecture. Also, it’s very common for large projects to support multiple architectures. And since no architecture lives forever, we usually don’t want our project’s lifetime to be tied to a specific architecture.

Platform. I distinguish platform from architecture; they are much the same in problem (we have irreconcilable differences), but our solutions to the problems are not the same. To handle multiple architectures, you generally either have entirely different object code, or you emulate, whereas with platform you can sometimes shim or accomodate without too much loss of performance or feature set. It is still a big problem, because platforms offer features that you build upon, and those features are generally not easy for you to reproduce. In fact, platform is shared code, but it’s a giant body of shared code that you get all at once; your choice is between whole platforms, and not picking of pieces of platforms. This is generally the biggest issue for shared code.

Toolchains. Whether we are given binaries, or object code, or source code, we have to bind that somehow with our code. We typically have a toolchain that we like, or sometimes have to use, and so we might have to do work to bend the shared code so that it can be built with this toolchain.

Runtime dependencies

Once I have the shared code, and once I can build it into my project, there’s still the matter of making it all function together. I have to call its functions somehow, passing data to it and getting data from it. I also have to wire up the shared code to its dependencies, because it’s the rare piece of shared code that actually sits at the bottommost layer. There are vanishingly few examples of shared code that have no runtime dependencies – at best, it’s math things like hash functions or random numbers or the bottom layer of a compression algorithm.

Some solutions

Coming soon.

msysgit

I love Git. But msysgit, while not horrible, could be a lot better.

For example, there’s an extension called git-cheetah. When you install msysgit, that’s one of the questions for you to answer, and it’s asked in a nicely slanted way:

  • Simple context menu (Registry based)
  • Advanced context menu (git-cheetah plugin)

I’ve found people who say “avoid this like the plague”, and others who use it. It actually took a little while to find out that this is shell integration a la TortoiseSVN. Shell integration is wonderful, except when it breaks, and it’s just complex enough that most people can’t troubleshoot it to fix the problem.

And it’s really something that you could ignore. Do you use context menus while developing? If not, then this choice is irrelevant. If you do use context menus, then (1) they need to always work, and (2) you should be able to use either or both.

Another issue with msysgit is the dreadful out-of-the-box performance with ssh connections. This is particularly bad because ssh is a very common way to connect to remote git repositories; it’s quite nearly the default for pushing to remote repositories. So you’d think that, of all the things to make sure worked well, it would be SSH. But if you want good SSH, you need to drag PuTTY in and use pagent and plink.

There’s little every written about msysgit, I don’t know much about how it’s developed or what choices are made. On the other hand, the main Git development is about as visible as you can be.

Another reason I feel awkward about msysgit is that there’s a lot of whinging on the part of the main developer; at least as of several years ago, he clashed with the Git mainline developers, and was pretty rude and insulting to boot. I mean, irrespective of who’s right and wrong, you don’t cut off your oxygen. If you’re dependent on someone, you want them to look favorably on you.

I really wish that the mainline Git developers would take a more cross-platform approach and mindset, but they are Linux developers first and foremost. It’s not surprising, given that Git’s parent was Linus Torvalds; I definitely wouldn’t expect him to care about Windows or Mac. However, Git is becoming the best revision control system on the planet, and a lot of people use Windows and Mac machines, and they aren’t going to switch just because of Git.

We need a first-class Windows client for Git, but it has to be part of the mainline development, it can’t be some parallel development process. Those always start out well, and then die after a year or so.