Mailpile

Mailpile is a new project to do self-hosting email. The front-end is a webmail client, but the back-end is a Python server you host somewhere.

http://www.mailpile.is/

LWN has an article on it: Mailpile targets webmail and attemps to raise funds.

Slide deck: http://www.mailpile.is/files/OHM2013%20-%20Rescuing%20e-mail%20from%20the%20cloud.pdf

Most importantly, the source is on Github, because it’s open-source: https://github.com/pagekite/Mailpile

 

Notes on designing a system

Seek the simplest possible system that works well now as well as into the future.

  • works well now – most important
  • simple – almost trumps “works” but #2 by a hair
  • extends to the future – distant third

This is not to be confused with the uber-system that solves all problems. In fact, it’s an indictment of that mentality. Uber-systems don’t work well now, they take a long time to get to that state. They are by definition not simple. And, as it turns out, they don’t extend to the future, because the uber-system turns out to be incomplete when the future changes.

This is also not to be confused with “minimum viable product” or “prototyping” or being reactive. You do want to have a clear design. But you want to spend all your effort on the parts that will be used right now. The worst you want is a Pareto distribution, where 20% of your functionality is used 80% of the time. You do not want a system where half of it is used 1% of the time or less. The ideal is that all of the system is used all of the time. We will never get there, but that is the goal.

Notes on building software

As a side rant, more and more I dislike that software papers are often locked up behind pay walls tied to large academic publishers. What good does it do me that this knowledge is not easily available?

Program Changes and the Cost of Selective Recompilation, Ellen Ariel Borison, Ph.D dissertation, 1989. The thesis is that most recompilation is wasted effort due to poor dependency management by make tools, and indirectly due to consequences of modularization. It’s 25 years later, and our builds are somewhat faster only because our machines are thousands of times faster. Instead, she proposed a variant of a dependency graph called a manufacturing graph, and the update process is guided by difference predicates as a generalization of selective recompilation mechanisms.

What I also want to read is “A model of software manufacture”, a paper she published in 1986, but it was published in a Springer-Verlag book and so I can’t read it unless I find a copy of that book (probably in excess of $100). I like what I can infer from what others have said (including her own thesis), but I’d like to read the original.

One of the projects submitted for the Software Carpentry Builder competion was Quilt, which explicitly mentioned Borison’s work as the model being followed. The paper referenced goes into some details about the guiding principles behind Quilt.

Recent talk by the LLVM folks in making a linker project for Clang indicate they want to go towards a very fine-grained model like what Borison calls for.

Essence versus accidents

In a paper titled No silver bullet: essence and accidents of software engineering, written in 1987 by Frederick Brooks, as a follow-up to his seminal book The Mythical Man-Month, he introduced the idea of essence versus accidents, as a guide to where to spend effort to speed up software engineering. Or rather, he re-introduced concepts from Aristotle.

Essence is inherent in the nature of the software, and accidents are what currently attend development but are not inherent.

Now, of course, this is almost arguing the consequent, unless there is a way to separate essence from accident ahead of time. There is a related concept from Toyota’s TPS that talks about waste, and the idea of “apparently necessary waste”. All waste is bad, but some waste is harder to get rid of than others, because it appears to be necessary. If it’s actually necessary, of course, then it would not be waste. Toyota labels this “apparently necessary” to allow us to move ahead (we are pragmatic) but not to forget that we should keep attacking this from time to time until we get rid of it (or prove that it is essential).

So, let’s talk about the effort that goes into maintaining build systems. If there was ever a prime candidate to call “accidental” in software engineering, it would be the development and maintenance of build systems, but we don’t know how much of a build system is essential, and how much are accidents we could get rid of. Another is the difficulty of doing postmortem analysis on problems.

There are things that are probably essence, because they have resisted solution despite large amounts of expended effort. For over 50 years, one persistent idea was “complete specification” as something that would speed us up; we still have no such thing, and in fact in recent years have backed away from demanding that. Another was “automatic programming”, the idea that non-engineers could cause programs to be generated to solve their problems; this does not yet exist, and perhaps won’t until the day we have truly intelligent programs to write these programs for us.

SCons documentation links

Unlike for other build systems, there is no single source of truth for SCons design and architecture, and thus for use.

SCons documentation

Documentation page at scons.org points to the pieces of the SCons documentation.

man page, Steven Knight and Anthony Roach, last updated March 2013. While technically a man page, this is also large and comprehensive, and is probably the single best existing piece of documentation on SCons.

User Guide, 2.3.0. This has useful and complementary information, but is not as complete as the man page.

API, 2.3.0. This is quickly becoming dangerous to use. This is generated automatically from comments in the source, and the comments in the source are often old. This is more useful as a quick-start before working on the SCons source than as a source of information about SCons. And even then, it’s probably more useful to just dive into the source code.

SCons source on bitbucket, and SCons Development practices on scons.org. Sometimes you just have to go to the source.

FAQ, from the SCons Wiki.

Papers

sccons Design Overview, Steven Knight, June 2000 (rescued via the Internet Archive). This was the second round draft of the Software Carpentry competition to create a modern build tool.

SCons Design version 0.91, Steven Knight, 2001. This was the last draft of the original SCons design, but is quickly getting out of date.

SCons Design and Implementation, Steven Knight, 2002 (paper presented at the Python 10 conference in 2002).

Empirical Comparison of SCons and GNU Make, Ludwig Hähne, 2008. This is a re-examination of theory and practice for make and SCons.

Related

Cons – A Software Construction System, FSF, 1996-2000. Cons is the predecessor to SCons, written by Bob Siddenbotham in 1991 in Perl, and first released in 1996.

Popular, Fast, or Usable: Pick One, 2010, an article somewhat unhappy with SCons performance.

Make Alternatives, Adrian Neagu, July 2005. A great overview to what was the state of the art in 2005. SCons comes out favorably by his criteria.

The Quest for the Perfect Build System, Noel Llopsis, 2005. Another great overview from 2005 (what was it about that year?). Here’s a working link to his generate_libs program: https://github.com/greatwolf/generate_libs.

Four Interesting Build Tools, Neil Mitchell, 2012. This is less comprehensive but far more up to date, and covers Redo, Ninja, Tup and Fabricate.

Wikipedia articles on SCons, Waf (an SCons derivative used by KDE for a few years until they switched to CMake), CMake (somewhat different direction than SCons).

waf is a rewrite of SCons by Thomas Nagy, obtained initial purchase from its use by KDE, who has since abandoned it in favor of CMake.

Tup, written by Mike Shal in C, is interesting and its techniques should be merged with SCons. Build System Rules and Algorithms is a 2009 paper describing the techniques used to make Tup. If we graded build systems by the combination of how funny and how analytical their authors are, Mike Shal would win.

ibb, written by Chad Austin in Python, is something similar, and its author even wrote Scalable Build Systems: An Analysis of Tup to compare it to ibb. See Your Version Control and Build Systems Don’t Scale for the main post about ibb.

redo is a new competitor to make, inspired by D.J. Bernstein. Purely top-down software rebuilding is a software thesis written by Alan Grosskurth in 2007 on redo.

Tundra is also something to look at – source is on Github at deplinenoise/tundra.

Shake, written by Neil Mitchell in Haskell, had several papers and videos released in 2012 describing it.

GBS, git-build-system, is a Tizen package build tool that builds from Git repositories.

gyp was written by Google to be Chromium’s build system. It takes an approach similar to CMake in that it generates project files for popular build environments instead of being the build environment as well as the build tool – e.g. it can generate Visual Studio or Xcode projects, as well as SCons or Make build files. The authors are Mark Mentovai and Steven Knight, although Steven Knight is not in the Members list for the project.

According to a Reddit thread on Tup, Steven Knight was hired to Google, the Chromium team used SCons for a bit, then Google switched to gyp (written by Steven Knight?) and subsequently there has been almost no progress on SCons itself.