Bye Make, hello SCons (Fortran)

Makefiles have been around for a long time and are used in many places. And not without reason: a straightforward dependency tree with one version of the target and a specific compiler and all the source in one directory is easy to setup and is handled well by (GNU) Make. It's a common situation, and if that describes your project, there's no need to switch.

But the code I'm working with has a build script and Makefile that can make a version based on the potential energy function used, on parallel/serial, static/dynamic linking, in debug or optimized mode and for many platforms and compilers (most of which are relics of the last century; this is Fortran after all). All wonderfully crafted by someone smart using five levels of makefile-recursion, with skills I admire but never hope to learn. It works, but it's about as readable as the Voynich manuscript.

I'm not touching that Makefile, but after this glimpse of the possible future, I decided to look for an alternative. Some languages have a dedicated method; Make is still used a lot for general purpose; long store short I tried SCons, chosen based merely on reading the introductions of several tools.

SCons is a general purpose construction tool. It's open source and has been around a while. It's made in Python, although it's Python 2.4 (doesn't work in 3) and uses a lot of CamelCase. The Makefile, which is now called SConstruct, is also written in Python, so it's pretty powerful. Like Make though (and unlike Python usually), it's essentially declarative: you tell SCons what to do, and SCons figures out how to do it. Which is quite the oversimplification. But you do for example provide sources -> target recipes which SCons executes in the order it likes (possibly parallel), just like Make.

An immediately visible difference is that, whereas Make uses recipes and constructs a lot of commands, in SCons you instead construct a few environments and then only tell what to make from what; you do not provide the build command. Like this (with gfortran flags):

env = Environment(
        FORTRANFLAGS=['-std=f2008', '-ffree-form', '-fbacktrace', '-Og', '-g', '-Wall',],
        LIBS=['blas', 'lapack',]
)
env.Program(join(dir, 'main.run'), source=glob('src/*.F08'))

This might compile the whole thing if it's not too complicated. No need to create each of the .o files manually and link them all; SCons does all that with just the above command. It has a dependency scanner for Fortran files, so if you use explicit use statements, the files will compile in the correct order, without having to specify the interdependencies. (To be fair, if you use implicit interfaces, the order doesn't matter so it'd be simple with Make too. But we're in the 21th century where explicit is better than implicit - and this way the compiler can check that all arguments to procedures in other files are correct).

You can install SCons with

pip install --egg scons

Advantages

  • SCons figures a lot of things out for you, so you'll typically need less code than with make.
  • It's easy to make portable builds, e.g. leave the output name above and it'll be firstsource.exe on Windows and firstsource on Linux.
  • It's easily extensible using custom classes like builders and scanners.
  • You can easily make multiple versions of a target and place all (intermediate) files in a separate directory.
  • It's just Python, so no weird rules for variable expansions, problems for files with whitespace or bugs due to single $-signs.
  • Environments are much more isolated, increasing portability and easing the making of mixed builds.

Disadvantages

  • Much less popular than Make, so harder to find information. There's actually quite some documentation, but much fewer questions have already been answered on StackOverflow, and good examples are sorely lacking (for Fortran especially).
  • Due to the above, the greater expressiveness and some convenient magic happening, I feel it's harder to learn than Make. You might earn the time back though.
  • It's less likely to already be installed on systems. It's easy to install as Python package, and Python2.4+ is pre-installed on many systems except Windows (which doesn't have Make either), but it's still a downside. EDIT: "SCons be distributed with the software product, so users do not need to install it" (but they still need Python2).
  • The SConstruct file is included/evaled or something by SCons so you don't need to import the SCons commands you use. In fact, I couldn't easily get commands imported explicitly even though I wanted to. I think this implicitness is bad practise (see earlier), and it confuses IDE_s.

Overall, I like it. I liked Make too at first, but lately it's really been annoying me. SCons seems like a more high-level alternative. It just needs some more examples, so I'll post mine. It also gives a nice overview of gfortran flags I think are useful (another thing that needs more examples). I use it here:

#!python

from glob import glob
from os.path import join

general=['-fautomatic', '-funderscoring',
    '-fno-protect-parens', '-fno-signed-zeros', '-fno-trapping-math',
    '-fimplicit-none', '-fmodule-private', '-pedantic', # 'Werror
    '-Wuse-without-only', '-Wimplicit-interface', '-Wimplicit-procedure']
FLAGS = dict(
    minimal=['-g', '-fbacktrace'],
    debug=general + ['-Og', '-g', '-Wall', '-Wextra', '-fbacktrace', '-fbounds-check',
        '-fmax-errors=1', '-ffpe-trap=invalid,zero,overflow,underflow,denormal', '-fcheck=all'],
    optimize=general + ['-Ofast', '-march=native', '-ffrontend-optimize'],
)

envs, targets = {}, {}
for mode in ('minimal', 'debug', 'optimize',):
    envs[mode] = Environment(
        FORTRANMODDIRPREFIX='-J',
        FORTRANMODDIR='${TARGET.dir}',
        FORTRANFLAGS=FLAGS[mode] + ['-fopenmp',],
        F08PATH='${TARGET.dir}/../lib',
        F08FLAGS=['-std=f2008', '-ffree-form', '-ffree-line-length-none', '$FORTRANFLAGS'],
        F90FLAGS=['-std=legacy', '-ffixed-form',] + FLAGS['minimal'],
        CPPFLAGS='-cpp',
        CPPDEFINES=[mode[:3].upper()],
        LIBS=['blas', 'lapack',],
        LINKFLAGS=['-fopenmp'],
    )
    dir = 'build/{0:s}'.format(mode)
    envs[mode].VariantDir(variant_dir=dir, src_dir='.', duplicate=True)
    sources = list(join(dir, file) for file in glob('src/*.F08'))
    sources += list(join(dir, file) for file in ['lib/blas.f90', 'lib/lapack.f90',])
    envs[mode].Depends(target=sources, dependency=['SConstruct'])
    targets[mode] = envs[mode].Program(join(dir, 'main.run'), source=sources)
    envs[mode].Alias('{0:.3s}'.format(mode), targets[mode])

default_mode = 'debug'
test = Command(target='build/test.log', source=targets[default_mode], action="$SOURCE --initonly 2>&1 | tee $TARGET" )
envs[default_mode].Depends(test, targets[default_mode])
AlwaysBuild(test)
envs[default_mode].Alias('test', test)

envs[default_mode].Default(targets[default_mode])

This allows different compile modes, accessible by typing scons deb, scons opt, scons test opt etc. The default is to build in debug mode only. Each of these versions builds in build/modename. Dependencies are all explicitly provided in the Fortran code, so they are resolved by SCons and interfaces are checked by gfortran. There's a test command, which executes the code with a flag that only initializes and then quits (you can run any code, though for compiling you'll find that you rarely need to construct commands).

One useful addition for debugging. You can show the whole environment. This also includes the build commands, which can help you find out which variables you should set.

if 'env' in COMMAND_LINE_TARGETS:
    print(env.Dump())
    exit()

Simply add this code and type scons env every time you want the environment. If you want to know which command is being used, you can add --debug=presub to a normal compile, which will print the command before substitution. See --debug in --help for more debug options.

Next time Make drives you crazy, give SCons a try!

Comments

You need to be logged in to comment.