SCons Fortran example

I wrote about SCons several weeks ago. It took some getting used to, but I'm very happy with the result! It's found it to be more expressive and flexible than Make. Or at least non-standard things are much more straightforward; I guess Make can do almost anything with enough recursion and voodoo. I can't really comment on the speed; it's plenty fast, but there are less than 50 files.

I took some time putting it together, partially because there aren't that many Fortran examples. So I figured I'll post mine (copied below & attached)! Some properties:

  • The environments used are mostly isolated, for reproducibility across machines. The exception is that several PATHS are taken from the parent environment: it's not good for portability to hard-code those. PATH, LIBRARY_PATH and LD_LIBRARY_PATH are used from the environment. Notably FPATH can be used to help find Fortran dependencies (.so or .h files for example). Finally, INTEL_LICENSE_FILE is inherited.
  • Dependencies are resolved automatically. All the code is organized into modules, which SCons automatically compiles in the correct order (also in parallel mode). This comes with SCons, the only thing I added is pre-processing for header files.
  • Different builds don't interfere. For example, if you build an optimized version, which crashes or you change some code, and then you build a debug version the next time, Make might give you a mixed version but with SCons you'll get a complete debug version. This SConstruct goes a little further and uses VariantDir to build different versions in different directories, so you can even have different versions side-by-side (it also keeps all the derived files in one place). This applies not just for debug/optimized but also for compilers, math libraries, MPI(y/n) and for which potential (external function) to use.

When reading the file, note that SConstruct files are declarative: it specifies what you want (which targets etc); it's not a recipe of sequential steps. This is like Make but unlike most Python code.

The file adds several command-line options:

  • --nompi - turn off MPI mode, which is on by default
  • --opt - turn off debug mode, which is on by default
  • --debug_subset=<name1.F,name2.F> - compile the named files in debug and the rest in optimized mode (comma-separated); balance between performance and helpful info
  • --FC=<compiler> - specifies which compiler to use (see --help for options)
  • --mathlib=<libname> - specifies which mathematics library to use (see --help for options); should fit the compiler
  • --potential=<potential> - the potential to compile for (see --help for options)
  • --warn-less - hide some subset of warnings that the code often throws in innocent (although outdated) situations
  • --name=<name> - name of the symlink to the executable
  • --cpp-define=<MYMODE,THING=val> - define these extra pre-processing constants (comma-separated); easily toggleable functionality without changing SConstruct

Also note that SCons offers several useful default options: --tree=prune (show dependencies); --clean (remove build/ etc); -j1 (serial)

#!python

from copy import copy
from glob import glob
from multiprocessing import cpu_count
from sys import stderr
from os import environ, symlink, remove
from os.path import join, abspath, exists, islink, basename


"""
Use parallel mode by default.
"""
SetOption('num_jobs', min(cpu_count(), 16))

"""
Define flags for different purposes, which vary by compiler.

Update this if you want to add a new compiler, and it should work.
"""
UNIVERSAL_FLAGS = dict(
        mpi=dict(
                LIBS=['mpi_f90', 'mpi_f77', 'mpi'],
        ),
        nompi=dict(),
        blas=dict(
                CPPDEFINES={'BLAS': None},
                LIBS=['blas', 'lapack',],
        ),
        mkl=dict(
                CPPDEFINES={'BLAS': None},
                LINKFLAGS=['-mkl',],
        ),
        nag=dict(
                CPPDEFINES={'NAG': None},
                LIBS=['nag_nag',],
        ),
        hco=dict(
                CPPDEFINES={'HCO': None},
                POTENTIAL_FILES=[join('pot', 'hcopot.F')],
        ),
        h3=dict(
                CPPDEFINES={'H3': None},
                POTENTIAL_FILES=[join('pot', 'bkmp2.F')],
        ),
)
COMPILER_FLAGS = dict(
        ifort=dict(
                always=dict(
                        FORTRANMODDIRPREFIX='-module ',
                        FORTRANFLAGS=[['-free', '-extend-source', '132']],#, '-no-wrap-margin'], # -no-wrap-margin for newer iforts
                ),
                opt=dict(
                        CPPDEFINES={'NODEBUG': None},
                        FORTRANFLAGS=['-fast'],
                        # FORTRANFLAGS=['-ipo', '-O3', '-no-prec-div', '-static', '-xHost'],  # same as -fast on linux
                ),
                deb=dict(
                        CPPDEFINES={'DEBUG': None},
                        FORTRANFLAGS=['-O0', '-g', '-traceback', '-fpe0', '-fp-stack-check', ['-check', 'all'], ['-warn', 'all'], ['-debug', 'extended'], ['-diag-error-limit', '1']],
                ),
                warn_less=dict(
                        FORTRANFLAGS=[['-warn', 'nounused'], ['-warn', 'nodeclarations']],
                ),
        ),
        gfortran=dict(
                always=dict(
                        FORTRANMODDIRPREFIX='-J',
                        FORTRANFLAGS=['-ffree-form', '-ffree-line-length-132', '-fautomatic', '-funderscoring', '-fno-protect-parens', '-fno-signed-zeros', '-fno-trapping-math', '-fimplicit-none', '-fmodule-private', '-pedantic', '-Wimplicit-interface', '-Wimplicit-procedure'],  # -Wuse-without-only  # '-std=legacy',
                ),
                opt=dict(
                        CPPDEFINES={'NODEBUG': None},
                        FORTRANFLAGS=['-Ofast', '-march=native', '-ffrontend-optimize'],
                ),
                deb=dict(
                        CPPDEFINES={'DEBUG': None},
                        FORTRANFLAGS=['-Og', '-g', '-Wall', '-Wextra', '-fbacktrace', '-fbounds-check', '-fmax-errors=1', '-ffpe-trap=invalid,zero,overflow,underflow,denormal', '-fcheck=all'],
                ),
                warn_less=dict(
                        FORTRANFLAGS=['-Wno-unused-variable', '-Wno-unused-dummy-argument', '-Wno-unused-parameter', '-Wno-character-truncation'],
                ),
        ),
)
FLAGS = {}
for compiler, flags in COMPILER_FLAGS.items():
        FLAGS[compiler] = copy(UNIVERSAL_FLAGS)
        for name, vals in flags.items():
                if name in FLAGS[compiler]:
                        stderr.write('"{0:s}" defined in universal flags and compiler flags ({1:s})\n'.format(name, compiler))
                FLAGS[compiler][name] = vals

"""
Collect all the source files.
"""
sources = glob('*.F') + [join('interface', 'blas.f'), join('interface', 'lapack.f')]

"""
Define command-line options, which are parsed later.
"""
AddOption('--nompi', dest='mpi', action='store_false', default=True, help='Don\'t compile with multiprocessing enabled.')

AddOption('--opt', dest='opt', action='store_true', default=False, help='Compile an optimized version, without debugging sacrifices.')

FC_options = ['ifort', 'gfortran',]
AddOption('--FC', '--compiler', dest='FC', nargs=1, choices=FC_options, default=None, help='Choose the compiler type (implementation should be in PATH): {0:s}.'.format(', '.join(FC_options)))

ML_options = ['blas', 'lapack', 'mkl', 'nag',]
AddOption('--mathlib', dest='mathlib', nargs=1, choices=ML_options, default='blas', help='Choose the math library: {0:s}.'.format(', '.join(ML_options)))

potentials = ['hco', 'HCO', 'h3', 'H3',]
AddOption('--potential', '--V', dest='potential', nargs=1, choices=potentials, default=None, help='Choose the potential to use: {0:s}.'.format(', '.join(potentials)))

AddOption('--warn-less', dest='warn_less', action='store_true', default=False, help='Hide certain warnings, like unused variables.')

AddOption('--name', dest='name', action='store', default='sccalc', help='Change the name of the executable.')

AddOption('--debug_subset', dest='debug_subset', nargs=1, default=None, help='Specify a comma-separated list of files that are compiled in debug mode (only useful in combination with --opt).')

AddOption('--cpp-define', dest='cpp_defines', nargs=1, default='', help='Which variables to define for the C pre-processor.')

"""
Create the default environment, which is extended later based on arguments.
"""
env = Environment(
        FORTRANMODDIR='${TARGET.dir}',
        FORTRANPATH=['${TARGET.dir}'] + list(pth for pth in environ.get('FPATH', '').split(':') if pth),
        ENV=dict(
                PATH=environ.get('PATH', ''),
                LIBRARY_PATH=environ.get('LIBRARY_PATH', ''),
                LD_LIBRARY_PATH=environ.get('LD_LIBRARY_PATH', ''),
                INTEL_LICENSE_FILE=environ.get('INTEL_LICENSE_FILE', ''),
        ),
        CPPDEFINES={'LINUX': None, 'FLUID': None, 'BLAS': None,},
)

"""
Update the environment based on flags provided.
"""
if GetOption('FC') is None:
        if env.WhereIs('ifort'):
                compiler = 'ifort'
        elif env.WhereIs('gfortran'):
                compiler = 'gfortran'
        else:
                raise AssertionError('No compiler found (and --FC not set)')
else:
        compiler = GetOption('FC')

"""
Print the compiler path, for reproducibility assuming there is logging.
"""
print('using {0:s} compiler at "{1:s}"'.format(compiler, env.WhereIs(compiler)))
env['FORTRAN'] = compiler

env.Append(**FLAGS[compiler]['always'])

"""
Add several (compiler-dependent) flags based on command-line choices.
"""
parallelism = 'mpi' if GetOption('mpi') else 'nompi'
if GetOption('mpi'):
        env.Append(
                CPPDEFINES={'MPI': None,},
                **FLAGS[compiler]['mpi']
        )
else:
        env.Append(**FLAGS[compiler]['nompi'])

mathlib = GetOption('mathlib')
if mathlib == 'lapack': mathlib = 'blas'
env.Append(
        **FLAGS[compiler][mathlib]
)

if GetOption('potential') is None:
        stderr.write('potential not given; using HCO\n')
        potential = 'hco'
else:
        potential = GetOption('potential').lower()
env.Append(
        **FLAGS[compiler][potential]
)

if GetOption('warn_less'):
        env.Append(
                **FLAGS[compiler]['warn_less']
        )

"""
For mixed mode, find the files to be used in debug mode.
"""
optimization = optimization_name = 'opt' if GetOption('opt') else 'deb'
debug_subset = GetOption('debug_subset')
if debug_subset and optimization == 'deb':
        stderr.write('--debug_subset was given so --opt has been turned on\n')
        optimization = 'opt'
        debug_subset = None
if debug_subset:
        debug_subset = debug_subset.split(',')
        optimization_name = 'optdeb'

"""
Add extra pre-processing definitions - fairly literal from scons command to the build one.
"""
cpp_defines= {}
for definition in GetOption('cpp_defines').split(','):
        if '=' in definition:
                name, val = definition.split('=', maxsplit=1)
        else:
                name, val = definition, None
        cpp_defines[name] = val
env.Append(
        CPPDEFINES = cpp_defines
)

"""
Specify an output directory, where all files will be stored. This prevents interference between for example debug and optimized builds.
"""
dir = join('build', '_'.join([compiler, mathlib, parallelism, optimization_name, potential]))
env.VariantDir(variant_dir=dir, src_dir='.', duplicate=True)

"""
This is for FORTRANPPCOM which is used to pre-process and compile .F files into .o (always needed).
"""
env.Append(FORTRANPATH=[join(dir, 'pot'), join(dir, 'interface'), ])

"""
This is for LINKCOM which is used to merge .o files into an executable (reference to same list).
Some versions of gfortran and ifort don't need this, but others do (somehow).
"""
env.Append(LIBPATH=env['FORTRANPATH'])

"""
Speed project analysis up a little by using timestamps before hashing.
"""
env.Decider('MD5-timestamp')

"""
Create a custom builder (using `cpp`) for processing the header files.
"""
HeaderPP = Builder(
        action=Action('cpp -P $CPPFLAGS $_CPPDEFFLAGS $_FORTRANINCFLAGS $SOURCE > $TARGET'),
        suffix='.h', src_suffix='.H',
)
env.Append(BUILDERS = {'FHpp': HeaderPP})

"""
Optimization has debug, optimized but also mixed options.

For mixed mode, we clone the environment and make one optimized, one debug.

This would get confusing if there were mixed modes for many parameters, but it works well for now.
"""
already_made = []
if debug_subset:
        env_deb = env.Clone()
        env_deb.Append(
                **FLAGS[compiler]['deb']
        )
        for sub in debug_subset:
                if sub not in sources:
                        raise AssertionError('source "{0:s}" in debug_subset is not known - known sources: [{1:s}]'
                                .format(sub, ', '.join(sources)))
                sources.remove(sub)
                objs = env_deb.Object(join(dir, basename(sub) + '.o'), source=[join(dir, sub)])
                already_made.extend(obj for obj in objs if str(obj).endswith('.o'))

"""
Adding the debug/optimized flags needs to happen after cloning for mixed mode (if applicable).
"""
env.Append(
        **FLAGS[compiler][optimization]
)

"""
Python function to make a symbolic link (e.g. to the result in variantdir), which can be used as a compile action.
"""
def make_link(target, source, env):
        frm, to = str(source[0]), str(target[0])
        if islink(abspath(to)) or exists(abspath(to)):
                remove(abspath(to))
                print('Symlink {0:s} -> {1:s} [replaced]'.format(frm, to))
        else:
                print('Symlink {0:s} -> {1:s} [new]'.format(frm, to))
        symlink(abspath(frm), abspath(to))

"""
Specify how all files depend on each other, which SCons will resolve and compile.
"""
for header in list(join(dir, file) for file in glob('*.H')):
        env.FHpp(source=header)
sources += env['POTENTIAL_FILES']
pth_sources = list(join(dir, src) for src in sources) + already_made
target = env.Program(join(dir, 'sccalc'), source=pth_sources)
link_cmd = Command([GetOption('name')], [target], make_link)
Default(link_cmd)
Clean(link_cmd, link_cmd)
Clean(link_cmd, 'build')

"""
This is a debug hack: if `env` is given as target, print the environment (it'll crash automatically after).
"""
if 'env' in COMMAND_LINE_TARGETS:
        print(env.Dump())
        exit()

Comments

No comments yet

You need to be logged in to comment.