Processing in the Fab Base Class#

This chapter describes the processing of an application script using the Fab base class. The knowledge of this process will indicate how a derived, application-specific build script can overwrite methods to customise the build process.

The full class documentation is at the end of this chapter.

The constructor sets up ultimately the Fab BuildConfig for the build. It takes the name of the application as argument. The name of the application will be used when creating the name of the build directory and it is also the default root_symbol when analysing the source code if the script creates an executable (see analyse_step).

The actual build is then started calling the build method of the created script. A typical outline of a build script is therefore:

from fab.fab_base import FabBase

class ExampleBuild(FabBase):
    # Additional methods to be overwritten or added

if __name__ == '__main__':
    # Adjust logging level as appropriate
    logger = logging.getLogger('fab')
    logger.setLevel(logging.DEBUG)
    example = ExampleBuild(name="example")
    example.build()

The two main steps, construction and building, are described next.

Constructor#

As mentioned, the constructor will ultimately create the Fab BuildConfig object, which is then used to compile the application. The setup of this BuildConfig can be modified by overwriting the appropriate methods in a derived class, and site- and platform-specific setup can be done in a site- and platform-specific configuration script.

Defining site and platform#

The method define_site_platform_target() is first called. This method parses the arguments provided by the user and looks only for the options --site and --platform - any other option is for now ignored. These two arguments are then used to define a target attribute, which is either default or in the form {site}_{platform}. This name is used to identify the directory that contains the site-specific configuration script to be executed. The following three property getters can be used to access the values:

property FabBase.platform: str | None
Returns:

the platform, or None if not specified.

property FabBase.site: str | None
Returns:

the site, or None if no site is specified.

property FabBase.target: str
Returns:

the target (=”site-platform”), or “default” if nothing was specified.

Site-specific Configuration#

After defining the site and platform, a site- and platform-specific configuration script is executed (if available). By default, the base class will try to import a module called config from the path "site_specific/{target}" (see above for the definition of target), relative to the directory in which the application script is located. By overwriting the method setup_site_specific_location an application can setup its own directories by adding paths to sys.path.

FabBase.setup_site_specific_location()

This method adds the required directories for site-specific configurations to the Python search path. This implementation will search the call tree to find the first call that’s not from Fab, i.e. the user script. It then adds site_specific and site_specific/default to the directory in which the user script is located. An application can overwrite this method to change this behaviour and point at site-specific directories elsewhere.

Return type:

None

If the import is successful, the base class creates an instance of the Config class from the module. This all happens here:

FabBase.site_specific_setup()

Imports a site-specific config file. The location is based on the attribute target (which is set to be {site}_{platform}" based on the command line options, and the path is specified in ``setup_site_specific_location).

Return type:

None

Remember if no site- and no platform-name is specified, it will import the Config class from the directory called site_specific/default.

Chapter Site-Specific Configuration describes this class in more detail. An example for the usage of site-specific configuration is to add new compilers to Fab’s ToolRepository, or set the default compiler suite for a site.

Defining command line options#

After executing the site-specific configuration file, a Python argument parser is created with all command line options in the method define_command_line_options:

FabBase.define_command_line_options(parser=None)

Defines command line options. Can be overwritten by a derived class which can provide its own instance (to easily allow for a different description).

Parameters:

parser (Optional[ArgumentParser]) – optional a pre-defined argument parser. If not, a new instance will be created. (default: None)

Return type:

ArgumentParser

This method can be overwritten if an application want to add additional command line flags. The method gets an optional Python ArgumentParser: a derived class might want to provide its own instance to provide a better description message for the argument parser’s help message. See here for an example.

A special case is the definition of compilation profiles (like fast-debug etc). The base class will query the site-specific configuration object using get_valid_profiles() to receive a list of all valid compilation profile names. This allows each site to specify its own profile modes.

Parsing command line options#

Once all command line options are defined in the parser, the parsing of the command line options happens in:

FabBase.handle_command_line_options(parser)

Analyse the actual command line options using the specified parser. The base implementation will handle the –suite parameter, and compiler/linker parameters (including the usage of environment variables). Needs to be overwritten to handle additional options specified by a derived script.

Parameters:

parser (argparse.ArgumentParser) – the argument parser.

Return type:

None

The result of the parsing is stored in an attribute, which can be accessed using the args property of the script instance.

Again, this method can be overwritten to handle the added application-specific command line options.

Once the application’s handle_command_line_options has been executed, the method with the same name in the site-specific config file will also be called. It gets the argument namespace information from Python’s ArgumentParser as argument:

Config.handle_command_line_options(args)

Additional callback function executed once all command line options have been added. This is for example used to add Vernier profiling flags, which are site-specific.

Parameters:

args (argparse.Namespace) – the command line options added in the site configs

Return type:

None

This can be used for further site-specific modifications, e.g. it might add additional flags for the compiler or linker. Handling a new command line option shows an example of doing this.

Defining project name#

By default, the base class will use "{name}-{self.args.profile}-$compiler" as the name for the project directory, i.e. the name of the project as specified in the constructor, followed by the compilation profile and compiler name ($compiler is a Python template parameter and will be replaced by Fab).

A user script can overwrite define_project_name and define a different name:

FabBase.define_project_name(name)

This method defines the project name, i.e. the directory name to use in the Fab workspace. It defaults to name-profile-compiler.

Parameters:

name (str) – the base name of the project as specified by the caller.

Return type:

str

Returns:

the project name

Here an example where -mpi is added if MPI has been enabled on the command line. It calls the base class to add the compilation profile and compiler name.

def define_project_name(self, name: str) -> str:

    if self.args.mpi:
        name = name + "-mpi"
    return super().define_project_name(name)

BuildConfig creation#

After parsing the command line options, the base class will first create a Fab ToolBox which contains the compiler and linker selected by the user (see Fab documentation for details). Then it will create the BuildConfig object, providing the ToolBox and the appropriate command line options:

label = f"{name}-{self.args.profile}-$compiler"
self._config = BuildConfig(tool_box=self._tool_box,
                           project_label=label,
                           verbose=True,
                           n_procs=self.args.nprocs,
                           mpi=self.args.mpi,
                           openmp=self.args.openmp,
                           profile=self.args.profile,
                           )

Building#

While Fab provides a very flexible way in which the different phases of the build process can be executed, the base clas provides a fixed order in which these steps happen (though of course the user could overwrite the build method to provide their own order). If additional phases need to be inserted into the build process, this can be done by overwriting the corresponding steps, see Adding a new phase into the build process for an example.

The naming of the steps follows the Fab naming, but adds a _step as suffix to distinguish the methods from the Fab functions. Typically, an application will need to overwrite at least some of these methods (for example to specify the source files). This will require either adding calls to Fab methods, or just calling the base-class. Details will be provided in each section below.

grab_files_step#

This step is responsible for copying the required source files into the Fab work space (under the source folder). This method’s template in the base class should not be called, otherwise it will raise an exception, since any script must specify where to get the source code from. Typically, in this step various Fab functions are used to get the source files:

grab_folder

Fab’s grab_folder recursively copies a file directory into the fab work space. It requires that the source files have been made available already, e.g. either as a local working copy, or a checkout from a repository.

git_checkout

Fab’s git_checkout checks out a git repository, and puts the files into the working directory.

svn_export, svn_checkout

Fab provides these two interfaces to svn, and similar to git_checkout these will either export or checkout a Subversion repository.

grab_archive

This wii unpack common archive formats like tar, zip, tztar etc.

fcm_export, fcm_checkout

Compatibility layer to the old fcm configuration. This basically runs the corresponding Subversion commands.

A script can obviously use any other Python function to get or create source files.

find_source_files_step#

This step is responsible for identifying the source files that are to be used in the build process. While Fab has the ability to analyse the source tree and determine the minimal necessary set of files, it is possible that different versions of the same file would be found in the source tree (e.g. different version of the same file coming from different repositories that have been checked out). Since Fab does not support using the same file name more than once (and since in general it would lead to inconsistency if the same file name is used), Fab provides the ability to include or exclude files from its source directory in the Fab work space.

TODO: link to Fab’s documentation

This is typically done by specifying a list of path files. Each element in this list can be either an Exclude or an Include object, indicating that files of a specified pattern should be included or excluded. An example code:

path_filters = [
    Exclude('my_folder'),
    Include('my_folder/my_file.F90'),
]

These path files are then passed to Fab’s find_source_files function. For example:

# Setting up path_filters as shows above
find_source_files(self.config,
                  path_filters=([Exclude('unit-test', '/test/')] +
                                path_filters))

This step will not affect any files, it will just set up Fab’s ArtefactStore to be aware of the available source files.

Often, suites will provide FCM configuration that include a long list of files to exclude (and include) to avoid adding duplicated files into a complex build environment based on many source repositories.

extract.path-excl[um] = / # everything
extract.path-incl[um] =                              \
    src/atmosphere/AC_assimilation/iau_mod.F90       \
    src/atmosphere/PWS_diagnostics/pws_diags_mod.F90 \
    src/atmosphere/aerosols/aero_params_mod.F90      \
    ...

For convenience during porting, Fab provides a small tool to interface with existing FCM configuration files. This tool can read existing FCM configuration files, and convert the path-incl and path-excl directives into Fab’s Exclude and Include objects. Example usage:

extract_cfg = FcmExtract(self.lfric_apps_root / "build" / "extract" /
                          "extract.cfg")

science_root = self.config.source_root / 'science'
path_filters = []
for section, source_file_info in extract_cfg.items():
    for (list_type, list_of_paths) in source_file_info:
        if list_type == "exclude":
            # Exclude in this application removes the whole
            # app (so that individual files can then be picked
            # using include)
            path_filters.append(Exclude(science_root / section))
        else:
            # Remove the 'src' which is the first part of the name
            # in this script, which we don't have here
            new_paths = [i.relative_to(i.parents[-2])
                         for i in list_of_paths]
            for path in new_paths:
                path_filters.append(Include(science_root /
                                            section / path))

define_preprocessor_flags_step#

This method is called before preprocessing, and it allows the application to specify all flags required for preprocessing all C, and Fortran files.

FabBase.define_preprocessor_flags_step()

Top level function that sets preprocessor flags. The base implementation does nothing, should be overwritten.

Return type:

None

The base class provides its own method of adding preprocessor flags:

FabBase.add_preprocessor_flags(list_of_flags)

This function appends a preprocessor flags to the internal list of all preprocessor flags, which will be passed to Fab’s various preprocessing steps (for C, Fortran, and X90).

Each flag can be either a str, or a path-specific instance of Fab’s AddFlags object. For the convenience of the user, this function also accepts a single flag or a list of flags.

No checking will be done if a flag is already in the list of flags.

Parameters:

list_of_flags (Union[AddFlags, str, List[AddFlags], List[str]]) – the preprocessor flag(s) to add. This can be either a str or an AddFlags, and in each case either a single item or a list.

Return type:

None

Flags can be specified either as a single flag, or as a list of flags. Each flag can either be a simple string, which is a command line option for the compiler, or a path-specific flag using Fab’s AddFlags class (TODO: link to fab). Example code:

def define_preprocessor_flags(self):
    super().define_preprocessor_flags()

    self.add_preprocessor_flags(['-DUM_PHYSICS',
                                 '-DCOUPLED',
                                 '-DUSE_MPI=YES'])

    path_flags = [AddFlags(match="$source/science/jules/*",
                           flags=['-DUM_JULES', '-I$output']),
                  AddFlags(match="$source/large_scale_precipitation/*",
                           flags=['-I$relative/include',
                                  '-I$source/science/shumlib/common/src'])]

    self.add_preprocessor_flags(path_flags)
    # Add a preprocessor flag depending on compilation profile:
    if self.args.profile == "full-debug":
        self.add_preprocessor_flags("-DDEBUG")

preprocess_c_step#

There is usually no reason to overwrite this method. It will use the preprocessor flags defined in the previous define_preprocessor_flags_step and preprocess all C files.

FabBase.preprocess_c_step()

Calls Fab’s preprocessing of all C files. It passes the common and path-specific flags set using add_preprocessor_flags.

Return type:

None

preprocess_fortran_step#

There is usually no reason to overwrite this method. It will use the preprocessor flags defined in the previous define_preprocessor_flags_step and preprocess all Fortran files.

FabBase.preprocess_fortran_step()

Calls Fab’s preprocessing of all fortran files. It passes the common and path-specific flags set using add_preprocessor_flags.

Return type:

None

analyse_step#

This steps does the complete dependency analysis for the application. There is usually no reason for an application to overwrite this step.

In case of creating a binary, the analyse step will use the root symbol, which defaults to the name of the application, but can be changed using set_root_symbol. This implies that set_root_symbol must be called before analyse_step is called, e.g. it can be called from any method called from the constructor (including defining and handling command line options).

FabBase.analyse_step(find_programs=False)

Calls Fab’s analyse. It passes the config and root symbol for Fab to analyze the source code dependencies.

Find_programs:

if set and an executable is created (see link_target), the flag will be set in Fab’s analyse step, which means it will identify all main programs automatically.

Return type:

None

compile_c_step#

This step compiles all C files. There is usually no reason for an application to overwrite this step.

FabBase.compile_c_step(common_flags=None, path_flags=None)

Calls Fab’s compile_c. It passes the config for Fab to compile all C files. Optionally, common flags, path-specific flags and alternative source can also be passed to Fab for compilation.

Return type:

None

compile_fortran_step#

This step compiles all Fortran files. As it takes a list of path-specific flags as an argument, child classes can overwrite this method to pass additional path-specific flags.

FabBase.compile_fortran_step(common_flags=None, path_flags=None)

Calls Fab’s compile_fortran. It passes the config for Fab to compile all Fortran files. Optionally, common flags, path-specific flags and alternative source can also be passed to Fab for compilation.

Parameters:

path_flags (Optional[List[AddFlags]]) – optional list of path-specific flags to be passed to Fab compile_fortran, default is None. (default: None)

Return type:

None

archive_objects#

This step creates an archive with all compiled object files.

Warning

Due to MetOffice/fab#310 it is not recommended to create archives. Therefore, this step is for now not executed at all!