Integration Testing#

The purpose of integration testing is to examine the interation of a small group of units. (i.e. procedures) This group should not be arbitrary but represent some coherent piece of functionality.

To aid in creating such tests the LFRic core provides a framework for writing them and support in the build system.

Test Structure#

Following the pattern determined by unit testing each project may have an integration-test directory. Tests may be added at this level but it is often neater to further organise them with a directory tree.

Each test consists of at least a Fortran source file containing a program program unit and simlarly named Python script. e.g. special_test.f90 and special_test.py.

There may be additional Fortran source files but the purpose of the resulting executable is to set up and exercise the functionality being tested. It should communicate the result of this usually by writing to standard out.

The python script is then resposible for interpreting the output of the Fortran program and determining whether the test has passed or not.

Test Framework#

To help implement the Python script a framework is provided in infrastructure/build/testframework.

All tests must ultimately derive from testframework.test.AbstractTest which manages the running of the test but also provides methods which may be overridden for your own purposes.

The first is test() which is abstract so must be over-ridden. This is where you implement your test. You are given the return code of the test execution along with the contents of standard out and standard error. You may raise a testframework.exception.TestFailed exception to indicate a failure. Otherwise return a string describing the test which succeeded.

Optionally you may override the post_execution() method which is called after execution is complete but before the test is performed. This is useful for gathering additional output from the execution, e.g. XIOS output files.

There are also methods filterOut and filterErr which may be overridden in order to groom the standard out and standard error text before it is used in the test. For example, removing chatter from the MPI library.

The TestFailed exception is constructed with a message but may also relay return code, standard out, standard error and log file, depending on what is relevant to your test.

Finally testframework.testengine provides the TestEngine class which is used to execute tests.

Common Test Types#

The abstract test class can be extended to support any kind of test you wish to carry out however a number of common cases are already provided by the framework:

Test: Simple serial tests which involve running the executable and inspecting the standard out and standard error.
MpiTest: Launches the executable under MPI and strips any chatter from the MPI library before passing on standard out and standard error for scrutiny.
LFRicLoggingTest: Runs the test in parallel but also harvests the PETxxx.Log log files generated by the LFRic logging framework. These log files may be obtained by calling the getLFRicLoggingLog() method, passing the MPI rank you are interested in.

Example#

All of this is best illustrated with an example. We will be looking at testing the command-line argument handling capabilities of the infrastructure.

Firstly we need a program in infrastructure/integration-tests/cli_mod_test.f90:

program cli_mod_test

  use, intrinsic :: iso_fortran_env, only : output_unit
  use cli_mod, only : get_initial_filename

  implicit none

  character(:), allocatable :: filename

  call get_initial_filename( filename )
  write( output_unit, '(A)' ) filename

end program cli_mod_test

As you can see, this program simply fetches the configuration filename from the command line and prints it out.

With that in place we need a script (infrastructure/integration-tests/cli_mod_test.py) to test the result of executing the program and determining whether it is working or not:

import sys

from testframework import Test, TestEngine, TestFailed

class cli_mod_normal_test(Test):
  def __init__(self):
    self._INJECT = 'onwards/waffles.nml'
    super().__init__( [sys.argv[1], self._INJECT] )

  def test(self, returncode: int, out: str, err: str) -> str:
    if returncode != 0:
      raise TestFailed(f"Unexpected failure of test executable: {returncode}")

    if out.strip() != self._INJECT:
      raise TestFailed(
        f"Expected filename '{self._INJECT}' but found '{out.strip()}'."
      )

    return "Filename extracted from command line"

This first test intercepts object construction in order to keep a copy of the argument it is passing to the executable.

When the test has been run the first check is that the executable exited with a success (zero) return code. Provided it did, the standard output is checked against the argument passed in. Assuming everything was okay the passing test is described in the return value.

class cli_mod_too_few_test(Test):
  def test(self, returncode: int, out: str, err: str):
    if returncode == 0:
      raise TestFailed("Unexpected success with no arguments")

    return "Command line with no arguments returned with error"

This test is simpler, it doesn’t intercept any of the set up and just tests that the return code was not zero, a successful execution.

if __name__ == '__main__':
  TestEngine.run(cli_mod_normal_test())
  TestEngine.run(cli_mod_too_few_test())

Finally, when the script is executed, we use the TestEngine to run the tests we’ve defined.

Test Resources#

Some tests will need additional supporting resources. They may need a mesh or a namelist file. These can be placed in a directory resources within the test directory. This directory will be copied into the execution location so the contents should be accessed using filenames like resources/mesh.nc.

Sometimes a group of tests need common resources. These can be found in integration-test/support/resources. A symlink from the exeuction directory to this directory will be created and should be referred to like this: shared-resources/mesh.nc.