The workflow

An overview of the workflow

CMEW performs the following steps:

install_env_file
Description:

Activates the environment for ESMValTool, based on the SITE provided

Runs on:

Localhost

Executes:

The install_env_file.sh script from the Rose app

Details:

Runs once at the start of the workflow

configure_process
Description:

Creates and modifies the ESMValTool user configuration file, and writes it to the cylc workflow share/etc directory

Runs on:

Localhost

Executes:

The configure_process.py script from the Rose app

Details:

Runs immediately after the successful completion of the install_env_file job. Temporarily, the modified ESMValTool developer configuration file is copied from the configure_process app to the share/etc directory in the installed workflow

configure_for
Description:

Copies an updated version of the ESMValTool recipe for the assessment area into the cylc workflow share/etc directory in the installed workflow

Runs on:

Localhost

Executes:

For the required assessment area, executes the esmvaltool recipes get command followed by the update_recipe_file.py script from the Rose app

Details:

Runs once for each assessment area, immediately after the successful completion of the install_env_file job. The recipe is updated with CMEW required variables (e.g. “Activity”: “ESMVal”) and also with user configurable variables from the Rose Edit GUI/rose-suite.conf

Families:

ASSESSMENT_AREA

configure_standardise
Description:

Creates the request.json file and variables list which are needed to run CDDS and creates the CDDS directory structure.

Runs on:

Localhost

Executes:

The configure_standardise.sh script from the Rose app

Details:

Runs once for each assessment_area, immediately after the successful completion of the configure_for job

standardise_model_data
Description:

Launches the CDDS workflow and converts the data into a CMIP compliant format for ESMValTool

Runs on:

Localhost

Executes:

The cdds_convert command and the restructure_dirs.sh script from the Rose app

Details:

Runs after the successful completion of the configure_standardise job. The restructure_dirs.sh script moves the standardised data into a directory with a BADC DRS structure so that ESMValTool can find the data

housekeeping
Description:

Removes any intermediate files from the cylc run directory that were created by the CDDS workflow in standardise_model_data

Runs on:

Localhost

Executes:

The rose_prune application

Details:

Runs after the successful completion of the standardise_model_data job

process
Description:

Runs the requested recipes using ESMValTool

Runs on:

COMPUTE, which depends on the SITE; at the Met Office, the process jobs will run on SPICE

Executes:

The ESMValTool command line script

Details:

Runs once for each assessment area, after the successful completion of the standardise_model_data and the configure_process jobs

Families:

COMPUTE, ASSESSMENT_AREA

compare
Description:

Ensures the expected output files are generated by the process jobs

Runs on:

Localhost

Executes:

The compare.sh script from the Rose app

Details:

Runs for every assessment area defined in the workflow, after the completion of the process job

unittest
Description:

Locates and runs unit tests for the python scripts inside the app directory

Runs on:

Localhost

Executes:

pytest from the Rose app

Details:

Runs on its own when -O unittest command is invoked, or runs alongside the full workflow when running with -O test

Design considerations

Portability

CMEW is portable; site-specific information can be found in the site and opt directories within the workflow. The files required are:

site/<site>.cylc

Contains task definitions specific to the SITE, for example, COMPUTE

site/<site>-standardise-env

Contains details on how to set up the environment for CDDS at the SITE

site/<site>-process-env

Contains details on how to set up the environment for ESMValTool at the SITE

opt/rose-suite-<site>.conf

Contains configuration items specific to the SITE, including SITE

Families

Several tasks in CMEW are grouped into families with shared configurations. The following families are used in the workflow:

COMPUTE

A family that is inherited by computationally intensive tasks

ASSESSMENT_AREA

A family that is inherited by tasks that run for each assessment area independently

Metadata

CMEW uses Rose metadata. Every item defined in the suite configuration file (rose-suite.conf) will have an entry in the main metadata configuration file (meta/rose-meta.conf).

Resources

The resources used by the process jobs are defined in the site/<site>.cylc file, allowing the jobs to be configured by SITE as well as by recipe. This ensures only the required resources are requested when running each of the process jobs.