.. (C) Crown Copyright 2022-2025, Met Office. .. The LICENSE.md file contains full licensing details. .. include:: ../common.txt The workflow ============ An overview of the workflow --------------------------- |CMEW| performs the following steps: ``install_env_file`` :Description: Activates the environment for |ESMValTool|, based on the ``SITE`` provided :Runs on: Localhost :Executes: The ``install_env_file.sh`` script from the |Rose| app :Details: Runs once at the start of the workflow ``configure_process`` :Description: Creates and modifies the |ESMValTool| user configuration file, and writes it to the cylc workflow ``share/etc`` directory :Runs on: Localhost :Executes: The ``configure_process.py`` script from the |Rose| app :Details: Runs immediately after the successful completion of the ``install_env_file`` job. Temporarily, the modified ESMValTool developer configuration file is copied from the ``configure_process`` app to the ``share/etc`` directory in the installed workflow ``configure_for`` :Description: Copies an updated version of the |ESMValTool| recipe for the assessment area into the cylc workflow ``share/etc`` directory in the installed workflow :Runs on: Localhost :Executes: For the required assessment area, executes the ``esmvaltool recipes get`` command followed by the ``update_recipe_file.py`` script from the |Rose| app :Details: Runs once for each assessment area, immediately after the successful completion of the ``install_env_file`` job. The recipe is updated with CMEW required variables (e.g. "Activity": "ESMVal") and also with user configurable variables from the |Rose Edit GUI|_/``rose-suite.conf`` :Families: ``ASSESSMENT_AREA`` ``configure_standardise`` :Description: Creates the ``request.json`` file and variables list which are needed to run |CDDS| and creates the |CDDS| directory structure. :Runs on: Localhost :Executes: The ``configure_standardise.sh`` script from the |Rose| app :Details: Runs once for each assessment_area, immediately after the successful completion of the ``configure_for`` job ``standardise_model_data`` :Description: Launches the |CDDS| workflow and converts the data into a |CMIP| compliant format for |ESMValTool| :Runs on: Localhost :Executes: The ``cdds_convert`` command and the ``restructure_dirs.sh`` script from the |Rose| app :Details: Runs after the successful completion of the ``configure_standardise`` job. The ``restructure_dirs.sh`` script moves the standardised data into a directory with a BADC DRS structure so that |ESMValTool| can find the data ``housekeeping`` :Description: Removes any intermediate files from the cylc run directory that were created by the |CDDS| workflow in ``standardise_model_data`` :Runs on: Localhost :Executes: The |rose_prune|_ application :Details: Runs after the successful completion of the ``standardise_model_data`` job ``process`` :Description: Runs the requested recipes using |ESMValTool| :Runs on: ``COMPUTE``, which depends on the ``SITE``; at the Met Office, the ``process`` jobs will run on SPICE :Executes: The |ESMValTool| command line script :Details: Runs once for each assessment area, after the successful completion of the ``standardise_model_data`` and the ``configure_process`` jobs :Families: ``COMPUTE``, ``ASSESSMENT_AREA`` ``compare`` :Description: Ensures the expected output files are generated by the ``process`` jobs :Runs on: Localhost :Executes: The ``compare.sh`` script from the |Rose| app :Details: Runs for every assessment area defined in the workflow, after the completion of the process job ``unittest`` :Description: Locates and runs unit tests for the python scripts inside the ``app`` directory :Runs on: Localhost :Executes: ``pytest`` from the |Rose| app :Details: Runs on its own when ``-O unittest`` command is invoked, or runs alongside the full workflow when running with ``-O test`` Design considerations --------------------- Portability ~~~~~~~~~~~ |CMEW| is portable; site-specific information can be found in the ``site`` and ``opt`` directories within the workflow. The files required are: ``site/.cylc`` Contains task definitions specific to the ``SITE``, for example, ``COMPUTE`` ``site/-standardise-env`` Contains details on how to set up the environment for CDDS at the ``SITE`` ``site/-process-env`` Contains details on how to set up the environment for ESMValTool at the ``SITE`` ``opt/rose-suite-.conf`` Contains configuration items specific to the ``SITE``, including ``SITE`` Families ~~~~~~~~ Several tasks in |CMEW| are grouped into `families`_ with shared configurations. The following families are used in the workflow: ``COMPUTE`` A family that is inherited by computationally intensive tasks ``ASSESSMENT_AREA`` A family that is inherited by tasks that run for each assessment area independently Metadata ~~~~~~~~ |CMEW| uses Rose metadata. Every item defined in the suite configuration file (``rose-suite.conf``) will have an entry in the main metadata configuration file (``meta/rose-meta.conf``). Resources ~~~~~~~~~ The resources used by the ``process`` jobs are defined in the ``site/.cylc`` file, allowing the jobs to be configured by ``SITE`` as well as by recipe. This ensures only the required resources are requested when running each of the ``process`` jobs.