.. (C) Crown Copyright 2022-2025, Met Office.
.. The LICENSE.md file contains full licensing details.
.. include:: ../common.txt

The workflow
============

An overview of the workflow
---------------------------

|CMEW| performs the following steps:

``install_env_file``
  :Description:
     Activates the environment for |ESMValTool|, based on the ``SITE`` provided
  :Runs on:
     Localhost
  :Executes:
     The ``install_env_file.sh`` script from the |Rose| app
  :Details:
     Runs once at the start of the workflow

``configure_process``
  :Description:
     Creates and modifies the |ESMValTool| user configuration file,
     and writes it to the cylc workflow ``share/etc`` directory
  :Runs on:
     Localhost
  :Executes:
     The ``configure_process.py`` script from the |Rose| app
  :Details:
     Runs immediately after the successful completion of the ``install_env_file`` job.
     Temporarily, the modified ESMValTool developer configuration file is copied from
     the ``configure_process`` app to the ``share/etc`` directory in the installed workflow

``configure_for``
  :Description:
     Copies an updated version of the |ESMValTool| recipe for the assessment
     area into the cylc workflow ``share/etc`` directory in the installed
     workflow
  :Runs on:
     Localhost
  :Executes:
     For the required assessment area, executes the ``esmvaltool recipes get``
     command followed by the ``update_recipe_file.py`` script from the |Rose|
     app
  :Details:
     Runs once for each assessment area, immediately after the successful
     completion of the ``install_env_file`` job. The recipe is updated with
     CMEW required variables (e.g. "Activity": "ESMVal") and also with user
     configurable variables from the |Rose Edit GUI|_/``rose-suite.conf``
  :Families:
     ``ASSESSMENT_AREA``

``configure_standardise``
  :Description:
     Creates the ``request.json`` file and variables list which are needed to run
     |CDDS| and creates the |CDDS| directory structure.
  :Runs on:
     Localhost
  :Executes:
     The ``configure_standardise.sh`` script from the |Rose| app
  :Details:
     Runs once for each assessment_area, immediately after the successful
     completion of the ``configure_for`` job

``standardise_model_data``
  :Description:
     Launches the |CDDS| workflow and converts the data into a |CMIP| compliant
     format for |ESMValTool|
  :Runs on:
     Localhost
  :Executes:
     The ``cdds_convert`` command and the ``restructure_dirs.sh`` script
     from the |Rose| app
  :Details:
     Runs after the successful completion of the ``configure_standardise`` job.
     The ``restructure_dirs.sh`` script moves the standardised data into
     a directory with a BADC DRS structure so that |ESMValTool| can find the data

``housekeeping``
  :Description:
     Removes any intermediate files from the cylc run directory that were
     created by the |CDDS| workflow in ``standardise_model_data``
  :Runs on:
     Localhost
  :Executes:
     The |rose_prune|_ application
  :Details:
     Runs after the successful completion of the ``standardise_model_data`` job

``process``
  :Description:
     Runs the requested recipes using |ESMValTool|
  :Runs on:
     ``COMPUTE``, which depends on the ``SITE``; at the Met Office, the
     ``process`` jobs will run on SPICE
  :Executes:
     The |ESMValTool| command line script
  :Details:
     Runs once for each assessment area, after the successful completion of
     the ``standardise_model_data`` and the ``configure_process`` jobs
  :Families:
     ``COMPUTE``, ``ASSESSMENT_AREA``

``compare``
  :Description:
     Ensures the expected output files are generated by the ``process`` jobs
  :Runs on:
     Localhost
  :Executes:
     The ``compare.sh`` script from the |Rose| app
  :Details:
     Runs for every assessment area defined in the workflow, after the
     completion of the process job

``unittest``
  :Description:
     Locates and runs unit tests for the python scripts inside the ``app`` directory
  :Runs on:
     Localhost
  :Executes:
     ``pytest`` from the |Rose| app
  :Details:
     Runs on its own when ``-O unittest`` command is invoked, or runs alongside the
     full workflow when running with ``-O test``

Design considerations
---------------------

Portability
~~~~~~~~~~~

|CMEW| is portable; site-specific information can be found in the ``site`` and
``opt`` directories within the workflow. The files required are:

``site/<site>.cylc``
  Contains task definitions specific to the ``SITE``, for example, ``COMPUTE``

``site/<site>-standardise-env``
  Contains details on how to set up the environment for CDDS at the ``SITE``

``site/<site>-process-env``
  Contains details on how to set up the environment for ESMValTool at the
  ``SITE``

``opt/rose-suite-<site>.conf``
  Contains configuration items specific to the ``SITE``, including ``SITE``

Families
~~~~~~~~
Several tasks in |CMEW| are grouped into `families`_ with shared configurations.
The following families are used in the workflow:

``COMPUTE``
  A family that is inherited by computationally intensive tasks

``ASSESSMENT_AREA``
  A family that is inherited by tasks that run
  for each assessment area independently

Metadata
~~~~~~~~

|CMEW| uses Rose metadata. Every item defined in the suite configuration file
(``rose-suite.conf``) will have an entry in the main metadata configuration
file (``meta/rose-meta.conf``).

Resources
~~~~~~~~~

The resources used by the ``process`` jobs are defined in the
``site/<site>.cylc`` file, allowing the jobs to be configured by ``SITE`` as
well as by recipe. This ensures only the required resources are requested when
running each of the ``process`` jobs.