Architecture¶

The architecture of CSET. This page first describes some concepts, then documents the various workflow apps used, and finally ends with an outline of where things live in the codebase.

Operators¶

Operators provide the core functionality of CSET. Each operator is a function that takes some input and returns some output, in some cases producing plots.

All of the operators in CSET are contained in modules in src/CSET/operators/, which group related operators. For example plot.py contains various plotting operators.

Recipes¶

To produce a diagnostic, operators must be combined in recipes, which are YAML files containing a graph of operators to execute, along with any needed arguments and some metadata providing the diagnostic’s title, description and broad category.

The included recipes in CSET can be found in src/CSET/recipes/.

A recipe may optionally contain variables, such as $VARIABLE, which is replaced by a value provided on the command line or by a loader.

Loaders¶

Loaders load recipes for use in the workflow, filling in any variables from the configuration provided in the rose-suite.conf.

The included loaders in CSET can be found in src/CSET/loaders/.

Workflow¶

To enable large scale running of CSET we provide a cylc workflow. The workflow’s source can be found in src/CSET/cset_workflow/, with its logic being defined in the flow.cylc file.

This workflow has three main aims:

Fetching the model and observation data.
Running all of the enabled recipes on that data.
Building a visualisation website for the produced diagnostics.

These aims are accomplished by a series of rose-apps that are run as part of the workflow. These apps live in src/CSET/cset_workflow/app/.

validate_environment¶

A small shell script that checks the conda environment has been loaded correctly and that the cset command line is available at the start of the workflow.

assign_model_colours¶

Runs at the start of the workflow to assign each model a colour, so that different line plots and such can use consistent colours for each model. The colours are assigned into a style file which is used when baking the recipes.

install_website_skeleton¶

Copies static files for the visualisation website to use and creates a symlink from the configured WEB_DIR to $CYLC_WORKFLOW_SHARE_DIR/web.

In future this app is likely to be merged into finish_website.

fetch_fcst¶

Runs for each model/obs source on each cycle to retrieve the required data. It can switch between different implementations for different data sources, such as the filesystem or HTTP.

parbake_recipes¶

Runs for each cycle. Reads the user configuration and the writes out all the enabled recipes with their variables filled in. This allows them to be “baked” (run) in parallel.

bake_recipes¶

Runs for each cycle, and additionally as bake_aggregation_recipes in the final cycle. This takes the parbaked recipes and runs them to produced the desired diagnostics.

This will be that task that takes the majority of the workflow’s runtime, and is the only one that needs significant compute resource.

Internally it runs the cset bake command line for each recipe in parallel using rose_bunch, however to allow the baked recipes to be decided at runtime the first thing that runs is the baker.sh script, which writes out the list of recipes to bake as a rose optional configuration, and then run the rose app using it.

finish_website¶

Runs at the end of the workflow and constructs the index for the visualisation website from all of the produced diagnostics.

housekeeping¶

Deletes the retrieved data at the end of the workflow to free up disk space.

send_email¶

Sends a notification email to the workflow owner letting them know the workflow is complete.

Other apps¶

The following apps aim to integrate METplus in the workflow but are not currently used:

demo_pointstat
metplus_ascii2nc
metplus_grid_stat
metplus_point_stat

Code outline¶

The code of CSET lives in the src/CSET/ directory, arranged as follows:

src/CSET¶

src/CSET
├── cset_workflow                  # Detailed below for clarity.
├── loaders
│   ├── __init__.py                # Imports all loaders for the rest of CSET.
│   └── ...                        # Then lots of loaders, as described above.
├── operators
│   ├── __init__.py                # Code for executing ("baking") recipes.
│   ├── _colorbar_definition.json  # Default colourbar definitions.
│   ├── _plot_page_template.html   # Template for diagnostic output page.
│   ├── _stash_to_lfric.py         # Mapping between STASH codes and LFRic variable names.
│   ├── _utils.py                  # Common utility code for operators.
│   └── ...                        # Then lots of operators, as described above.
├── recipes
│   ├── __init__.py                # Code for parbaking recipes.
│   └── ...                        # Then lots of recipes, as described above.
├── __init__.py                    # CLI entrypoint. Logging setup, argument parsing, etc.
├── __main__.py                    # Allows running `python -m CSET`.
├── _common.py                     # Common utility code.
├── extract_workflow.py            # Implementation of `cset extract-workflow`.
└── graph.py                       # Implementation of `cset graph`.

src/CSET/cset_workflow¶

src/CSET/cset_workflow
├── app                          # Contains the rose apps described above.
│   ├── assign_model_colours
│   │   ├── bin
│   │   │   └── assign_model_colours.py  # Executable for app.
│   │   └── rose-app.conf        # Rose app configuration. Mostly sets the executable.
│   └── ...                      # Lots more rose apps in here.
├── bin                          # Files in bin are automatically on the workflow's PATH.
│   └── app_env_wrapper          # Wrapper script to run things in the conda environment.
├── lib                          # Available for import into cylc's jinja2 templating.
│   └── python
│       └── jinja_utils.py       # A couple helper functions used in flow.cylc.
├── meta                         # Validation and GUI layout for user configuration.
│   ├── diagnostics
│   │   └── rose-meta.conf       # Diagnostic configuration.
│   ├── rose-meta.conf           # Automatically generated file, don't edit.
│   └── rose-meta.conf.jinja2    # Workflow configuration.
├── opt                          # Pre-made configurations for consistent evaluation.
│   └── rose-suite-RAL3LFRIC.conf
├── site                         # Site-specific cylc configuration.
│   └── localhost.cylc
├── flow.cylc                    # Workflow definition detailing how tasks are run.
├── install_restricted_files.sh  # Script for installing site-specific files.
├── README.md
├── rose-suite.conf              # User configuration of workflow and diagnostics.
└── rose-suite.conf.example      # Blank user configuration to be copied.