Generating CMORised data with CDDS for CMIP7 simulations using the CDDS Workflow

Tip

Use <script> -h or <script> --help to print information about the script, including available parameters.

Outline of CDDS processing

The following diagram gives a rough outline of the path for processing for CMIP7. Clicking on boxes will either take you directly to the appropriate form (UKCNSP github membership required) or to the appropriate section of this document.

flowchart TD
    A["(1) Register Simulation"] --> B["(2) Generate Request"]
    B --> C["(3) Create operational simulation issue (private)"]
    C --> D["(4) Modify Request file"]
    D --> E["(5-7) Run CDDS and monitor workflows"]
    E --> F["(8) Prepare operational simulation issue for review"]
    F --> G["(9) Tidy up"]

    click A "https://github.com/UKNCSP/CDDS-simulation-metadata/issues/new?template=add_workflow_metadata.yml" "Simulation registration form"
    click B "https://github.com/UKNCSP/CDDS-simulation-metadata/issues/new?template=generate_request_file.yml" "Request generation form"
    click C "https://github.com/UKNCSP/CDDS-CMIP7-processing/issues/new?template=ticket.yml" "Operational simulation issue form"
    click D "#4-modify-request-file"
    click E "#5-prerequisites-for-running"
    click F "#8-prepare-cdds-operational-simulation-issue-for-review-submission"
    click G "#9-run-cdds-teardown"

    style A fill:#FFADAD,stroke:#D77A7A,stroke-width:2px,color:#000000
    style B fill:#FFD6A5,stroke:#D6A26D,stroke-width:2px,color:#000000
    style C fill:#FDFFB6,stroke:#D6D777,stroke-width:2px,color:#000000
    style D fill:#CAFFBF,stroke:#9FD18F,stroke-width:2px,color:#000000
    style E fill:#A0C4FF,stroke:#6F96D6,stroke-width:2px,color:#000000
    style F fill:#BDB2FF,stroke:#8F84D6,stroke-width:2px,color:#000000
    style G fill:#FFC6FF,stroke:#D69FD6,stroke-width:2px,color:#000000

Please ensure that you have read the following documentation before starting your first set of CDDS processing or if you are unsure of the steps to take.

1. Simulation registration

Before processing data for CMIP7 your simulation metadata must be registered via the CDDS-simulation-metadata repository. Please create an issue via the Add/Modify Workflow Metadata form. Please be mindful that this is a public repository.

Once successful a workflow metadata file will be created in the workflow_metadata named according to the workflow id, and the table of runs here will be updated. A list of variables requested in the CMIP7 Data Request will be produced and added to the variables directory organised by Data Request version (e.g. v1.2.2.5). All variable list files are named in the format of <model_workflow_id>_<experiment>_<model>.txt (e.g. if you are running u-dv341 using the experiment piControl and the model UKCM2-0-LL, the variable list file will be named u-dv341_piControl_UKCM2-0-LL.txt). Please use the most recent Data Request version for processing unless you have a very good reason that you have discussed with the CDDS team.

Attempts to register workflow metadata that do not match the definition in the CMIP7 CVs will fail. If an experiment has not yet been registered or the parent information is inconsistent with the definition within CMIP7 the github actions will fail and a comment will be added to the issue to this effect.

Important

If the workflow metadata file contains incorrect information, this will be propagated through CDDS. As such, it is critically important that the information in these files is correct. Once data has been published it takes a non-trivial amount of effort to correct the problem, so please check that the data provided here is correct.

2. Request file generation

The request file is the core control file used to control CDDS and simulations need to be registered in order to generate one.
The form here can be used to generate a request; enter the workflow id, select the platform (Met Office or JASMIN), select the streams to be processed (see sub section below) and add a package name that is unique for the simulation being processed.

Note that certain fields within the request need to be manually updated and paths updated prior to use.

Important

Do not attempt to edit the request file directly on the CDDS-simulation-metadata repository. Changes should only be made to your locally downloaded copy. Any changes should be documented in the CDDS operational simulation issue (created in the next step).

Streams

Typical streams used for CDDS;

stream id	format	frequency	notes
`afx`	PP	fx	Fixed fields such as masks and orography
`ap4`	PP	monthly
`ap5`	PP	monthly
`apu`	PP	monthly	Unpacked data
`ap6`	PP	daily
`ap7`	PP	6 hourly
`ap8`	PP	3 hourly
`ap9`	PP	hourly
`apt`	PP	timestep	usually only used for site specific data
`ofx`	netCDF	fx	Fixed fields such as masks
`onm`	netCDF	monthly	ocean
`ond`	netCDF	daily	ocean
`inm`	netCDF	monthly	sea-ice
`ind`	netCDF	daily	sea-ice

Other streams may be usable, but please contact the CDDS team for advice.

Packages

CDDS is designed to handle a "package" of simulation data at one time; a set of variables from a particular simulation run. Multiple "packages" can be run for a given simulation to add new or corrected variables to the archive. Each package should be run using a separate processing (proc) and data directory. The simplest way to separate two run throughs of CDDS is to use a different package name. This is set through the package entry in the [common] section of the request file.

We recommend using either sequential names (round-1, round-2, etc) or a datestamp (round-2026-08-01) for the package field.

Important

Please ensure that package names are unique for a particular run through of CDDS. Failing to do this can lead to multiple problems and confusion when debugging problems.

Partial processing of a simulation

In certain circumstances it may be desirable to process and submit a subset of an entire simulation, i.e. the first 250 years of the esm-piControl simulation. Please contact the CDDS Team to discuss this prior to starting processing to:

Get appropriate guidance on the steps needed to correctly construct the request and variables file.
Arrange for an appropriate Errata to be issued following submission of data sets.

3. Creating an operational simulation issue

The CDDS operational simulation issue, see the list of CDDS operational simulation issues, is the primary route for documenting progress of processing and for requesting support with processing.

Once you have a request file (see above) and are ready to start processing please fill out the form to create a new operational issue, reporting the workflow ID, package and either linking to the request file and variables list or indicating that you would will attach the files directly to the issue.

When you are ready to start work on this issue please add the in progress label.

4. Modify Request file

We recommend a separate working directory for each CDDS package for clarity and run CDDS from this location. It would be helpful to document this in your CDDS operational simulation issue. Download the request config file and variables list and make any modifications updating your CDDS operational simulation issue accordingly.

Important

The request.cfg file contains all information that is needed to process the data through CDDS. The process of creating a request file on github does not set all values, it must be adjusted manually.

You will need to adjust your request.cfg:

Open the request.cfg via a text editor, e.g. vim or gedit.
The following values need to be set manually:

Section	Value	Description
`[data]`	`variable_list_file`	Full Path to your variable list file

Note

Please check the other values as well and make adjustments as needed. Ensure any adjustments are recorded on the CDDS operational simulation issue. For any help, please contact the CDDS Team or tag @UKNCSP/cdds on your issue.

Info

The MIP era (CMIP7) you are using is defined in the value mip_era of the metadata section.

Changes when working with local data

If you are working with data on disk, e.g. retrieved manually or through MOOMIN, the following changes will need to be made to the request file;

Section	Key	Value	Description
`[conversion]`	`skip_extract`	True	skip the extract tasks

To link your local data to the CDDS data input directory, activate the cdds installation (see below) and run

cdds_arrange_input_data <request file> <directory to search for data>

This will search the directory you specify for model output to be used and create symbolic links to each file.

Important

It is critially important that the data provided on disk covers the complete time range specified in the request file. Please ensure that in particular all data for the last and first years is present before proceeding with processing. Recovering from a missing month at the start or end of the process may be difficult so please ensure that any issues identified by the validation tasks are dealt with. Failing to do this may be difficult to recover from.

5. Prerequisites for running

Before proceeding with the CDDS Operational Procedure, please ensure that:

You belong to the cdds-data group.

Tip

Type groups on the command line to print the groups a user is in.
You have write permissions to moose:/adhoc/projects/cdds/ on MASS.
Tip

You can check if you have correct permissions by running following command and check if your moose username is included in the access control list output:
```
moo getacl moose:/adhoc/projects/cdds
```
If your user id is not included with the readwrite-delete permissions listed please contact the CDDS team so that you can be given the required permissions to archive data.
You use a bash shell. CDDS uses Conda which can experience problems running in a shell other than bash.
Tip

You can check which shell you use by following command:
```
echo $SHELL
```
If the result is not /bin/bash, you can switch to a bash shell by running:
```
/bin/bash
```

If any of the above are not true please contact the CDDS Team for guidance.

6. Activate the CDDS install

MOHCJASMIN

Setup the environment to use the central installation of CDDS and its dependencies:
```
source ~cdds/bin/setup_env_for_cdds <cdds_version>
```
where <cdds_version> is the version of CDDS you wish to use, e.g. 4.0.0. Unless instructed otherwise you should use the most recent version of CDDS available (to ensure that all bugfixes are picked up), and this version should be used in all stages of the package being processed. If in doubt contact the CDDS team for advice.
Issue: Record the version of CDDS being used on the CDDS operational simulation issue.

Setup the environment to use the central installation of CDDS and its dependencies:
```
source ~cdds/bin/setup_env_for_cdds <cdds_version>
```
where <cdds_version> is the version of CDDS you wish to use, e.g. 4.0.0. Unless instructed otherwise you should use the most recent version of CDDS available (to ensure that all bugfixes are picked up), and this version should be used in all stages of the package being processed. If in doubt contact the CDDS team for advice.
Issue: Record the version of CDDS being used on the CDDS operational simulation issue.

Note

The available version numbers for this script can be found here.
If you wish to deactivate the CDDS environment then you can use the command conda deactivate.

7. Run the CDDS workflow

If running a single request configuration, please follow the standard process as described in the Quickstart tutorial. Assuming you have your request file and variable list set up, this simply involves running the following 4 commands:

Activate the environment:

source ~cdds/bin/setup_env_for_cdds <the version of CDDS you are using, e.g. 4.0.0>

Create the directory structure:
```
create_cdds_directory_structure <path to your request.cfg>
```
. The output of this script suggests two symbolic links are created or the environment variables $CDDS_DATA_DIR and $CDDS_PROC_DIR are set. These variables will be used in examples below.

Create the internal variable lists used by CDDS:

prepare_generate_variable_list <path to your request.cfg>

Launch the cylc conversion workflow:

cdds_convert <path to your request.cfg>

This process can be monitored via the cylc gui or cylc review. If a workflow has issues, due to task failure, it will stall, and you will receive an e-mail.Further guidence on this process can be found in the Quickstart tutorial.

Monitor conversion workflow

A single cylc workflow will be launched by the cdds_convert command which requires monitoring by the user to handle any issues encountered. This can be done using cylchub, the cylc tui command or the cylc review service.

Conversion workflows will usually be named cdds_<workflow_base_name>_<stream> and each stream will run completely independently. If a workflow has issues, due to task failure, it will stall, and you will receive an e-mail.

If you hit issues or are unsure how to proceed update the CDDS operational simulation issue for your package with anything you believe is relevant (include the location of your working directory) add the label help needed and tag @UKNCSP/cdds or email the CDDS Team for advice.

The conversion workflows run the following steps:

run_extract_<stream>
Extract
- Run CDDS Extract for this stream.
- Runs in long queue with a wall time of 2 days.
- If there are any issues with extracting data they will be reported in the job.out log file in the workflow and the $CDDS_PROC_DIR/extract/log/cdds_extract_<stream>_<date stamp>.log log file and the task will fail.
- The extraction task will automatically resubmit 4 times if it fails and manual intervention is required to proceed.
- Most issues are related to either MASS (i.e. moo commands failing), file system anomalies (failure to create files /directories) or running out of time.
- Identify issues either by searching for "CRITICAL" in the job.out logs in Cylc Review or by using
```
grep CRITICAL $CDDS_PROC_DIR/extract/log/cdds_extract_<stream>_<date stamp>.log
```
- If the issue appears to be due to MASS issues you can re-run the failed CDDS Extract job by re-triggering the run_extract_<stream> task via the cylc gui or via the cylc command line tools:
```
cylc trigger cdds_<workflow_base_name>_<stream> run_extract_<stream>:failed
```
- If in doubt update your CDDS operational simulation issue and contact CDDS Team for advice.
validate_extract_<stream>

Extract Validation

Validation of the output is now performed as a separate task from extracting it. This task will report missing or unexpected files and unreadable netCDF files.
setup_output_dir_<stream>

Setup Output Directory

This task will create output directories for conversion output.
initialiser_<stream>

MIP Convert initialiser

Used to manage flow of tasks.
mip_convert_<stream>_<grid group>
MIP Convert
- Run MIP Convert to produce output files for a small time window for this simulation.
- Will retry up to 3 times before workflow stalls.
- CRITICAL issues are appended to $CDDS_PROC_DIR/convert/log/critical_issues.log. These will likely need user action to correct for. So, update your CDDS operational simulation issue and contact CDDS Team for advice.
- The CRITICAL log file will not exist if there are no critical issues.
- A variant named mip_convert_first_<stream>_<grid group> may be launched to align the cycling dates with the concatenation processing.
finaliser_<stream>

MIP Convert Finaliser

This ensures that Concatenation Tasks are launched once all MIP Convert tasks have been successfully performed for a particular time range. It also updates time, memory and disk usage requirements for the processing to optimise use of SLURM queues.
organise_files_<stream>
Organise Files
- Re-arranges the output files on disk from a directory structure created by the MIP Convert tasks of the form
```
$CDDS_DATA_DIR/output/<stream>_mip_convert/<YYYY-MM-DD>/<grid>/<files>
```
  to
```
$CDDS_DATA_DIR/output/<stream>_concat/<MIP table>/<variable name>/<files>
```
- Ready for concatenation. A variation named organise_files_final_<stream> does the same thing but at the end of the conversion process.
mip_concatenate_setup_<stream>
MIP Concatenate Setup
- This step constructs a list of concatenation jobs that must be performed
mip_concatenate_batch_<stream>
MIP Concatenate Batch
- Perform the concatenation commands (ncrcat) required to join small files together.
- Runs in long queue with a wall time of 2 days and can retry up to 3 times before workflow stalls (failures are usually due to running out of time while performing a concatenation).
- Only one mip_concatenate_batch_<stream> task can run at one time.
- Issues can be identified using:
```
grep CRITICAL $CDDS_PROC_DIR/convert/log/mip_concatenate_*.log
```
  If any critical issues arise or tasks fail update your CDDS operational simulation issue and tag @UKNCSP/cdds or contact the CDDS Team for advice.
- Output data is written to
```
$CDDS_DATA_DIR/output/<stream>/<MIP table>/<variable name>/<files>
```
run_repack_<stream>

Repacking

Runs the CMIP7 repacking tools on all output to optimise the data files for reading.
run_qc_<stream>
Quality Check (QC)
- Run the QC process on output data for this stream
- Produces a report at:
```
$CDDS_PROC_DIR/qualitycheck/report_<stream>_<datestamp>.json
```
  and a list of variables which pass the quality checks at:
```
$CDDS_PROC_DIR/qualitycheck/approved_variables_<stream>_<datestamp>.txt
```
  and a log file at:
```
$CDDS_PROC_DIR/qualitycheck/log/qc_run_and_report_<stream>_<datestamp>.log
```
- The approved variables file will have one line per successfully produced Dataset of the form:
```
<MIP table>/<variable name>;<Directory containing files>
```
- This task will fail if any QC issues are found and will not resubmit. If this occurs please update your CDDS operational simulation issue and contact the CDDS Team for advice.
run_transfer_<stream>
Transfer
- Archive data for variables that are marked active in the requested variables file produced by CDDS Prepare and have successfully passed the QC checks, i.e. are listed in the approved variables file.
- Will not automatically retry, even if failure was due to MASS/MOOSE issues.
- The location in MASS to which these data are archived is determined by the output_mass_suffix argument specified in the request configuration file.
- Task will fail if
- There are MASS issues: For example if the following command returns anything there has been a MASS outage and you can re-trigger the task:
```
grep SSC_STORAGE_SYSTEM_UNAVAILABLE $CDDS_PROC_DIR/archive/log/cdds_store_<stream>_<date stamp>.log
```
- An attempt is made to archive data that already exists in MASS. If this occurs please update your CDDS operational simulation issue and tag @UKNCSP/cdds add the help needed label to your issue or contact the CDDS Team for advice.
VERY IMPORTANT

Do not attempt to delete data from MASS without consultation with CDDS support.
run_sim_review

Simulation review

To ease the submission process our automated review script is run as the last task in the workflow. We will use this to determine whether data is fit for publication.
workflow_complete

Completion

This is a dummy task that is the last thing to run in the workflow -- this is to support inter workflow dependencies

If all goes well the workflow will complete. Please be sure to periodically check on your workflows progress and confirm that it has completed before updating your CDDS operational simulation issue.

What to do when things go wrong

On occasion issues will arise with tasks performed by users of CDDS and these will trigger CRITICAL error messages in the logs and usually require user intervention. Many simple issues (MASS/MOOSE or file system problems) can be resolved by re-triggering tasks. When you take any action please ensure that you update your CDDS operational simulation issue and if support is needed add the help needed label and mention @UKNCSP/cdds to alert the CDDS team.

A page describing common issues will be extended as new issues and their solutions are discovered.

8. Prepare CDDS operational simulation issue for review & submission

Once the workflows for a particular package have completed update your CDDS operational simulation issue confirming that the Extract, Convert, QC and Transfer tasks have been completed.

Copy the request JSON file and any logs to $CDDS_PROC_DIR.
```
cp request.json *.log $CDDS_PROC_DIR/
```
Add a comment to the CDDS operational simulation issue specifying the archived data is ready for submission, and include the full path to your request configuration location.
Add the label ready for submission to the CDDS operational simulation issue and assign it to Matthew Mizielinski by selecting this name from the list.
The issue will then be reviewed according to the CDDS simulation review procedure by members of the CDDS team.

9. Run CDDS Teardown

Once the approved issue has been returned to you following submission, delete the contents of the data directory:
```
cd <path to the data directory>;   
rm -rf input output
```

Delete all workflows used:

cdds_clean <path to the request configuration>

Update and close the CDDS operational simulation issue.