Changelog (CDDS)

Release 3.1.2rc1, April 11 2025

• The conversion workflow now runs on JASMIN/LOTUS2 (CDDS-584)

• Added a jasmin_account option to the conversion section in the request (CDDS-584)

• Removed unnecessary module load nccmp from the setup script (CDDSO-644)

• Fixed cylc_args not being passed through to cylc in cdds_convert (CDDSO-641)

• Fixed atmos_timestep not being included in the configure config files (CDDSO-645)

Release 3.1.1, March 19 2025

• Added the 365_day calendar to supported calendars (CDDSO-634)

• Documentation improvements including, migrating remaining sphinx pages,

  removal of the legacy sphinx docs, updating the release procedure,

  and adding abbreviations (CDDSO-598, CDDSO-605, CDDSO-609)

• Fixed a race condition in qc when creating the sqlite database (CDDSO-366)

• The ensemble_workflow setup task will now fail properly when there are issues

  with running commands (CDDSO-624)

• Renamed the HadGEM3-GC50-N640ORCA12 model to HadGEM3-GC50-EERIE-N640ORCA12,

  in GCModelDev (CDDSO-611)

• Fixed a bug in model parameters validation (CDDSO-627)

• Archiving of experiments with sub-experiment id now works correctly (CDDSO-631)

Release 3.1.0, February 13 2025

• MIP Convert and CMOR log files are copied into the cylc task directory and

  are accessible via cylc review (CDDSO-495)

• Cylc workflows have been integrated into the CDDS code base rather than

  being checked out from the roses-u repository (CDDSO-494)

• Processing for multiple streams is now handled within a single Cylc workflow

  (CDDSO-302)

• A validation tool has been added for the model parameters file validate_model_parameters

  (CDDSO-511)

• CDDS Prepare will remove MIP Convert templates from the configure proc directory

  to avoid inconsistencies in processing. A --reconfigure command line argument

  has been added to enable the regeneration of the mip convert config templates

  from the command line (CDDSO-593)

• Ancillary STASH variables can now be configured via the CDDS Plugins (CDDSO-582)

• Added support for HadGEM3-GC50-N640ORCA12 model, which includes allowing for

  alternative coordinate bounds variables in NEMO output files (CDDSO-543)

• Updated CMOR to v3.9.0 (CDDSO-589)

• Migrated content from Sphinx docs to mkdocs (CDDSO-598, CDDSO-600)

Release 3.0.6, January 21, 2025

• Pass model parameters file to CDDS suite during convert process

  (CDDSO-578)

• Handling quarterly data during the extract process correctly

  (CDDSO-568)

• Add documentation to deployment page (CDDSO-556)

• Allow QC file size limit to be overloaded in the request file

  (CDDSO-577, CDDSO-587)

• Fix checkout processing workflow on JASMIN (CDDSO-574)

• Add documentation for model parameters file (CDDSO-545)

Release 3.0.5rc1, December 5, 2024 (only to be used for LESFMIP)

• Ensured that quarterly PP files, i.e. those that have 3 months of

  output at monthly/daily frequency can be extracted and picked up by

  the wrapper to MIP convert (CDDSO-560, CDDSO-568)

Release 3.0.4, November 19, 2024

• Enabled sub-annual slicing through the slicing option in the

  common section of the request configuration file. Streams can

  individually be set to have month slicing where large amounts of

  memory usage are expected (e.g. hourly data) while the default is

  year, i.e. annual slicing (CDDSO-342)

• Corrected bug preventing QC from completing for CMIP6 like

  processing introduced in CDDS v3.0.2 (CDDSO-532)

• Changed concatenation code to handle monthly and daily Datasets in

  the same stream where the first two facets of the file name are not

  unique to a Dataset (CDDSO-548)

• Updated HadREM3 sizing info (CDDSO-548)

Release 3.0.3, October 24, 2024

• Grid descriptions can now be modified through model parameter files

  (CDDSO-534, CDDSO-528)

• The inclusion of rotated coordinates in output files can now be

  forces through the introduction of the force_coordinate_rotation

  flag into the request configuration files (CDDSO-529)

• Global attribute caching has been introduced to improve QC speed

  (CDDSO-514)

Release 3.0.2, September 27, 2024

• Restored the approved variables file argument to the move_in_mass

  script (CDDSO-523)

• Ensure that hourly data and submonthly data is correctly recognised

  by CDDS when running MIP Convert (CDDSO-512)

• Prevent Extract tasks from failing when assessing moose command

  block size if some data is already present (CDDSO-515)

• Correctly handle leap years (Gregorian calendar) during

  concatenation task (CDDSO-513)

• Correctly handle sub annual chunk size for files in concatenation

  setup task (CDDSO-504)

• Correctly overload model parameters file when reading request file

  (CDDSO-508)

• Add functionality to remove halo columns and rows from atmosphere

  data (CDDSO-521)

• Correct parsing of filename facet for CORDEX in QC (CDDSO-519)

• Ensure clear checking of time coordinate in QC with the

  corresponding frequency and length in QC (CDDSO-517)

Release 3.0.1, August 21, 2024

• Verbose mode is not added as default option when running CDDS

  Convert workflow to avoid Cylc issue (CDDSO-490)

• Added new options to delete pre-existing proc and data directories

  in request.cfg (CDDS-408)

• Changes to port CDDS to Azure (CDDS-481, CDDSO-497)

• QC and archive will now fail if nothing is to check or to archive

  (CDDSO-459)

• Improved handling of failed moo commands (CDDSO-487)

• Extract will now fail when it cannot identify the chunking for moo

  commands (CDDSO-487)

• Prepare to accept variables like 6hrPlevPt/zg7h (CDDSO-473)

• The sim review script now accepts the request configuration file as its

  input argument (CDDSO-461)

• Added a new script cdds_clean to clean out the CDDS Convert

  workflows (CDDSO-450)

• Added support for CORDEX (CDDSO-421)

• Replacde hardcoded paths using the CDDS_ETC variable in tests

  (CDDSO-492)

• Extract will now fail if the tape check fails (CDDSO-482)

• Added validations for many fields within the request configuration file

  (CDDSO-292)

• Allow inheritance of request fields from a template (CDDSO-496)

Release 3.0.0, May 15, 2024

• The request JSON file, used to control CDDS in previous versions,

  has been replaced with a config file containing all required options

  and the command line interfaces to CDDS tools have been

  correspondingly updated (CDDSO-434, CDDSO-466, CDDSO-424, CDDSO-301,

  CDDSO-411, CDDSO-299, CDDSO-298, CDDSO-296, CDDSO-297, CDDSO-438,

  CDDSO-423, CDDSO-386, CDDSO-300, CDDSO-395)

• CDDS Documentation is now maintained through mkdocs and served via

  github pages (CDDSO-427, CDDSO-430, CDDSO-409, CDDSO-429, CDDSO-426,

  CDDSO-425, CDDSO-428, CDDSO-443, CDDSO-446, CDDSO-442)

• MIP Convert failure will now lead to cylc task failure if the

  continue_if_mip_convert_failed setting in the request

  ([conversion] section) is set to False in the request configuration file

  (CDDSO-387)

• The branch of the CDDS Convert workflow being used by default is

  controlled by the CDDS_CONVERT_WORKFLOW_BRANCH environment

  variable (CDDSO-452)

• Log messages are now sent to standard out by default, and will

  appear in job.out files within the cylc log directory (CDDSO-34)

• CMOR has been updated to v3.8 and Python to 3.10 (CDDSO-411)

• Simplified model config files by removing

  files_per_year field (CDSSO-307)

Release 2.5.10, December 12, 2024

• No changes

Release 2.5.9, June 25, 2024

• Fixed a bug in submission code that made archiving script unusable

  (CDDSO-477)

• QC now allows missing standard name attribute if it is defined as

  empty in MIP Tables (CDDSO-478)

Release 2.5.8, February 29, 2024

• Added submission queues to support CMIP6Plus publication (CDDSO-413)

• Removed modified time check from CDDS workflow to avoid workflow

  freezing when moved to a different cylc server (CDDSO-405)

Release 2.5.7, February 21, 2024

• Chunking of large moose queries in Extract is now correctly handled

  (CDDSO-398)

• QC now supports Conventions field in a more portable way (CDDSO-407)

• Concatenation code can now cope with monthly files of monthly mean

  data (CDDSO-405)

• Extensions and fixes to support CP4A processing via GCModel Dev

  (CDDSO-354, CDDSO-401, CDDSO-405)

Release 2.5.6, February 09, 2024

• Added HadGEM3-GC5 and eUKESM1-1-ice model configurations to

  GCModelDev plugin (CDDSO-390, CDDSO-391)

• Extraction code now correctly handles coordinate variables when

  extracting data from NEMO diaptr and scalar files (CDDSO-394)

• The data version used in archiving can now be specified in the CDDS

  workflow to support additional runs of CDDS appending to the same

  data set (CDDSO-393)

Release 2.5.5, January 18, 2024

• The name of netCDF bounds coordinates variables are now configurable

  via plugins to support retrieval of NEMO4 ocean data (CDDSO-346)

• GCModelDev plugin now has an N96 GC3.05 generic model

  configuration and the list streams for the N216 equivalent has been

  extended (CDDSO-381)

• The option to overload model configuration via command line now

  correctly operates (CDDSO-376, CDDSO-379, CDDSO-380)

• The create_variables_csv_file script now also outputs

  the stream for each variable (CDDSO-332)

Release 2.5.4, December 13, 2023

• Corrected an issue with a legacy configuration file that prevented

  the submission of CMIP6 data (CDDSO-371)

• Implement annual ocean diagnostics for the GCOyr MIP

  table (CDDSO-369)

Release 2.5.3, November 22, 2023

• Adaptations needed for new CMIP6Plus MIP tables (CDDSO-336,

  CDDSO-365)

• Adaptations needed to support use of sub_experiment_ids that are not

  "none" (CDDSO-365, CDDSO-362)

• Extract can again correctly retrieve sub-daily data (CDDSO-358)

• The update_directives script in the per-stream suite now correctly

  calculate upper limits for memory / wall time / local storage

  (CDDSO-360)

Release 2.5.2, October 18, 2023

• Polar row masking is now specified directly in the MIP Convert

  config files (CDDSO-331)

• Memory, storage and time limits for SLURM jobs are now dynamically

  calculated after the first three cycles of the processing suite

  (CDDSO-212)

• CDDS can now handle hourly data files within a stream (CDDSO-349)

• The correct filenames for ocean data from ensemble suites are now

  used again (CDDSO-350)

• The CDDS suite uses a consistent calendar to the request, which

  ensures that concatenation processes, QC and archiving are correctly

  triggered (CDDSO-351)

• All options are now passed through to CDDS Convert by the CDDS

  processing workflow (CDDSO-348)

Release 2.5.1, August 4, 2023

• Ensure that recent versions of the NCO tools used for concatenation

  of files do not modify metadata, leading to publication issues

  (CDDSO-327)

• Fix creation_date validation bug in QC when data is written on the

  31st of a month (CDDSO-326)

Release 2.5.0, July 27, 2023

• Migrate CDDS to Cylc 8 (CDDSO-282, CDDSO-285, CDDSO-184)

• Introduce gregorian calendar support (CDDSO-231, CDDSO-235,

  CDDSO-240, CDDSO-244)

• Plugin and code development to support Regional models for projects

  such as CORDEX (CDDSO-233, CDDSO-68, CDDSO-74, CDDSO-261, CDDSO-149)

• CDDS now uses ISO datetime format for all time values in CDDS and

  MIP Convert config files (CDDSO-313, CDDSO-45)

• Relocate test and etc directories to support alternate platforms and

  migration to azure SPICE (CDDSO-131, CDDSO-273)

• Replace pep8 with pycodestyle (CDDSO-207)

• Separated extract validation into a separate script & task

  (CDDSO-243)

• Introduced retries to workflow tasks when submission fails

  (CDDSO-314)

• Amalgamated CDDS processing and ensemble processing workflows.

  Previous tools based on u-cq805 and u-cr273 are deprecated

  (CDDSO-319)

• Updated CMOR to version 3.7.2, which required iris update to 3.4.1

  (CDDSO-321)

Release 2.4.7, June 29, 2023

• Fix a bug in the dehaloing script that prevented it from reading

  default command line arguments (CDDSO-308)

Release 2.4.6, June 23, 2023

• Fix a bug in the extraction code where the non-uniform allocation of

  data in a simulation to a large number of tapes can lead to failure

  of cdds extract (CDDSO-245, CDDSO-289)

• Correct a bug in the HadGEM3 model config files that prevented

  extraction of data from the diaptr sub-stream in the NEMO ocean data

  for the GCModelDev project (CDSO-293)

Release 2.4.5, May 22, 2023

• Added HighResMIP models to CMIP6 plugin (CDDSO-158, cherry-picked

  from main)

Release 2.4.4, May 4, 2023

• Added options to use a "relaxed" approach to MIP name (activity_id),

  and experiment id, that does not force checks from the CVs. This

  will allow users to process data as any experiment id they like, but

  at the risk of typos due to the lack of checks against a controlled

  vocabulary (CDDSO-253 , CDDSO-267, CDDSO-264)

• prepare_alter_variable_list can now perform insert

  commands again as plugin was not being loaded, which was needed for

  setting streams (CDDSO-269)

• checkout_processing_suite now sets the streams in

  the suite based on those in the request file. (CDDSO-271)

Release 2.4.3, March 31, 2023

• Add support for retries in the run_extract task within the CDDS

  Convert Suite (CDDSO-258)

Release 2.4.2, March 1, 2023

• Add support for filepaths with ensemble_id for netCDF model output

  (CDDSO-238).

Release 2.4.1, January 18, 2023

• Create a Cylc workflow for end to end processing (CDDSO-198, 200)

• Create a Cylc workflow for ensemble processing (CDDSO-199)

• Implement the checkout_processing_suite script for

  checking a copy of the CDDS Processing Suite (CDDSO-210)

• Create a Rose app for generating request.json files

  (CDDSO-3)

• Fix a bug where cdds_convert expect external plugins

  environmental variables to be set even if no plugin was used

  (CDDSO-216)

Release 2.4.0, September 12, 2022

• Refactoring of all CDDS packages into one package excluding MIP

  Convert (CDDSO-132, 133, 134, 135, 136, 137, 138, 139, 140, 141,

  142, 144)

• Move test execution from nose to pytest (CDDSO-128)

• Add sphinx docs (CDDSO-169)

• diaptr files for HadGEM3-GC31-MM can now be extracted from MASS

  (CDDSO-191)

Release 2.3.2, September 01, 2022

• (cdds_common): Default stream information for variables now held by

  Project level plugins (CDDSO-143)

• (cdds_configure): Log message issued when a stream cannot be found

  for a variable is now CRITICAL (CDDSO-178)

• (cdds_configure): GCAmon and GCLmon MIP tables from GCModelDev are

  now included in dictionary of default grids and will not be ignored

  (CDDSO-178)

• (cdds_convert): Logger now set up before configure step is called

  allowing log information to be output (CDDSO-177)

• (cdds_convert): MIP Era now correctly propagated through to CDDS

  rose suite (CDDSO-180)

• (hadsdk): Change log message to CRITICAl for missing grid/stream.

• (hadsdk): Prepare a MIP convert branch to produce sample decadal

  data with additional seasonal forecasting metadata

Release 2.3.1, June 29, 2022

• (cdds_common): Added auto-genereated table of CDDS variable mappings

  (CDDSO-120)

• (cdds_common): Fix inconsistency in case between mip era and plugin

  name (CDDSO-132)

• (extract): remove_ocean_haloes script now works after change to load

  plugin (CDDSO-152)

Release 2.3.0, May 24, 2022

• (cdds_common): Development moved to github

• (cdds_common): Version string now includes git commit hash

  (CDDSO-93)

• (cdds_configure): Development moved to github

• (cdds_convert): Development moved to github

• (cdds_prepare): Development moved to github

• (cdds_qc): Development moved to github

• (cdds_qc_plugin_cf17): Development moved to github

• (cdds_qc_plugin_cmip6): Development moved to github

• (cdds_transfer): Development moved to github

• (extract): Development moved to github

• (hadsdk): Development moved to github

• (transfer): Development moved to github

Release 2.2.5, May 4, 2022

• (cdds_common): The atmosphere timestep, used in some mappings, is

  now supplied via the ModelParameters class rather than a dictionary.

  This is only used for request JSON file creation (#2571)

• (cdds_configure): For non-CMIP6 projects the correct grid label and

  grid information is applied for variables where the grid is

  different to the default for that mip table (#2570)

• (cdds_prepare): When inserting variables into CMIP6 processing the

  correct stream information is included (#2569)

• (hadsdk): For non-CMIP6 projects the correct grid label and grid

  information is applied for variables where the grid is different to

  the default for that mip table (#2570)

• (hadsdk): The atmosphere timestep, used in some mappings, is now

  supplied via the ModelParameters class rather than a dictionary.

  This is only used for request JSON file creation (#2571)

Release 2.2.4, April 22, 2022

• (cdds_common): Added UKESM1-1-LL model to CMIP6 and GCModelDev

  projects (#2545)

• (transfer): Removed further calls to the moo test command when

  archiving data (#2559, #2560)

Release 2.2.3, April 7, 2022

• (cdds_convert): The default cycling frequency will reduce to match

  the run bounds length when the run bounds length is smaller than

  cycling frequency. (#2006)

• (cdds_prepare): Reintroduce the capability to fall-back on default

  CMIP6 stream mappings when parsing user variable file (#2555)

• (cdds_qc): Ignore QC errors caused by non-existant dimensions

  (#2548)

• (cdds_qc_plugin_cf17): The implementation of the geographical region

  checker has been corrected (#2548)

• (extract): MASS paths with and without the ensemble_id in them are

  now handled (#2522)

• (transfer): moose listing commands are used rather than `moo

  test` in order to limit the number of commands talking

  to MASS, thereby limiting our vulnerability to transient or load

  dependent errors, and increases performance (#2553)

Release 2.2.2, March 18, 2022

• (cdds_common): Addition of GCModelDev plugin with core CMIP6 models

  (#2519)

• (cdds_prepare): Streams can now be set on creation of the requested

  variables file (#2498)

• (cdds_qc): QC checks now refer to provided Controlled Vocabulary for

  non-CMIP6 projects (#2542)

• (cdds_qc_plugin_cmip6): QC checks now refer to provided controlled

  vocabulary for non-CMIP6 projects (#2542)

Release 2.2.1, February 15, 2022

• (cdds_common): UKESM1-ice-LL model information now loaded correctly

  (#2530)

• (cdds_convert): --external_plugins argument now

  passed through to all tasks (#2511)

• (cdds_qc): CF checks are now correctly filtered for CMIP6 data

  (#2531)

• (cdds_qc_plugin_cf17): CF checks are now correctly filtered for

  CMIP6 data (#2531)

Release 2.2.0, February 9, 2022

• (cdds_common): New module to take common code across CDDS and will

  ultimately replace hadsdk (#2460)

• (cdds_common): Introduced plugin system for project information

  (e.g. CMIP6) and model descriptions to enable use of CDDS outside of

  CMIP6 (#2460, #2461, #2462, #2468, #2494, #2502, #2503,

  #2504, #2509, #2510, #2512, #2513, #2514)

• (cdds_common): Update to use python 3.8 (#2438)

• (cdds_configure): Introduced plugin system for project information

  (e.g. CMIP6) and model descriptions to enable use of CDDS outside of

  CMIP6 (#2460, #2461, #2462, #2468, #2494, #2502, #2503,

  #2504, #2509, #2510, #2512, #2513, #2514)

• (cdds_configure): Update to use python 3.8 (#2438)

• (cdds_configure): Allow for the use of CDDS beyond CMIP6 (#2449,

  #2469, #2470)

• (cdds_convert): Introduced plugin system for project information

  (e.g. CMIP6) and model descriptions to enable use of CDDS outside of

  CMIP6 (#2460, #2461, #2462, #2468, #2494, #2502, #2503,

  #2504, #2509, #2510, #2512, #2513, #2514)

• (cdds_convert): Update to use python 3.8 (#2438)

• (cdds_convert): Enable the use of CDDS for ensemble class

  simulations (#2501, #2471)

• (cdds_convert): Allow for the use of CDDS beyond CMIP6 (#2449,

  #2469, #2470)

• (cdds_convert): When file concatenation is not required, i.e. when a

  single cycle covers the whole duration of the processing, files are

  now moved to the correct destination rather than being left in the

  _concat directory (#2521)

• (cdds_prepare): Introduced plugin system for project information

  (e.g. CMIP6) and model descriptions to enable use of CDDS outside of

  CMIP6 (#2460, #2461, #2462, #2468, #2494, #2502, #2503,

  #2504, #2509, #2510, #2512, #2513, #2514)

• (cdds_prepare): Update to use python 3.8 (#2438)

• (cdds_prepare): Enable the use of CDDS for ensemble class

  simulations (#2501, #2471)

• (cdds_prepare): Allow for the use of CDDS beyond CMIP6 (#2449,

  #2469, #2470)

• (cdds_qc): Introduced plugin system for project information (e.g.

  CMIP6) and model descriptions to enable use of CDDS outside of CMIP6

  (#2460, #2461, #2462, #2468, #2494, #2502, #2503, #2504,

  #2509, #2510, #2512, #2513, #2514)

• (cdds_qc): Update to use python 3.8 (#2438)

• (cdds_qc): CDDS QC now uses the community version of

  IOOS/compliance-checker rather than a fork (#2381)

• (cdds_qc): Allow for the use of CDDS beyond CMIP6 (#2449, #2469,

  #2470)

• (cdds_qc): Fixed a bug where if the string _concat

  was used in the branch name two tests in cdds_qc would

  fail (#2521)

• (cdds_qc_plugin_cf17): Introduced plugin system for project

  information (e.g. CMIP6) and model descriptions to enable use of

  CDDS outside of CMIP6 (#2460, #2461, #2462, #2468, #2494,

  #2502, #2503, #2504, #2509, #2510, #2512, #2513, #2514)

• (cdds_qc_plugin_cf17): Update to use python 3.8 (#2438)

• (cdds_qc_plugin_cf17): CDDS QC now uses the community version of

  IOOS/compliance-checker rather than a fork (#2381)

• (cdds_qc_plugin_cf17): Allow for the use of CDDS beyond CMIP6

  (#2449, #2469, #2470)

• (cdds_qc_plugin_cmip6): Introduced plugin system for project

  information (e.g. CMIP6) and model descriptions to enable use of

  CDDS outside of CMIP6 (#2460, #2461, #2462, #2468, #2494,

  #2502, #2503, #2504, #2509, #2510, #2512, #2513, #2514)

• (cdds_qc_plugin_cmip6): Update to use python 3.8 (#2438)

• (cdds_qc_plugin_cmip6): CDDS QC now uses the community version of

  IOOS/compliance-checker rather than a fork (#2381)

• (cdds_qc_plugin_cmip6): Allow for the use of CDDS beyond CMIP6

  (#2449, #2469, #2470)

• (cdds_transfer): Introduced plugin system for project information

  (e.g. CMIP6) and model descriptions to enable use of CDDS outside of

  CMIP6 (#2460, #2461, #2462, #2468, #2494, #2502, #2503,

  #2504, #2509, #2510, #2512, #2513, #2514)

• (cdds_transfer): Update to use python 3.8 (#2438)

• (extract): Introduced plugin system for project information (e.g.

  CMIP6) and model descriptions to enable use of CDDS outside of CMIP6

  (#2460, #2461, #2462, #2468, #2494, #2502, #2503, #2504,

  #2509, #2510, #2512, #2513, #2514)

• (extract): Update to use python 3.8 (#2438)

• (extract): Enable the use of CDDS for ensemble class simulations

  (#2501, #2471)

• (extract): Allow for the use of CDDS beyond CMIP6 (#2449, #2469,

  #2470)

• (hadsdk): Introduced plugin system for project information (e.g.

  CMIP6) and model descriptions to enable use of CDDS outside of CMIP6

  (#2460, #2461, #2462, #2468, #2494, #2502, #2503, #2504,

  #2509, #2510, #2512, #2513, #2514)

• (hadsdk): Update to use python 3.8 (#2438)

• (hadsdk): Retired umfilelist and all connected code (#2438)

• (transfer): Introduced plugin system for project information (e.g.

  CMIP6) and model descriptions to enable use of CDDS outside of CMIP6

  (#2460, #2461, #2462, #2468, #2494, #2502, #2503, #2504,

  #2509, #2510, #2512, #2513, #2514)

• (transfer): Update to use python 3.8 (#2438)

Release 2.1.2, November 25, 2021

• (extract): Extract will now repeat creating stream directories if

  the first time fails because of intermittent filesystem issues

  (#2429)

• (hadsdk): Fixed an import bug in the mip_table_editor script

  (#2463)

Release 2.1.1, October 26, 2021

• (cdds_prepare): prepare_generate_variable_list can now generate

  requested variables files using a user supplied list and the MIP

  tables being used, i.e. without reference to a Data Request (#2358)

• (extract): Added the path_reformatter command line tool from the

  cdds_utils directory (#2415)

• (hadsdk): Improved error logging for the

  write_rose_suite_request_json script (#2390)

• (transfer): Added scripts for listing submission queues and

  resending stored messages that failed during submission (#2427,

  #2428)

Release 2.1.0, September 6, 2021

• (cdds_configure): generate_user_config_files is now

  run by cdds_convert by default (#1902, #2425)

• (cdds_convert): The Extract, Configure, QC and archiving steps are

  now by default part of the conversion suite run by

  cdds_convert (#1902, #2383, #2425, #2432)

• (cdds_prepare): The --alternate_experiment_id

  argument no longer triggers CRITICAL issues` (#2380)

• (cdds_qc): qc_run_and_report is now run as part of the

  conversion suite by cdds_convert by default (#1902,

  #2425, #2432)

• (extract): cdds_extract is now run as part of the

  conversion suite by cdds_convert by default (#1902,

  #2425)

• (transfer): cdds_store is now run as part of the

  conversion suite by cdds_convert by default (#1902,

  #2425, #2432)

• (transfer): cdds_sim_review now handles per stream

  logs, qc reports and approved variables files (#1685)

Release 2.0.7, July 15, 2021

• (extract): The remove_ocean_haloes command line tool is now

  available to remove halo rows and columns from ocean data files

  (#1161)

Release 2.0.6, June 29, 2021

• (cdds_prepare): Added functionality to

  prepare_generate_variable_list to automatically deactivate

  variables following a set of rules hosted in the CDDS repository

  (default) or in a text file. This \"auto deactivation\" can be

  skipped using the --no_auto_deactivation argument and the

  --auto_deactivation_file_name option can be used to use a local

  file rather than the repository rules (#2405)

Release 2.0.5, June 11, 2021

• (cdds_qc): Floating point comparison will now not cause time

  contiguity checks to fail (#2396)

Release 2.0.4, May 17, 2021

• (cdds_prepare): Allow for correct handling of packages where there

  are multiple mips / activity ids. The primary, i.e. first, mip will

  be used for all directory and Dataset ids (#2369)

• (cdds_prepare): Altered output of create_cdds_directory_structure

  to be a little more user friendly (#2369)

• (cdds_transfer): Allow for correct handling of packages where there

  are multiple mips / activity ids. The primary, i.e. first, mip will

  be used for all directory and Dataset ids (#2369)

• (hadsdk): Allow for correct handling of packages where there are

  multiple mips / activity ids. The primary, i.e. first, mip will be

  used for all directory and Dataset ids (#2369)

• (transfer): Allow for correct handling of packages where there are

  multiple mips / activity ids. The primary, i.e. first, mip will be

  used for all directory and Dataset ids (#2369)

Release 2.0.3, April 28, 2021

• (transfer): CDDS store can now prepend files to an embargoed Dataset

  provided the correct data version argument is provided (#2278)

Release 2.0.2, April 22, 2021

• (cdds_convert): Implemented changes necessary for conda deployment

  of CDDS code on JASMIN (#2240)

• (cdds_qc): QC can now use a user specified directory with MIP tables

  provided by a command line option (#2331)

• (hadsdk): Inventory can now be constructed or updated from CREPP

  database dump on JASMIN (#2148)

• (hadsdk): Corrected masking of polar rows for the ORCA12 grid

  (#2211)

• (hadsdk): Implemented changes necessary for conda deployment of CDDS

  code on JASMIN (#2240)

Release 2.0.1, March 25, 2021

• (cdds_qc): QC now correctly interprets the values within string

  coordinates, such as basin name (#2284)

• (extract): When running for a particular stream the stream name is

  now included in the log file name (#2014)

• (hadsdk): write_rose_suite_request_json can now create request

  files for the UKESM1-ice-LL model (#2264)

• (hadsdk): write_rose_suite_request_json now allows the

  apt stream (#2283)

• (transfer): CDDS store can now recover gracefully when continuing

  archiving following task failure due to issues with MASS (#2205)

• (transfer): CDDS store can now correctly handle pre-pending files to

  Datasets and there are clearer error messages when issues arise

  (#2222)

Release 2.0.0, February 24, 2021

• (cdds_configure): Updated CDDS codebase to Python 3.6.

• (cdds_convert): Updated CDDS codebase to Python 3.6.

• (cdds_prepare): Updated CDDS codebase to Python 3.6.

• (cdds_qc): Updated CDDS codebase to Python 3.6.

• (cdds_qc_plugin_cf17): Updated CDDS codebase to Python 3.6.

• (cdds_qc_plugin_cmip6): Updated CDDS codebase to Python 3.6.

• (cdds_transfer): Updated CDDS codebase to Python 3.6.

• (extract): Updated CDDS codebase to Python 3.6.

• (hadsdk): Updated CDDS codebase to Python 3.6.

• (transfer): Updated CDDS codebase to Python 3.6.

Release 1.6.5, February 22, 2021

• (cdds_qc): Fixed a bug in diurnal cycle diagnostics checker

  (#2254).

Release 1.6.4, February 11, 2021

• (hadsdk): write_rose_suite_request_json now correctly handles

  fields that are commented in the rose-suite.info file, an issue that

  affects the use of this script for amip-like simulations in v1.6.3

  (#2232)

Release 1.6.3, February 9, 2021

• (cdds_qc): Implemented support for diurnal cycle diagnostics (1hrCM

  frequency) (#1894)

• (hadsdk): Errors in rose suite metadata (rose-suite.info file) now

  cause write_rose_suite_request_json to fail rather than warn and

  branch dates are correctly handled (#1477)

• (hadsdk): Users can now override the start and end dates for

  processing when creating request files with

  write_rose_suite_request_json (#2212)

• (transfer): The failure of [mooput commands is now

  handled more clearly (#2212)

Release 1.6.2, January 11, 2021

• (cdds_qc_plugin_cmip6): The cell_measures validation is now

  properly processing variables with --MODEL flag (#2193)

Release 1.6.1, November 26, 2020

• (cdds_qc_plugin_cmip6): The cell_measures validation is now

  properly processing global averages (#2171)

• (extract): Fixed a bug in the validation report, where the expected

  number of files would be printed instead of the actual file count

  (#2174)

Release 1.6.0, November 05, 2020

• (cdds_configure): All exceptions caught at the top script level are

  now logged as critical errors (#1968)

• (cdds_convert): All exceptions caught at the top script level are

  now logged as critical errors (#1968)

• (cdds_convert): Implemented code changes needed to support ARCHER

  filepath format and run CDDS on JASMIN (#2013)

• (cdds_convert): cdds_convert tasks now will fail if individual

  Concatenation Tasks do not succeed (#2045)

• (cdds_prepare): As a default option,

  prepare_generate_variable_list now performs CMIP6 inventory check

  to determine which variables have already been produced,

  deactivating them automatically (#1899)

• (cdds_prepare): Added requested ensemble size to the

  requested_variable_file file (#1900)

• (cdds_prepare): The write_rose_suite_request_json script now

  correctly writes the name of the rose suite branch in the

  [requestjson file (#2001)

• (cdds_prepare): All exceptions caught at the top script level are

  now logged as critical errors (#1968)

• (cdds_qc): All exceptions caught at the top script level are now

  logged as critical errors (#1968)

• (cdds_qc_plugin_cmip6): Consistency checks for netCDF variable

  attributes are now correctly applied (#2001)

• (cdds_qc_plugin_cmip6): Replaced the CMIP6 Controlled Vocabulary

  tables with the ones bundled with CMOR (#1808)

• (cdds_transfer): All exceptions caught at the top script level are

  now logged as critical errors (#1968)

• (extract): All exceptions caught at the top script level are now

  logged as critical errors (#1968)

• (hadsdk): Implemented a new command line tool search_inventory for

  exploring archived Datasets and checking their publication status

  (#1903)

• (hadsdk): Implemented code changes needed to support ARCHER filepath

  format and run CDDS on JASMIN (#2013)

Release 1.5.5, October 20, 2020

• (cdds_prepare): The requested variable list can now be constructed

  using an alternative experiment_id via a new command line argument

  to prepare_generate_variable_list. This should only be used under

  advice from the CDDS team (#2057)

• (cdds_transfer): move_in_mass now correctly includes variables,

  where the and name used in the output file is different, when

  submitting data for publication (#2049).

Release 1.5.4, October 7, 2020

• (cdds_convert): Updated file sizing configuration for the

  high-resolution MM model now yields files under 20 Gb (#2037)

Release 1.5.3, September 16, 2020

• (cdds_convert): Increased memory limits, TMPDIR storage and changed

  cycling frequencies (#2012)

Release 1.5.2, September 4, 2020

• (cdds_qc_plugin_cmip6): Fixed a bug that prevented from validation

  of some global attributes of simulations without a parent experiment

  (#2000)

Release 1.5.1, August 20, 2020

• (cdds_convert): The concatenation processes should no longer leave

  temporary netCDF files behind when the batch job hits its wall time

  (#1895)

• (cdds_convert): The check on the use of TMPDIR storage in the MIP

  Convert tasks is now correctly applied (#1943)

• (cdds_convert): The cycling frequencies for the conversion process

  can now been overridden from the command line (#1944)

• (extract): Information on the validation of the is now written to

  \<stream>_validation.txt as well as to the log files (#1811)

• (transfer): cdds_store can now archive CF sites datawith frequency

  subhrPt (#1945)

• (transfer): cdds_store will now raise CRITICAL log messages, rather

  than ERROR when something goes wrong (#1934)

Release 1.5.0, July 2, 2020

• (cdds_configure): Settings and files needed for production of CF

  sites data are now correctly written to the MIP Convert template

  files (#1838)

• (cdds_convert): CDDS can now produce CF Sites data (#1838)

• (cdds_prepare): Added do-not-produce information to the requested

  variables file via a new producible attribute for each variable

  (#681)

• (cdds_prepare): Added tool to export variable information from the

  requested variables file to a CSV or text file;

  create_variables_table_file (#1793)

• (cdds_qc): CDDS QC can now check CF sites (subhr frequency) data

  (#1783)

• (extract): CDDS Extract now correctly reports failure through error

  codes and e-mails (#1807)

• (extract): CDDS Extract can now correctly retrieve daily MEDUSA data

  (#1803)

Release 1.4.5, June 16, 2020

• (hadsdk): Institution names and branching dates are now correctly

  processed and written by the write_rose_suite_request_json script

  (#1848, #1849)

Release 1.4.4, June 1, 2020

• (cdds_convert): Memory limits for ap4 stream have been extended to

  12G to avoid task failure (#1843)

• (cdds_convert): The copying of files to $TMPDIR will now be

  retried twice and if all attempts fail the task will fail rather

  than allowing CRITICAL errors from MIP Convert (#1839)

Release 1.4.3, May 12, 2020

• (cdds_convert): CDDS can now process data for the model

  UKESM1-ice-LL (#1513)

• (cdds_convert): Added --scale-memory-limits argument to CDDS

  Convert to allow memory limits to be altered at run time if

  necessary (#1806)

• (cdds_prepare): CDDS can now process data for the model

  UKESM1-ice-LL (#1513)

• (cdds_prepare): Requested variable lists can now be generated using

  Data Request version 01.00.32 (#1800)

• (cdds_qc): CDDS can now process data for the model UKESM1-ice-LL

  (#1513)

• (cdds_qc_plugin_cf17): CDDS can now process data for the model

  UKESM1-ice-LL (#1513)

• (cdds_qc_plugin_cmip6): CDDS can now process data for the model

  UKESM1-ice-LL (#1513)

• (cdds_transfer): CDDS can now process data for the model

  UKESM1-ice-LL (#1513)

• (extract): CDDS can now process data for the model UKESM1-ice-LL

  (#1513)

• (hadsdk): CDDS can now process data for the model UKESM1-ice-LL

  (#1513)

• (hadsdk): Added write_rose_suite_request_json to construct a

  request JSON file from the rose-suite.info file within a CMIP6 suite

  (#1732)

• (transfer): CDDS can now process data for the model UKESM1-ice-LL

  (#1513)

Release 1.4.2, April 30, 2020

• (cdds_qc_plugin_cmip6): CDDS QC now handles variables where the

  cell_measures variable attribute is optional (#1801)

• (hadsdk): write_request_json now gives a useful error message when

  there is inconsistent data in CREM for a package (#1765)

Release 1.4.1, April 28, 2020

• (cdds_convert): CDDS Convert now does not raise an AttributeError

  relating to the --email-notifications argument when run (#1780)

• (cdds_qc): CDDS QC can now check hourly frequency Datasets and does

  not raise a KeyError exception (#1782)

Release 1.4.0, April 23, 2020

• (cdds_convert): Sub-daily data can now be produced (#825, #1577,

  #1704)

• (cdds_convert): The memory requested for different tasks within the

  suites is now controlled on a per grid/stream basis (#941, #1474)

• (cdds_convert): E-mail alerts now issued by suites launched by CDDS

  Convert when completing or stalling (#1669)

• (cdds_qc): Variables where a coordinate has text values, e.g.

  Omon/hfbasin, can now be properly validated (#1456)

• (cdds_qc): Datasets at sub-daily frequencies can now be validated

  (#1578)

• (cdds_qc): CRITICAL failures are now logged for each Dataset that

  fails a test (#1681)

• (extract): Extract can now extract data from sub-daily PP streams

  (#359)

• (extract): Extract processes for individual streams can now be run

  in separate tasks (#1619)

• (hadsdk): When writing request JSON files from CREM, warnings are

  issued if base dates are not 1850-01-01 (#1703)

Release 1.3.4, March 27, 2020

• (cdds_prepare): prepare_alter_variable_list now includes

  additional variable metadata allowing cdds_store_spice to operate

  correctly for these variables (#1621)

• (cdds_qc): QC has better support for processing a single stream

  using the --stream command line option (#1622)

• (cdds_transfer): move_in_mass now only lists directories in MASS

  connected with the Request provided, significantly reducing run time

  (#1649).

• (transfer): Errors arising from MOO failures are now properly

  handled and logged (#1701)

• (transfer): Transfer now supports processing variables with

  different output names to their variable ID, and has better support

  for processing a single stream using the --stream command line

  option (#1622)

Release 1.3.3, February 25, 2020

• (cdds_convert): cdds_convert script now correctly returns a

  non-zero error code on error (#1501).

• (cdds_convert): The conversion process can now be limited to

  specified streams via the --streams command line argument (#1594)

• (cdds_convert): The conditions under which an organise_files_final

  task have been modified to avoid scheduling duplicate file

  management processes (#1585)

• (extract): Extract will no longer log CRITICAL errors when

  processing data streams containing no retrievable variables (#1583,

  #1354).

Release 1.3.2, January 27, 2020

• (cdds_convert): CRITICAL error messages from MIP Convert are

  correctly captured in the critical_issues.log file, and known

  exceptions from within MIP Convert will no longer lead to task

  failure in CDDS Convert suites (#1533).

• (cdds_qc): Implemented file size checker (#1530)

• (extract): Clarified CRITICAL messages when PP files are found to be

  incomplete (#1528)

• (extract): Extract does not now raise an error when attempting to

  interpret the request file for HadGEM3-GC31-MM packages (#1563)

• (transfer): cdds_store now handles experiment ids including the

  - character (#1556)

Release 1.3.1, January 20, 2020

• (cdds_configure): Moved to updated scitools environment

  production_legacy-os43-1; python v2.7, iris v2.2 (#1512)

• (cdds_convert): Moved to updated scitools environment

  production_legacy-os43-1; python v2.7, iris v2.2 (#1512)

• (cdds_prepare): Moved to updated scitools environment

  production_legacy-os43-1; python v2.7, iris v2.2 (#1512)

• (cdds_qc): Moved to updated scitools environment

  production_legacy-os43-1; python v2.7, iris v2.2 (#1512)

• (cdds_qc_plugin_cf17): Moved to updated scitools environment

  production_legacy-os43-1; python v2.7, iris v2.2 (#1512)

• (cdds_qc_plugin_cmip6): Moved to updated scitools environment

  production_legacy-os43-1; python v2.7, iris v2.2 (#1512)

• (cdds_transfer): Moved to updated scitools environment

  production_legacy-os43-1; python v2.7, iris v2.2 (#1512)

• (extract): Moved to updated scitools environment

  production_legacy-os43-1; python v2.7, iris v2.2 (#1512)

• (hadsdk): Moved to updated scitools environment

  production_legacy-os43-1; python v2.7, iris v2.2 (#1512)

• (transfer): Moved to updated scitools environment

  production_legacy-os43-1; python v2.7, iris v2.2 (#1512)

• (transfer): cdds_store_spice now passes all arguments through to

  spice job (#1511)

• (transfer): cdds_store behaves correctly when there are no files

  to archive (#1520)

Release 1.3.0, January 14, 2020

• (cdds_prepare): The new format of the approved variables file is now

  handled (#1175).

• (cdds_prepare): Added EmonZ/sltbasin to the list of known good

  variables to avoid CRITICAL issue for UKESM1 processing (#1437)

• (cdds_qc): A column with Dataset filepaths has been added to the

  approved variables file (#1175)

• (cdds_qc): The quality checking can now be restricted to a single

  stream (#1248)

• (cdds_qc): CDDS QC now validates parent information when the

  branch_method is no parent (#1453)

• (cdds_transfer): The new format of the approved variables file is

  now handled (#1175).

• (cdds_transfer): Added new archiving tool cdds_store to replace

  send_to_mass and cdds_transfer_spice (#1384)

• (extract): CDDS Extract no longer connects to CREM to retrieve

  request information (#1079, #1080).

• (extract): --root_proc_dir and --root_data_dir arguments are now

  correctly interpreted by the cdds_extract_spice command (#1478).

• (extract): Error messages related to unreadable pp files are now

  being correctly logged (#1154).

• (transfer): Added new archiving tool cdds_store to replace

  send_to_mass and cdds_transfer_spice (#1384)

Release 1.2.3, November 26, 2019

• (hadsdk): thetaot and sltbasin can now be produced after

  correcting their stream (#1009)

Release 1.2.2, November 13, 2019

• (cdds_prepare): The create_cdds_directory_structure script now

  sets permissions on $CDDS_PROC_DIR/archive/log appropriately so

  users don\'t have to (#1347)

• (cdds_qc): The Data Request version can now be overridden from the

  command line when running qc_run_and_report (#1375)

• (cdds_transfer): The failure to submit a RabbitMQ message, which

  triggers the publication process, is now accompanied by a critical

  log message (#1346)

• (extract): CDDS Extract will now log WARNING rather than CRITICAL

  messages when there are no variables to be extracted for a stream

  (#1354)

• (extract): All moose commands are now non-resumable, in order to

  avoid cases where MASS outages lead to conflicts with CDDS Extract

  processes (#1374)

Release 1.2.1, October 29, 2019

• (cdds_convert): The cdds_convert.log file is now written to the

  convert/log processing directory rather than the working directory

  (as seen at v1.2.0) (#1341)

• (cdds_convert): CDDS Convert now ignores streams where there are no

  to be processed rather than raising a KeyError (#1282)

• (cdds_qc): The validation of the CMIP6 license in now compares the

  license global attribute against the text provided in the

  request.json file (#1297)

• (extract): A bug in the validation process, introduced in v1.2.0,

  that resulted in all NEMO model output files being incorrectly

  identified as unreadable, and then deleted, has been fixed (#1333)

Release 1.2.1, October 28, 2019

• (hadsdk): PP header fixes are now correctly applied when producing

  information required to construct query files for moo select

  commands in CDDS Extract. This enables the correct extraction of

  data for clisccp (#1330)

Release 1.2.0, October 17, 2019

• (cdds_configure): The cdds_configure script is now called

  generate_user_config_files (#1043)

• (cdds_configure): The --mohc and --root_config command line

  options for the generate_user_config_files script have now been

  removed (all arguments provided by the CDDS configuration files are

  now provided via command line options, see below; #1043)

• (cdds_configure): The --data_request_version, --template_name,

  --root_proc_dir and root_data_dir command line options for the

  generate_user_config_files script have now been added (#1043,

  #1070)

• (cdds_convert): The --root_config command line option for the

  cdds_convert script has now been removed (all arguments provided

  by the CDDS configuration files are now provided via command line

  options, see below; #1070)

• (cdds_convert): The --root_proc_dir and root_data_dir command

  line options for the cdds_convert script have now been added

  (#1070)

• (cdds_prepare): The --root_config command line option for the

  create_cdds_directory_structure and

  prepare_generate_variable_list scripts has now been removed (all

  arguments provided by the CDDS configuration files are now provided

  via command line options, see below; #1164)

• (cdds_prepare): The --root_proc_dir and root_data_dir command

  line options for the create_cdds_directory_structure script have

  now been added (#1164)

• (cdds_prepare): The --data_request_version,

  --data_request_base_dir, --mapping_status, --root_proc_dir and

  root_data_dir command line options for the

  prepare_generate_variable_list script have now been added (#1164)

• (cdds_prepare): Error

  RuntimeError: requested_variables targeted by rule 0 already active

  or

  RuntimeError: requested_variables targeted by rule 0 already inactive

  no longer occurs. Instead the comment describing the change will be

  added to the log and comments in the and the state will remain

  unchanged (#1210)

• (cdds_prepare): Error

  ExperimentNotFoundError: Experiment name "<experiment identifier>" not found in this version of the Data Request

  no longer occurs when calling the prepare_generate_variable_list

  script when it is looking in the version of the used for model

  configuration. Instead if the is not defined in that version, a

  fallback and the associated with that will be used for comparison

  with the current version of the Data Request (#1256)

• (cdds_prepare): Additional , for which the description in the data

  request has changed between the versions used to configure the model

  and perform the processing, can now be produced (#1018)

• (cdds_prepare): Ocean that use with constraints can now be produced

  (#995)

• (cdds_qc): The --root_config command line option for the

  qc_run_and_report script has now been removed (all arguments

  provided by the CDDS configuration files are now provided via

  command line options, see below; #1167)

• (cdds_qc): The standard_names_dir, controlled_vocabulary_dir,

  --root_proc_dir and root_data_dir command line options for the

  qc_run_and_report script have now been added (#1167)

• (cdds_qc_plugin_cmip6): CDDS QC now confirms that the

  parent_experiment_id attribute in the is consistent with the

  Request and , allowing for the choice of parent_experiment_id

  where there are options within the (#1083)

• (cdds_transfer): The send_to_mass and cdds_transfer_spice

  scripts now raise critical log messages rather than error log

  messages, where appropriate (#1045)

• (extract): CDDS Extract now handles TSSC_SPANS_TOO_MANY_RESOURCES

  errors raised by moo commands that attempt to access files on too

  many tapes (#814)

• (hadsdk): The mip_era and --root_config command line options for

  the write_request_json script have now been removed (all arguments

  provided by the CDDS configuration files are now provided via

  command line options; #1078)

• (hadsdk): The --root_config command line option for the

  check_request_json_against_crem script has now been removed (all

  arguments provided by the CDDS configuration files are now provided

  via command line options; #1078)

• (hadsdk): The value of the parent_time_units attribute can now be

  interpreted by cf_units (#1220)

• (hadsdk): The write_request_json script now writes the

  parent_experiment_id to the request JSON file (#1109)

Release 1.1.4, September 2, 2019

• (cdds_convert): To avoid causing storage issues on SPICE the MIP

  Convert tasks in the CDDS Convert Rose suites now report the local

  disk usage (if suite variable \$STAGING_DIR is set). If this limit

  is specified then tasks that exceed the limit will fail (#1158).

• (cdds_convert): To enable HadGEM3-GC31-MM (N216) processing the

  requests for local temporary storage (\$TMPDIR), in the MIP Convert

  tasks within the Rose suites, have been extended (#1158).

• (extract): When validating model output files, Extract now

  identifies faulty files with no time records (#992).

• (hadsdk):

  Experiment name not found in this version of the data request

  errors in prepare_generate_variable_list, which occurred for some

  in HadGEM3, have been resolved (#1189)

Release 1.1.3, July 31, 2019

• (cdds_qc): Only the directory containing the are searched when

  looking for files to check (temporary files in other directories are

  ignored; (#1046)

• (cdds_qc): The rather than the name used in the filename (the

  out_name) is now used when creating the approved list of (#1052)

• (cdds_transfer): To ensure messages sent to CEDA when withdrawing

  data contain the correct version date stamp, the version date stamp

  used in the path in MASS now doesn\'t change during state changes

  (#919)

• (extract): To avoid issues related to extracting a large number of

  netCDF from MASS, the extraction is now performed in multiple chunks

  (#991)

Release 1.1.2, July 3, 2019

• (cdds_convert): Following updates to the MIP Concatenate sizing file

  to ensure that containing daily output on model levels are now

  correctly sized (#1002)

• (hadsdk): Ocean ancillary variables are now correctly recognised in

  several expressions (#944)

Release 1.1.1, June 27, 2019

• (cdds_prepare): RFMIP is now included in the list of responded to

  by default when constructing (#994)

• (cdds_qc_plugin_cmip6): Fixed a bug in CF Standard Names parser that

  prevented validation of Emon/ta (#993).

• (extract): Faulty are now removed when issues are identified

  (#918).

Release 1.1.0, June 12, 2019

• (cdds_convert): CDDS Convert now preferentially uses cftime,

  falling back to using netcdftime if cftime is not available in

  the environment (#249)

• (cdds_convert): A temporary local storage allocation is now

  requested in SPICE job scripts in order to avoid job failures due to

  lack of disk space (#824) and the settings controlling the temporal

  extent of the output files has been updated to include daily IPCC

  critical variables (#824, #883, #921)

• (cdds_convert): can now be processed from the apm (#922)

• (cdds_prepare): that do not exist in the version of the used to

  configure the are now correctly accounted for when determining

  whether the has changed significantly between the version of the

  used to setup the and the specified version of the (#249)

• (cdds_prepare): The list of that are included by default when

  constructing now includes PAMIP and CDRMIP (plus the correct

  spelling of AerChemMIP) (#832)

• (cdds_prepare): Ocean biogeochemistry field definitions from the

  model suites are now used when identifying that can be produced

  (#820)

• (cdds_qc): The start and end dates in the file name for each are now

  validated against the (#909)

• (cdds_qc_plugin_cmip6): The CMIP6 Compliance Checker Plugin now

  preferentially uses cftime, falling back to using netcdftime if

  cftime is not available in the environment (#249)

• (extract): Daily ocean data can now be extracted from MASS (#882)

• (extract): Extract logs now contain the CRITICAL keyword for

  critical issues (#903)

• (extract): Exceeded step memory limit errors in

  cdds_extract_spice have been resolved (#915)

• (extract): The production of MEDUSA ocean biogeochemistry are now

  supported (#582)

• (hadsdk): Exceeded step memory limit errors in

  cdds_extract_spice have been resolved (#915)

• (hadsdk): The production of MEDUSA ocean biogeochemistry are now

  supported (#582)

• (hadsdk): HadSDK now preferentially uses cftime, falling back to

  using netcdftime if cftime is not available in the environment

  (#249)

Release 1.0.5, May 10, 2019

• (cdds_prepare): All in the can now be deactivated except those

  specified (which is useful for testing purposes) using the new

  prepare_select_variables script (#887)

Release 1.0.4, May 2, 2019

• (cdds_configure): MIP Convert template files for the ocean are now

  produced separately for each sub-stream to avoid an issue where

  ocean files _may be produced with incorrect spatial coordinates

  (#871). This change also improves CDDS Convert end-to-end

  performance by splitting processing into several smaller batch jobs

  that can be run in parallel.

• (cdds_convert): Candidate concatenation file is now written to an

  intermediate directory, so that temporary files do not land up in

  the output directory and then in the publication pipeline (#849)

• (cdds_convert): Fixed a bug where after a timeout, a concatenation

  task would not produce all the expected output (#849)

• (cdds_transfer): Added command line option to move_in_mass script

  to allow state changes, such as embargoed to available or

  available to withdrawn, to be limited to a list of variables

  (#876)

Release 1.0.4, April 30, 2019

• (cdds_qc_plugin_cmip6): The validation of the activity_id global

  attribute in , as written by , now allows for multiple to be

  specified in the CMIP6 for a particular . For example, data produced

  for ssp370 should have an activity_id of

  ScenarioMIP AerChemMIP (#865)

• (extract): Some unit test utility functions were moved from extract

  to hadsdk (#865).

• (hadsdk): Some unit test utility functions were moved from extract

  to hadsdk (#865)

Release 1.0.3, April 18, 2019

• (cdds_convert): Mip_convert and mip_concatenate now write to three

  separate directories: one for mip_convert output, a second for input

  files to the Concatenation Tasks and a third for the final

  Concatenation Tasks which is where qc and transfer will look for the

  output (#840)

• (cdds_convert): Updated the calculation of the concatenation task

  cycle time for suites that only have 1 scheduled concatenation task

  due to the short run time (#852)

• (cdds_transfer): The MOOSE command-id is now reported via

  run_moo_cmd to allow CEDA\'s CREPP tools to kill processes left

  running on the MOOSE controllers that should have ended when the

  command line MOOSE client exited (#855)

Release 1.0.2, April 5, 2019

• (hadsdk): The hadsdk.common.netCDF_regexp() function now returns a

  regular expression that contains 4 match groups, allowing the

  extraction of NEMO (#834)

Release 1.0.1, April 2, 2019

• (cdds_configure): The grid label gn is now used for Datasets that

  use on all T/U/V/UV grids (#761)

• (cdds_convert): The and grid identifiers are now obtained directly

  from the (#555)

• (cdds_convert): One Rose suite per is now submitted rather than one

  Rose suite per request (#710)

• (cdds_convert): The argument provided to the --rose-suite-branch

  parameter can now be a path to a checked out version of the Rose

  suite (for development purposes only; #739)

• (cdds_convert): The Rose suite now removes directories where outputs

  are written prior to producing those outputs (#750)

• (cdds_convert): The Concatenation Periods are now aligned with the

  reference date (#762)

• (cdds_convert): A single is now produced if the Concatenation Period

  is more than the run bounds (#778)

• (cdds_convert): The correct files are now included in each

  Concatenation Period (#780)

• (cdds_convert): The cycles to run MIP Convert are now aligned with

  the reference date (#781)

• (cdds_prepare): The significant changes between the in the version

  of the used to setup HadGEM3 and version 01.00.29 of the were

  approved and added to KNOWN_GOOD_VARIABLES (#747)

• (cdds_qc): The request JSON file is now a positional argument to

  qc_run_and_report (#697)

• (cdds_qc): Single timestep are now handled appropriately (#769)

• (cdds_qc): Bound checks related to single depth levels are now

  ignored by default (the parameter --do_not_filter can be used to

  report these issues; #792)

• (cdds_qc): A detailed report can now be generated using the

  --details parameter (#793)

• (cdds_qc): An approved list of can now be generated, which can be

  used to deactivate in the for the next round of processing (#819)

• (cdds_transfer): The new command line script cdds_transfer_spice

  can now be used to run CDDS Transfer on SPICE (#688)

• (cdds_transfer): The name of the directory used to store on MASS can

  now be specified via the --mass_location parameter (#756)

• (cdds_transfer): The SPICE logs for cdds_transfer_spice are now

  written to the correct location when using the --use_proc_dir

  parameter (#828)

• (extract): A workaround was implemented to deal with the fact that

  the moo ls command sometimes returns a truncated list of filenames

  (#771)

• (hadsdk): The write_request_json script now works correctly for

  experiments without parents, and retrieves the correct value of the

  from CREM (#730)

Release 1.0.0, February 1, 2019

• (cdds_configure): First implementation of CDDS.

• (cdds_convert): First implementation of CDDS.

• (cdds_prepare): First implementation of CDDS.

• (cdds_qc): First implementation of CDDS.

• (cdds_qc_plugin_cf17): First implementation of CDDS.

• (cdds_qc_plugin_cmip6): First implementation of CDDS.

• (cdds_transfer): First implementation of CDDS.

• (extract): First implementation of CDDS.

• (hadsdk): First implementation of CDDS.

• (transfer): First implementation of CDDS.