Installation

On Azure (Met Office)On Premise (Met Office)On Jasmin

Note

This installation process requires a Conda installation of at least v4.9, earlier versions will raise errors around the setting of environment variables as part of the Conda environment creation commands

• Login to the cdds account (please see Science Shared Accounts for more details): bash xsudo -u cdds bash -l
• Activate Conda environment (to make installation quicker):

  conda activate
  

• Obtain environment file:

  wget https://github.com/MetOffice/CDDS/blob/<tagname>/environment.yml
  

  or download it manually from Github.
• Update locations pointed to within the environment file:

  sed -i "s/<location>/X.Y.Z/" environment.yml
  

• Create environment

  conda env create -f environment.yml -p $HOME/conda_environments/cdds-X.Y.Z
  

  where X.Y.Z is the new version number of CDDS

Note

This has been updated following the roll out of Conda to MO systems. If the -p option is omitted then the installation will end up under $HOME/.conda and will not be visible to other users.

Info

If the wheel installation fails then you can end up with #!python rather than the full paths – this is known to be caused by not having _DEV updated in the packages, possibly due to tagging without pulling the release branch from the repository first

• Activate environment and set CDDS_ENV_COMMAND variable:

  conda activate cdds-X.Y.Z
  conda env config vars set CDDS_ENV_COMMAND="conda activate $HOME/conda_environments/cdds-X.Y.Z"
  

  where X.Y.Z is the new version number of CDDS ```
• Set platform in the CDDS_PLATFORM variable:

  conda activate cdds-X.Y.Z
  conda env config vars set CDDS_PLATFORM=AZURE
  

• Confirm environment variables:

  # get out of cdds environment
  conda deactivate
  # load environment again
  conda activate cdds-X.Y.Z
  # print environment variables
  echo $LC_ALL
  echo $CDDS_ENV_COMMAND
  echo $CDDS_PARTITION
  

  You should see en_GB.UTF-8 for LC_ALL plus the command set above for CDDS_ENV_COMMAND and the CDDS_PARTITION.

Note

This installation process requires a Conda installation of at least v4.9, earlier versions will raise errors around the setting of environment variables as part of the Conda environment creation commands

• Login to the cdds account (please see Science Shared Accounts for more details): bash xsudo -u cdds bash -l
• Activate Conda mamba environment (to make installation quicker):

  . $HOME/software/miniconda3/bin/activate mamba
  

• Obtain environment file:

  wget https://github.com/MetOffice/CDDS/blob/<tagname>/environment.yml
  

  or download it manually from Github.
• Update locations pointed to within the environment file:

  sed -i "s/<location>/X.Y.Z/" environment.yml
  

• Create environment

  mamba env create -f environment.yml -p $HOME/software/miniconda3/envs/cdds-X.Y.Z
  

  where X.Y.Z is the new version number of CDDS

Note

This has been updated following the roll out of Conda to MO systems. If the -p option is omitted then the installation will end up under $HOME/.conda and will not be visible to other users.

Info

If the wheel installation fails then you can end up with #!python rather than the full paths – this is known to be caused by not having _DEV updated in the packages, possibly due to tagging without pulling the release branch from the repository first

• Activate environment and set CDDS_ENV_COMMAND variable:

  conda activate cdds-X.Y.Z
  conda env config vars set CDDS_ENV_COMMAND="source $HOME/software/miniconda3/bin/activate cdds-X.Y.Z"
  

  where X.Y.Z is the new version number of CDDS ```
• Set platform in the CDDS_PLATFORM variable:

  conda activate cdds-X.Y.Z
  conda env config vars set CDDS_PLATFORM=EXETER
  

• Confirm environment variables:

  # get out of cdds environment
  conda deactivate
  # load environment again
  conda activate cdds-X.Y.Z
  # print environment variables
  echo $LC_ALL
  echo $CDDS_ENV_COMMAND
  echo $CDDS_PARTITION
  

  You should see en_GB.UTF-8 for LC_ALL plus the command set above for CDDS_ENV_COMMAND and the CDDS_PARTITION.

Warning

This section will be updated soon

Info

The recommended location for installation is miniconda3 environment under /gws/smf/j04/cmip6_prep/cdds-env-python3/miniconda3

• Login to the sciX JASMIN server.
• Activate conda without loading an environment

  . /gws/smf/j04/cmip6_prep/cdds-env-python3/miniconda3/bin/activate
  

• Obtain environment file

  wget https://github.com/MetOffice/CDDS/blob/<tagname>/environment.yml
  

  or download it manually from Github
• Update locations pointed to within the environment file:

  sed -i "s/<location>/X.Y.Z/" environment.yml
  

• Create environment

  conda env create -f environment.yml -n cdds-X.Y.Z
  

  where X.Y.Z is the new version number of CDDS
• Activate environment and set CDDS_ENV_COMMAND variable

  conda activate cdds-X.Y.Z
  conda env config vars set CDDS_ENV_COMMAND="source /gws/smf/j04/cmip6_prep/cdds-env-python3/miniconda3/bin/activate cdds-X.Y.Z"
  

• Set OS partition at CDDS_PARTITION variable

  conda activate cdds-X.Y.Z
  conda env config vars set CDDS_PARTITION=Jasmin
  

• Install nco library

  conda install -c conda-forge nco
  

• Confirm environment variables:

  # get out of cdds environment
  conda deactivate
  # load environment again
  conda activate cdds-X.Y.Z
  # print environment variables
  echo $LC_ALL
  echo $CDDS_ENV_COMMAND
  echo $CDDS_PARITION
  

  You should see en_GB.UTF-8 for LC_ALL and the command set above for CDDS_ENV_COMMAND and CDDS_PARTITION.

Ensure all the tests pass in the 'real live environment'

• The tests must be executed as the cdds user
• Set the following environment variable, making sure to replace X.Y.Z with the relevant version.

  export SRCDIR=$HOME/conda_environments/cdds-X.Y.Z/lib/python3.10/site-packages
  

• Run the following tests.

  echo "# Executing tests for cdds:"
  pytest -s $SRCDIR/cdds --doctest-modules -m 'not slow and not integration and not rabbitMQ and not data_request'
  pytest -s $SRCDIR/cdds -m slow
  pytest -s $SRCDIR/cdds -m integration
  pytest -s $SRCDIR/cdds -m data_request
  echo "# Executing tests for mip_convert:"
  pytest -s $SRCDIR/mip_convert --doctest-modules -m 'not slow and not mappings and not superslow'
  pytest -s $SRCDIR/mip_convert -m mappings
  pytest -s $SRCDIR/mip_convert -m slow
  

Info

Slow unit tests for transfer and cdds_configure will display error messages to standard output. This is intentional, and does not indicate the tests fail (see transfer.tests.test_command_line.TestMainStore.test_transfer_functional_failing_moo() for details).