Merge, Test & Commit#

The process for committing a ticket follows this sequence with details for each of these steps outlined below.

Important

Before You Start:

Is anyone else committing?
- Trunk Status is used to coordinate trunk commits all projects.
- Simple, not conflicting commits can be done in parallel if reviewers all agree.
- Changes with KGO or Macros usually require sole access to the trunk.
Check how many commits have happened today. Suggested limit per day, per repository is 4.
- More than 4 can be committed if all groups have been run by a team member.

Important

Linked Tickets?

If this is a set of linked tickets then the commit process will need to be followed for each repository in parallel. See Committing Linked Tickets for more details of how this works.

1. Merge#

Check out the trunk, merge the changes from the branch into that trunk copy and resolve any conflicts.

fcm co fcm:um.x_tr chosen_name
cd chosen_name
fcm merge fcm:um.x_br/dev/dev_name/branch_name
fcm cf

JULES

fcm co fcm:jules.x_tr chosen_name
cd chosen_name
fcm merge fcm:jules.x_br/dev/dev_name/branch_name
fcm cf

UKCA

fcm co fcm:ukca.x_tr chosen_name
cd chosen_name
fcm merge fcm:ukca.x_br/dev/dev_name/branch_name
fcm cf

LFRic Apps

fcm co fcm:lfric_apps.x_tr chosen_name
cd chosen_name
fcm merge fcm:lfric_apps.x_br/dev/dev_name/branch_name
fcm cf

LFRic Core

fcm co fcm:lfric.x_tr chosen_name
cd chosen_name
fcm merge fcm:lfric.x_br/dev/dev_name/branch_name
fcm cf

UM docs

fcm co fcm:um_doc.x_tr chosen_name
cd chosen_name
fcm merge fcm:um_doc.x_br/dev/dev_name/branch_name
fcm cf

JULES docs

See JULES documentation changes

Always merge in the developers dev branch, not the test branch.

Only resolve conflicts that appear simple and you are comfortable with. If there are more complicated conflicts ask the developer to create a head of trunk branch to resolve the conflicts themselves and retest the changes.

If there are conflicts in versions.py then see the details in the macro section below.

2. Macros (if required)#

If the ticket includes metadata changes, upgrade macro changes or a new rose-stem app then you will need to upgrade the test-suite.

3. Test (if no KGO)#

The amount of testing to be done at this stage depends on the complexity of the ticket, and what has already been done. A minimum level is required for even trivial tickets to check that the merge has not caused issues, or that there are no clashes with what else has gone on trunk.

Note

Linked tickets will need to be tested together as discussed on the Committing Linked Tickets page.

Run any necessary testing; at the very least run a compile group, generally run developer, and more complex tickets warrant running everything:

rose stem --group=debug_compile
OR rose stem --group=developer,ex1a_developer
OR rose stem --group=all,ex1a

If there is a change to the build configs then you may need to turn off prebuilds. To do so update rose-stem/site/meto/variables.rc such that

{% do SITE_VARS.update({"PREBUILDS" : false}) %}

JULES

The JULES test suite is quick to run, so it’s usual to test all for any ticket. If you have the appropriate environment setup then include the fab group too.

rose stem --group=all,fab

UKCA

The UKCA rose-stem contains minimal tests at the moment, but should be run to confirm the style checker passes.

rose stem --group=all

UKCA testing should also be carried out using the UM rose stem. Check out the UM trunk, and then run

rose stem --group=developer,ukca --source=. --source=/path/to/UKCA/working/copy

LFRic Apps

LFRic Apps rose-stem contains tests spanning all the applications included in the repository. At the very least run the developer group which gives a basic level of tests spanning everything. The full set of tests may be warranted for any application that has had more complex changes.

export CYLC_VERSION=8

rose stem --group=developer
OR e.g. rose stem --group=developer,gungho_model

cylc play <working copy name>

LFRic Core

Run the test suite command from the top level of the repository to run a complete set of the rose-stem developer suites.

export CYLC_VERSION=8

rose stem --group=developer

cylc play <working copy name>

UM docs

Check the documentation builds correctly:

./build_umdoc.py [XXX YYY etc]

where XXX YYY are the details of which docs require building.

JULES docs

JULES documentation is hosted within the JULES GitHub repository. To review and build the documentation branch locally, move to your local clone of the JULES GitHub, then:

git pull
git checkout <branch name>
cd <path_to>/user_guide/doc
conda activate jules-user-guide
make html
firefox build/html/index.html

To build and check the LaTeX PDF:

make latexpdf
evince build/latex/JULES_User_Guide.pdf

4. KGO & Supporting Data (if required)#

If your change is known to alter answers, you need to update rose-stem KGO for all affected tests before you commit to the trunk.

Supporting data is stored in the filesystems of our machines and changes to use will require the reviewer to update those files (BIG DATA).

NB: These instructions are Met Office specific, other sites may manage their KGO differently

UM + LFRic Inputs

KGO files are stored in $UMDIR/standard_jobs/kgo or $UMDIR/standard_jobs/lfricinputs/kgo and are installed there using a script.

Run the rose stem tasks that require a KGO update, plus any other testing required (see above) - if unsure run the all,ex1a.

rose stem --group=all,ex1a --new

You will need access to both your merged working copy and a clone of the SimSys_Scripts github repo (one is available in $UMDIR). Run the script kgo_updates/meto_update_kgo.sh which is located in SimSys_Scripts.
The script will ask you to enter some details regarding the ticket.
- Platforms: enter each platform which has a kgo change, lower case and space seperated, e.g. spice xc40 ex1a
- Path to your merged working copy - the script will check this exists and will fail if it can’t be found.
- KGO directory: this will default to vnXX.X_tYYYY where XX.X is the version number and YYYY is the ticket number.
- There are further prompts to the user through the script - in particular to check the shell script produced.
If running on xc40s the script will ask whether to rsync UM files or lfricinputs files to the XCS. Select the appropriate option.
Check that the new KGO has been installed correctly by restarting your suite, retriggering the failed rose-ana tasks and checking they now pass.

e.g. add –reload or –restart to the rose-stem command ran previously.

Once committed, update the bit comparison table.

JULES

Run the standalone rose-stem with housekeeping switched off to generate new KGO.

rose stem --group=all,ex1a --source=. -S HOUSEKEEPING=false --new

Update KGO_VERSION in rose-stem/include/variables.rc.
Copy the new KGO to the correct locations:

ssh -Y frum@localhost
KGO_VERSION=vnX.X_txxxx
USER_NAME=<user>
SUITE=<suite>

# Copy Linux output to the KGO location for Linux
KGO_DIR=/project/jules/rose-stem/jules-kgo/$KGO_VERSION; mkdir -p $KGO_DIR && cp ~$USER_NAME/cylc-run/$SUITE/work/1/meto_linux_*/output/* $KGO_DIR

# Copy Cray output to the KGO location for the Cray
# If something goes wrong with the copy, try passing the full path (eg. /home/d01/USER/), not just ~$USER_NAME
ssh -Y xcel00
KGO_VERSION=vnX.X_txxxx
USER_NAME=<user>
SUITE=<suite>
KGO_DIR=/projects/jules/rose-stem-kgo/$KGO_VERSION; mkdir -p $KGO_DIR && cp ~$USER_NAME/cylc-run/$SUITE/work/1/meto_xc40_*/output/* $KGO_DIR

# DON'T forget the xcs!!!
rsync -avz $KGO_DIR xcslr0:/projects/jules/rose-stem-kgo/

exit
# check the xcslr0
ssh -Y xcslr0
KGO_VERSION=vnX.X_txxxx
KGO_DIR=/projects/jules/rose-stem-kgo/$KGO_VERSION
ls $KGO_DIR
exit

# Copy EXZ output to the KGO location for EXZ (note <USERNAME> format is firstname.surname!)
# If something goes wrong with the copy, try passing the full path (eg. /home/users/USER/), not just ~$USER_NAME
ssh -Y login.exz
KGO_VERSION=vnX.X_txxxx
USER_NAME=<user>
SUITE=<suite>
KGO_DIR=/common/jules/rose-stem-kgo/$KGO_VERSION; mkdir -p $KGO_DIR && cp ~$USER_NAME/cylc-run/$SUITE/work/1/meto_ex1a_*/output/* $KGO_DIR

# DON'T forget the exa!!!
rsync -avz $KGO_DIR login.exa.sc:/common/internal/jules/rose-stem-kgo/

Rerun the rose-stem tests to make sure nothing is broken.

LFRic Apps

KGO Checksums are stored in the repository alongside the code and can be updated using a script. This can be done by either the code reviewer or by the developer (before submitting their changes for review). In the latter case, the update will need redoing by the reviewer before commit if there are merge conflicts in the checksum files.

Fix any merge conflicts in the checksums - it shouldn’t matter which merge option is selected as you will be overwriting these checksum files again in the following steps.
Run the rose stem tasks that require a KGO update, plus any other testing required (see above) - if unsure run the all group.

export CYLC_VERSION=8
rose stem --group=all
cylc play <suite name>

Ensure the failing KGO’s match those on the branch.
Run the checksum update script stored in <working copy>/rose-stem/bin.

python3 ./rose-stem/bin/update_branch_kgos.py -s <suite name/runX> -w <path to working copy>

Note

This script requires at least python 3.9. This can be achieved on Met Office machines by running module load scitools

Note

The numbered run directory must be included in the suite name, eg. name-of-suite/run1.

5. Verify the checksums updated properly by retriggering the failed checksums. First retrigger export-source, and then when complete export-source_xc40 if new checksums are present there (there is no need to retigger spice). You may need to change the maximum window extent of the gui in order to see the succeeded tasks. Now you can retrigger the failed checksums - these should now pass if the kgo was updated in the working copy correctly.

LFRic Core

KGO Checksums are stored in the repository alongside the code. If there is a merge conflict within these files it is the developers responsibility to update them.

Organise a trunk freeze for LFRic at a time when the developer is available
Developer updates their branch to the head of trunk and regenerates the KGO checksums.
If there were also code conflicts in the science code then the new KGO checksums will need to be signed off by the science reviewer.
Once the ticket is back with you, you can merge the branch to the trunk and run the test-suite as described above to confirm that all is working.

Tip

Between running any required testing and installing the KGO check that the failing rose-ana tasks match those in the developers trac.log. If any have failed for other reasons (e.g. timeout) then these should be re-triggered before attempting to install the KGO files.

4.1 Managing BIG DATA#

Static input data, such as initialisations and ancilliaries, are required by many tests.

LFRic apps

LFRic apps tests use a BIG_DATA_DIR environment variable to provide a platform based path prefix to provide direct access to data required for tests.

The master copy of this is held on XCS at /common/lfric/data/.

This BIG_DATA_DIR is not versioned nor source controlled on any platform. Care is required. The ability to log in as the lfric user is required, e.g. via

sudo -u lfric -i

As reviewer, you should work with the developer, prior to moving to the commit stage, to:

Place new files in the appropriate location on XCS under /common/lfric/data
Run relevant tests on XCS.
Wait for the daily cron job to run to synchronise data between XCS XCE/F & SPICE
Ensure that you are in charge of the in charge of the trunk for the repositories involved.
Update your working copy if other commits have happened.
Rerun relevant tests on XCE/F and SPICE

If the requirement is to update existing files, then further care is required.

Ensure that you are in charge of the in charge of the trunk for the repositories involved.
Retain a temporary copy of the existing files, using a .old suffix.
Place updated files in the appropriate location on XCS under /common/lfric/data
Run all tests on XCS only
- revert changes immediately if there are any issues, and consult with the developer.
Manually trigger the synchronisation script to synchronise data between XCS XCE/F & SPICE
- Waiting for the daily cron job to run can introduce a misalignment or race condition for scheduled testing.
Rerun relevant tests on XCE/F and SPICE
- revert changes immediately if there are any issues, and consult with the developer.
Remove any .old files that you created on XCS.

5. Commit#

Take a final review of the changes about to be applied looking for any obvious merge errors

fcm diff -g

Note

Linked tickets will need to follow the sequence described here.

Commit the change to the trunk

fcm commit

An editor will open requesting a log message which should be in this format:

All others

#ticket_number : Author : Ticket title

where author is the SRS username of the developer - usually the Reported By field on the ticket.

LFRic Core

#<ticket number> for <original author>: <ticket title>

where original author is the dveloper’s proper name.

Note

New!! Remove any blocks: and blockedby: keywords from this ticket and any referenced. Comment on any unblocked tickets to alert the developers.

Update the ticket with the revision number of the commit, e.g. [100000] for revision 100000, comment whether the change is expected to alter results or not and update the ticket status to committed.

Tip

Don’t forget to let the team know you’ve finished with the trunks.

6. Close#

The following day review the nightly test harness results (details on Trunk Status).

If nothing is broken then close the ticket, returning it to the original author.

If something is broken:

Announce to the team and on Trunk Status.
If there is an obvious bug, or a simple fix then update the original branch and re-merge into the trunk.
If there isn’t an easy fix then reverse the change to allow time for investigation.

Tip

Logging in as frum

To access the frum account your ssh key will need to be added to frum authorised keys.
When logged in to your linux desktop run ssh -Y frum@localhost and this will log you in as frum. At this point you will be in UMDIR on the platform SPICE. You can then access frum on other machines via ssh -Y <HOSTNAME>.
Apart from on SPICE the frum home directories and UMDIR are separate. XCE/F share the same UMDIR and the UMDIR on XCS is kept in sync with this one.