Upgrade Macros#

Important

When developing Upgrade Macros, they must be tested using a test branch (see Testing Your Change).

To create an upgrade macro, the developer must edit a versions.py file which is used to update the various apps in the rose stem suite to accept the namelist changes. The upgrade macros also form the basis of the rose app-upgrade script applied by a user wishing to upgrade from one version of a model to the next.

The versions.py file containing upgrade macros can be found in the following locations:

rose-meta/um-atmos/versions.py

rose-meta/jules-standalone/versions.py

applications/<APPLICATION>/rose-meta/lfric-<APPLICATION>/versions.py

Variations on this theme occur, e.g. LFRic Apps science sections or Components in LFRic Core. Metadata directories are also symlinked from a top level rose-meta directory.

Within the file a blank upgrade macro will typically look like this:

class vn130_tXXXX(MacroUpgrade):

    """Upgrade macro for ticket #XXXX by <author>."""

    BEFORE_TAG = "vn13.0"
    AFTER_TAG = "vn13.0_tXXXX"

    def upgrade(self, config, meta_config=None):
        """Upgrade a runtime app configuration."""
        # Input your macro commands here
        return config, self.reports

Note: The BEFORE_TAG should match the AFTER_TAG of the previous macro in the chain. So if this is not the first macro since the release then the BEFORE_TAG will be the version number with an added ticket number as well. For github developments, the ticket number can be either an Issue or PR number. For example:

BEFORE_TAG = "vn13.0_t123"

Example of an upgrade macro#

Developer Sally Smith wishes to add the logical l_bugfix to the &run_bl namelist in the UM. To do this, she replaces the XXXX line in the upgrade macro with her PR or Issue number and adds herself as the author. Within the Python function upgrade, she adds the appropriate command to include the new logical. The macro then looks like this:

class vn130_t1234(MacroUpgrade):

    """Upgrade macro for ticket #1234 by Sally Smith."""

    BEFORE_TAG = "vn13.0"
    AFTER_TAG = "vn13.0_t1234"

    def upgrade(self, config, meta_config=None):
        """Add l_bugfix to namelist run_bl"""
        self.add_setting(config, ["namelist:run_bl", "l_bugfix"], ".true.")
        return config, self.reports

Note that the settings added and their values are Python strings. This command can then be run on a test branch (see Testing Your Change).

Note

Further information about upgrade macros can be found in the Rose user guide. This contains information about more complex changes, such as removing variables from namelists and changing the value that a particular variable takes. A tutorial is also available.

Upgrade Macros in LFRic#

Warning

Namelist files in application example directories are not currently updated by the Apply Macros script. This feature is intended to be introduced, but for now, developers still need to manually update those files.

The organisation of LFRic metadata is different from other repositories (UM + Jules) as the metadata is stored with the Science or Application section it is associated with and is then imported by other apps that require it. This helps modularise the LFRic code but complicates macro chains.

To solve this, macros in LFRic Apps are applied using a wrapper script which will read the added macros and combine them into the versions.py files for the apps where that metadata is imported. Therefore when adding macros, the macro should be added in the versions.py file in the same metadata directory as the metadata change being made. It will then be shared as appropriate by the apply_macros.py script.

Tip

The macro will only end up in versions.py files for metadata that is directly imported by a rose-stem app. Therefore if adding to e.g. Science/gungho, the macro will be deleted from that file by the script. In this case ensure you are ready for the macros to be deleted, e.g. commit all changes.

For example, if a change to metadata is made in science/gungho/rose-meta/lfric-gungho, the macro should be added to the versions.py file in that directory. This will then be copied to other versions.py files that import gungho metadata, e.g. lfric_atm, transport etc.

It is expected that all metadata changes in LFRic Core will require change to the rose-apps in LFRic Apps, but changes to Apps must not affect Core. Therefore, the apply_macros script requires a clone of LFRic Apps to work, but will source it’s own copy of Core if required. If your only changes are to LFRic Core metadata, then you will also require an LFRic Apps branch and pull request.

Important

Some complex macro commands may be dependent on the order in which they are applied. As macros are copied by the wrapper script, the order they are applied will always be determined by the reverse metadata import order. For example, lfric_atm imports gungho metadata, which itself imports components/driver. If all 3 sections have an associated macro, then the macro commands would be applied in the order: components/driver, gungho, lfric_atm.

Tip

The script will read the dependencies.yaml file in your LFRic Apps clone and will checkout a temporary copy of the LFRic Core source if required. Some Core metadata changes will also modify the Core rose apps. In this case make sure to also commit these changes back to the core branch.

To add upgrade macros to LFRic the following steps can be followed:

  1. In your local LFRic Apps clone update the core source in dependencies.yaml if you have LFRic Core changes.

  2. Add your upgrade macros. These must be added to the versions.py file in the same rose-meta directory as the metadata being changed.

  3. Run the Upgrade Macro script in a test branch(see Testing Your Change). This is located in the SimSys_Scripts GitHub repo (at the MetOffice an up to date clone is available in $UMDIR/SimSys_Scripts). The syntax for running is:

    export CYLC_VERSION=8

SimSys_Scripts/lfric_macros/apply_macros.py vnX.Y_tZZZZ \
    [--apps=/path/to/apps] [--core=/path/to/core] [--jules=/path/to/jules]

Important

Test branches must be used for running the Apply Macros script. Do not commit the changes made by apply_macros.py to a Dev Branch

The Apps, Core and Jules options are paths to sources for each of these. Apps will default to the present location (so it is recommended to launch from an Apps clone). Core and Jules will default to reading the dependencies.yaml file in the Apps source if not provided.

The vnXX.Y_tTTTT option must match the After Tag of your upgrade macro. When setting this, the version is the last released version of LFRic Apps. If it’s a linked Apps-Core PR, then set the ticket number based on the Apps Issue or PR. This avoids potential ticket number clashes between the repositories.