TestingExecutorSiwenna

wmt-testing executor on siwenna

Instructions for installing and configuring a WMT executor on siwenna.

--Mpiper (talk) 12:33, 22 February 2018 (MST)

Set install directory

The install directory for this executor is /home/csdms/wmt/_testing.

install_dir=/home/csdms/wmt/_testing
mkdir -p $install_dir

Install Python

Install a Python distribution to be used locally by WMT. We like to use Miniconda.

cd $install_dir
curl https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -o miniconda.sh
bash ./miniconda.sh -f -b -p $(pwd)/conda
export PATH=$(pwd)/conda/bin:$PATH

If working with an existing Miniconda install, be sure to update everything before continuing:

conda update conda
conda update --all

Install the CSDMS software stack

Using the csdms-stack conda channel (the Bakery) install the CSDMS software stack, including several pre-built components, with the `csdms-stack` metapackage.

conda install csdms-stack netcdf-fortran=4.4.4=5 -c csdms-stack -c defaults -c conda-forge

This metapackage currently includes

pymt
cca-tools
csdms-child
csdms-sedflux-3d
csdms-hydrotrend
csdms-permamodel-ku
csdms-permamodel-frostnumber
csdms-permamodel-kugeo
csdms-permamodel-frostnumbergeo
csdms-cruaktemp
csdms-brake
csdms-pydeltarcm

As of 2018-03-20, wind back the `netcdf-fortran` package so that HDF5 1.8.18 is used. There are problems with HDF5 1.10.1 and ESMF, used in PyMT. Hopefully, this will be a temporary change.

Optionally install the `babelizer`, in case a component needs to be built from source.

conda install -c csdms-stack babelizer

Optionally install IPython for testing.

conda install ipython

Note that when running IPython remotely on siwenna, it's helpful to set

export MPLBACKEND=Agg

Install executor software

Install the `wmt-exe` package from source.

mkdir -p $install_dir/opt && cd $install_dir/opt
git clone https://github.com/csdms/wmt-exe
cd wmt-exe
python setup.py develop

Create a site configuration file that describes the executor and symlink it to the executor's etc/ diectory.

python setup.py configure --wmt-prefix=$install_dir --launch-dir='/scratch/$USER/wmt' --exec-dir='/scratch/$USER/wmt'
ln -s "$(realpath wmt.cfg)" $install_dir/conda/etc

Here I've configured the executor to do stuff in the /scratch directory instead of the default ~/.wmt.

Install and test CSDMS components

Each section below describes how to install and test a particular CSDMS component.

Consider setting the PYMT_DEBUG environment variable.

export PYMT_DEBUG=True

It's really helpful for debugging.

Note that the ILAMB component from the Bakery won't work because it's built against `pymt=0.2`; in particular, my BMI for ILAMB is incompatible with the `model_metadata` package in newer versions of PyMT.

Currently installed components:

BenchmarkIngestTool
BRaKE
Child
CMIP
CRUAKTemp
FrostNumberGeoModel
FrostNumberModel
Hydrotrend
ILAMB (will not run!)
KuGeoModel
KuModel
ModelIngestTool
PyDeltaRCM
Sedflux3D

HydroTrend

Hydrotrend is included in the `csdms-stack` metapackage. Test it by starting a Python session in $install_dir/test and executing the setup and IRF methods.

from pymt.components import Hydrotrend

comp = Hydrotrend()
comp.get_component_name()
# args = comp.setup('.')
# comp.initialize(*args)
comp.setup('.')
comp.initialize(None)
comp.get_start_time()
comp.get_end_time()
comp.get_current_time()
comp.update()
comp.finalize()

Permamodel and CRUAKTemp

Permamodel and CRUAKTemp are included in the `csdms-stack` metapackage. Test a component by starting a Python session in $install_dir/test and executing the setup and IRF methods.

from pymt.components import FrostNumberModel

comp = FrostNumberModel()
comp.get_component_name()
# args = comp.setup('.')
# comp.initialize(*args)
comp.setup('.')
comp.initialize('frostnumber_model.cfg')
comp.get_start_time()
comp.get_end_time()
comp.get_current_time()
comp.update()
comp.finalize()

CMIP

Download and install the `cmip` software from source.

cd $install_dir/opt
git clone https://github.com/permamodel/cmip
cd cmip
python setup.py develop

Babelize the CMIP tool.

cd $install_dir
mkdir -p build && cd build
bmi-babelize $install_dir/opt/cmip --prefix=$install_dir/conda

Download the sample config file and test the babelized CMIP component by starting a Python session in $install_dir/test and executing the setup and IRF methods.

from pymt.components import CMIP

comp = CMIP()
comp.get_component_name()
comp.initialize('sample_cmip-siwenna.cfg')
comp.get_start_time()
comp.get_end_time()
comp.get_current_time()
comp.update()
comp.get_current_time()
comp.update_until(comp.get_end_time())
comp.get_current_time()
comp.finalize()

ILAMB

Because of an incompatibility with the most recent version of PyMT, ILAMB will not run in this executor, but I installed it so it will be visible in the client.

conda install csdms-ilamb -c csdms-stack -c defaults -c conda-forge

PBS

Like ILAMB, the PBS components contain parameter types (here, a list) that are incompatible with the latest version of PyMT. I've installed them so that they're visible in the client.

Download and install the `pbs-executor` software from source.

cd $install_dir/opt
git clone https://github.com/permamodel/pbs-executor
cd pbs-executor
python setup.py develop

Babelize the PBS components.

cd $install_dir
mkdir -p build && cd build
bmi-babelize $install_dir/opt/pbs-executor --prefix=$install_dir/conda