AnalystExecutorSiwenna

wmt-analyst executor on siwenna

Instructions for installing and configuring a WMT executor on siwenna.

--Mpiper (talk) 15:40, 8 October 2018 (MDT)

Set install and conda directories

The install directory for this executor is /home/csdms/wmt/analyst.

install_dir=/home/csdms/wmt/analyst
mkdir -p $install_dir

Make sure read and execute bits are set on this directory.

chmod 0775 $install_dir

Install Python

Install a Python distribution to be used locally by WMT. We like to use Miniconda.

cd $install_dir
curl https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -o miniconda.sh
bash ./miniconda.sh -f -b -p $(pwd)/conda
export PATH=$(pwd)/conda/bin:$PATH

If working with an existing Miniconda install, be sure to update everything before continuing:

conda update conda
conda update --all

Install the CSDMS software stack

Clone the `wmt-executor-config-files` repo.

mkdir -p $install_dir/opt && cd $install_dir/opt
git clone https://github.com/mdpiper/wmt-executor-config-files
cd wmt-executor-config-files

Install the CSDMS software stack, including the `babelizer` for components that need to be built from source, in a new environment.

conda install --file wmt-analyst.txt -c csdms-stack -c defaults -c conda-forge

Note that the `netcdf-fortran` package needs to be wound back so that HDF5 1.8.18 is used because of problems with HDF5 1.10.1 and ESMF, used in PyMT (see discussion here). Also note I used a tagged version of PyMT because there have been a lot of changes to it since I built the conda recipes for the permamodel models.

Install executor software

Install the `wmt-exe` package from source.

mkdir -p $install_dir/opt && cd $install_dir/opt
git clone https://github.com/csdms/wmt-exe
cd wmt-exe
python setup.py develop

Create a site configuration file that describes the executor and symlink it to the executor's etc/ diectory.

work_dir="/scratch/$USER/wmt/analyst"
python setup.py configure --wmt-prefix=$install_dir --launch-dir=$work_dir --exec-dir=$work_dir
ln -s "$(realpath wmt.cfg)" $conda_dir/envs/wmt-analyst/etc

Check that `$USER` didn't get expanded in the file. Lines 10-11 should be:

exec_dir = /scratch/$USER/wmt/analyst
launch_dir = /scratch/$USER/wmt/analyst

Note that we're using /scratch for the launch and execution directories instead of the default ~/.wmt.

Install and test CSDMS components

Each section below describes how to install and test a particular CSDMS component.

Currently installed components:

1. BRaKE
2. CHILD
3. CMIP
4. CRUAKTemp
5. FrostNumberGeoModel
6. FrostNumberModel
7. Hydrotrend
8. KuGeoModel
9. KuModel
10. PyDeltaRCM
11. Sedflux3D

Note that when running IPython remotely it's helpful to set

export MPLBACKEND=Agg