HydrologyExecutorBlanca

wmt-hydrology executor on blanca

Instructions for installing and configuring a WMT executor on blanca.

--Mpiper (talk) 10:26, 1 October 2018 (MDT)

Set build environment on blanca

Login to summit.

ssh mapi8461@login.rc.colorado.edu

Load the correct slurm for blanca.

module load slurm/blanca

Login to a compute node.

sinteractive

Build everything on the compute node.

Set install and conda directories

The install directory for this executor is /work/csdms/wmt/hydrology.

install_dir=/work/csdms/wmt/hydrology
mkdir -p $install_dir

Make sure read and execute bits are set on this directory.

chmod 0775 $install_dir

An Anaconda distribution is located at /work/csdms/anaconda3. This distribution is used by all WMT executors on blanca. Store this location and add it to your path.

conda_dir=/work/csdms/anaconda3
PATH=$conda_dir/bin:$PATH

WMT executors are installed as environments within this distribution.

Install the CSDMS software stack

Load blanca's `git` module.

module load git

Clone the `wmt-executor-config-files` repo to a common location.

cd $install_dir/..
git clone https://github.com/csdms/wmt-executor-config-files
cd wmt-executor-config-files

Install the base of the CSDMS software stack in a new environment.

conda env create -f wmt-hydrology.yml

The stack will be installed in $conda_dir/envs/wmt-hydrology.

Note that there's no PyMT: this executor predates it. There are install instructions below for the `coupling` environment.

Install executor software

Install the `wmt-exe` package from source.

mkdir -p $install_dir/opt && cd $install_dir/opt
git clone https://github.com/csdms/wmt-exe
cd wmt-exe
python setup.py develop

Create a site configuration file that describes the executor and symlink it to the executor's etc/ directory.

work_dir="/rc_scratch/$USER/wmt/hydrology"
python setup.py configure --wmt-prefix=$install_dir --launch-dir=$work_dir --exec-dir=$work_dir
ln -s "$(realpath wmt.cfg)" $conda_dir/envs/wmt-hydrology/etc

Check that `$USER` didn't get expanded in the file. Lines 10-11 should be:

exec_dir = /rc_scratch/$USER/wmt/hydrology
launch_dir = /rc_scratch/$USER/wmt/hydrology

Note that we're using /rc_scratch for the launch and execution directories instead of the default ~/.wmt. Also note that we needed an SbatchLauncher class for wmt-exe because blanca uses Slurm instead of Torque for job control.

Install Topoflow

The following directions rely on ancient technology, but they work!

Start by installing the coupling framework and the componentized topoflow package.

conda install coupling --override-channels -c csdms/channel/dev -c defaults
conda install csdms-topoflow --override-channels -c csdms/channel/dev -c csdms -c defaults

Next, some trickery. Uninstall `topoflow`, which also uninstalls `topoflow-bridge` and `csdms-topoflow`, but doesn't uninstall the deps of `csdms-topoflow`, like `babel`. Reinstall only `csdms-topoflow`, which is still needed.

conda uninstall topoflow
conda install csdms-topoflow --no-deps --override-channels -c csdms/channel/dev -c csdms -c defaults

Also uninstall the coupling framework.

conda uninstall coupling

Then get the development version of the coupling framework and install it.

mkdir -p $install_dir/opt && cd $install_dir/opt
git clone https://github.com/csdms/coupling
cd coupling/
git checkout mcflugen/fix-for-new-bmi 
python setup.py develop

Get the development versions of `topoflow` and `topoflow-bridge` and install them.

cd ..
git clone https://github.com/mdpiper/topoflow
cd topoflow
python setup.py develop

cd ..
git clone https://github.com/csdms/topoflow-bridge
cd topoflow-bridge
python setup.py develop

Manually copy files from the development version of `topoflow-bridge` into the executor's version.

src=$install_dir/opt/topoflow-bridge/.bmi
dst=$conda_dir/envs/wmt-hydrology/share/csdms
cp $src/channels_diffusive_wave/* $dst/ChannelsDiffusiveWave
cp $src/channels_dynamic_wave/* $dst/ChannelsDynamicWave
...

That's it!