HydrologyExecutorBlanca
Instructions for installing and configuring a WMT executor on blanca.
--Mpiper (talk) 10:26, 1 October 2018 (MDT)
Set build environment on blanca
Login to summit.
ssh mapi8461@login.rc.colorado.edu
Load the correct slurm for blanca.
module load slurm/blanca
Login to a compute node.
sinteractive
Build everything on the compute node.
Set install and conda directories
The install directory for this executor is /work/csdms/wmt/hydrology.
install_dir=/work/csdms/wmt/hydrology mkdir -p $install_dir
Make sure read and execute bits are set on this directory.
chmod 0775 $install_dir
An Anaconda distribution is located at /work/csdms/anaconda3. This distribution is used by all WMT executors on blanca. Store this location and add it to your path.
conda_dir=/work/csdms/anaconda3 PATH=$conda_dir/bin:$PATH
WMT executors are installed as environments within this distribution.
Install the CSDMS software stack
Load blanca's `git` module.
module load git
Clone the `wmt-executor-config-files` repo to a common location.
cd $install_dir/.. git clone https://github.com/csdms/wmt-executor-config-files cd wmt-executor-config-files
Install the base of the CSDMS software stack in a new environment.
conda env create -f wmt-hydrology.yml
The stack will be installed in $conda_dir/envs/wmt-hydrology.
Note that there's no PyMT: this executor predates it. There are install instructions below for the `coupling` environment.
Install executor software
Install the `wmt-exe` package from source.
mkdir -p $install_dir/opt && cd $install_dir/opt git clone https://github.com/csdms/wmt-exe cd wmt-exe python setup.py develop
Create a site configuration file that describes the executor and symlink it to the executor's etc/ directory.
work_dir="/rc_scratch/$USER/wmt/hydrology" python setup.py configure --wmt-prefix=$install_dir --launch-dir=$work_dir --exec-dir=$work_dir ln -s "$(realpath wmt.cfg)" $conda_dir/envs/wmt-hydrology/etc
Check that `$USER` didn't get expanded in the file. Lines 10-11 should be:
exec_dir = /rc_scratch/$USER/wmt/hydrology launch_dir = /rc_scratch/$USER/wmt/hydrology
Note that we're using /rc_scratch for the launch and execution directories instead of the default ~/.wmt. Also note that we needed an SbatchLauncher class for wmt-exe because blanca uses Slurm instead of Torque for job control.
Install Topoflow
The following directions rely on ancient technology, but they work!
Start by installing the coupling framework and the componentized topoflow package.
conda install coupling --override-channels -c csdms/channel/dev -c defaults conda install csdms-topoflow --override-channels -c csdms/channel/dev -c csdms -c defaults
Next, some trickery. Uninstall `topoflow`, which also uninstalls `topoflow-bridge` and `csdms-topoflow`, but doesn't uninstall the deps of `csdms-topoflow`, like `babel`. Reinstall only `csdms-topoflow`, which is still needed.
conda uninstall topoflow conda install csdms-topoflow --no-deps --override-channels -c csdms/channel/dev -c csdms -c defaults
Also uninstall the coupling framework.
conda uninstall coupling
Then get the development version of the coupling framework and install it.
mkdir -p $install_dir/opt && cd $install_dir/opt git clone https://github.com/csdms/coupling cd coupling/ git checkout mcflugen/fix-for-new-bmi python setup.py develop
Get the development versions of `topoflow` and `topoflow-bridge` and install them.
cd .. git clone https://github.com/mdpiper/topoflow cd topoflow python setup.py develop
cd .. git clone https://github.com/csdms/topoflow-bridge cd topoflow-bridge python setup.py develop
Manually copy files from the development version of `topoflow-bridge` into the executor's version.
src=$install_dir/opt/topoflow-bridge/.bmi dst=$conda_dir/envs/wmt-hydrology/share/csdms cp $src/channels_diffusive_wave/* $dst/ChannelsDiffusiveWave cp $src/channels_dynamic_wave/* $dst/ChannelsDynamicWave ...
That's it!