Instructions for installing and configuring a WMT executor on blanca.
Set build environment on blanca
Login to summit.
Load the correct slurm for blanca.
module load slurm/blanca
Login to a compute node.
Build everything on the compute node.
Set install and conda directories
The install directory for this executor is /work/csdms/wmt/analyst.
install_dir=/work/csdms/wmt/analyst mkdir -p $install_dir
Make sure read and execute bits are set on this directory.
chmod 0775 $install_dir
An Anaconda distribution is located at /work/csdms/anaconda3. This distribution is used by all WMT executors on blanca. Store this location and add it to your path.
WMT executors are installed as environments within this distribution.
Install the CSDMS software stack
Load blanca's `git` module.
module load git
Clone the `wmt-executor-config-files` repo.
mkdir -p $install_dir/opt && cd $install_dir/opt git clone https://github.com/mdpiper/wmt-executor-config-files cd wmt-executor-config-files
Install the CSDMS software stack, including the `babelizer` for components that need to be built from source, in a new environment.
conda env create -f wmt-analyst.yml
The stack will be installed in $conda_dir/envs/wmt-analyst.
Note that the `netcdf-fortran` package needs to be wound back so that HDF5 1.8.18 is used because of problems with HDF5 1.10.1 and ESMF, used in PyMT (see discussion here). Also note I used a tagged version of PyMT because there have been a lot of changes to it since I built the conda recipes for the permamodel models.
Install executor software
Install the `wmt-exe` package from source.
mkdir -p $install_dir/opt && cd $install_dir/opt git clone https://github.com/csdms/wmt-exe cd wmt-exe python setup.py develop
Create a site configuration file that describes the executor and symlink it to the executor's etc/ diectory.
work_dir='/rc_scratch/$USER/wmt/analyst' # note single quotes python setup.py configure --wmt-prefix=$install_dir --launch-dir=$work_dir --exec-dir=$work_dir ln -s "$(realpath wmt.cfg)" $conda_dir/envs/wmt-analyst/etc
Check that `$USER` didn't get expanded in the file. Lines 10-11 should be:
exec_dir = /rc_scratch/$USER/wmt/analyst launch_dir = /rc_scratch/$USER/wmt/analyst
Note that we're using /rc_scratch for the launch and execution directories instead of the default ~/.wmt. Also note that we needed an SbatchLauncher class for wmt-exe because blanca uses Slurm instead of Torque for job control.
Install and test CSDMS components
Each section below describes how to install and test a particular CSDMS component.
Currently installed components:
Note that when running IPython remotely it's helpful to set
I deleted the wmt-analyst environment and reinstalled it with Hydrotrend as the only component. It works. (It did not work when I also installed the permamodel components.)