HydrologyExecutorBlanca: Difference between revisions

Revision as of 10:26, 1 October 2018

wmt-hydrology executor on blanca

Instructions for installing and configuring a WMT executor on blanca.

--Mpiper (talk) 10:26, 1 October 2018 (MDT)

Set build environment on blanca

Login to summit.

ssh mapi8461@login.rc.colorado.edu

Load the correct slurm for blanca.

module load slurm/blanca

Login to a compute node.

sinteractive

Build everything on the compute node.

Set install and conda directories

The install directory for this executor is /work/csdms/wmt/hydrology.

install_dir=/work/csdms/wmt/hydrology
mkdir -p $install_dir

Make sure read and execute bits are set on this directory.

chmod 0775 $install_dir

An Anaconda distribution is located at /work/csdms/anaconda3. This distribution is used by all WMT executors on blanca. Store this location and add it to your path.

conda_dir=/work/csdms/anaconda3
PATH=$conda_dir/bin:$PATH

WMT executors are installed as environments within this distribution.

Install the CSDMS software stack

Load blanca's `git` module.

module load git

Clone the `wmt-executor-config-files` repo to a common location.

cd $install_dir/..
git clone https://github.com/csdms/wmt-executor-config-files
cd wmt-executor-config-files

Install the CSDMS software stack, including the `babelizer` for components that need to be built from source, in a new environment.

conda env create -f wmt-hydrology.yml

The stack will be installed in $conda_dir/envs/wmt-analyst.

Note that the `netcdf-fortran` package needs to be wound back so that HDF5 1.8.18 is used because of problems with HDF5 1.10.1 and ESMF, used in PyMT (see discussion here). Also note I used an older version of PyMT.

Install executor software

Install the `wmt-exe` package from source.

mkdir -p $install_dir/opt && cd $install_dir/opt
git clone https://github.com/csdms/wmt-exe
cd wmt-exe
python setup.py develop

Create a site configuration file that describes the executor and symlink it to the executor's etc/ directory.

work_dir="/rc_scratch/$USER/wmt/hydrology"
python setup.py configure --wmt-prefix=$install_dir --launch-dir=$work_dir --exec-dir=$work_dir
ln -s "$(realpath wmt.cfg)" $conda_dir/envs/wmt-hydrology/etc

Check that `$USER` didn't get expanded in the file. Lines 10-11 should be:

exec_dir = /rc_scratch/$USER/wmt/hydrology
launch_dir = /rc_scratch/$USER/wmt/hydrology

Note that we're using /rc_scratch for the launch and execution directories instead of the default ~/.wmt. Also note that we needed an SbatchLauncher class for wmt-exe because blanca uses Slurm instead of Torque for job control.

Install and test CSDMS components

Each section below describes how to install and test a particular CSDMS component.

Currently installed components:

BRaKE
CHILD
CMIP
CRUAKTemp
FrostNumberGeoModel
FrostNumberModel
Hydrotrend
KuGeoModel
KuModel
PyDeltaRCM
Sedflux3D

Note that when running IPython remotely it's helpful to set

export MPLBACKEND=Agg

@@ Line 5: / Line 5: @@
 <div>{{TOC limit|3}}</div>
-------
+--[[User:Mpiper|Mpiper]] ([[User talk:Mpiper|talk]]) 10:26, 1 October 2018 (MDT)
 ==Set build environment on blanca==
@@ Line 49: / Line 49: @@
   module load git
-Clone the `wmt-executor-config-files` repo.
+Clone the `wmt-executor-config-files` repo to a common location.
-  mkdir -p $install_dir/opt && cd $install_dir/opt
+  cd $install_dir/..
-  git clone https://github.com/mdpiper/wmt-executor-config-files
+  git clone https://github.com/csdms/wmt-executor-config-files
   cd wmt-executor-config-files
@@ Line 59: / Line 59: @@
 in a new environment.
-  conda env create -f wmt-analyst.yml
+  conda env create -f wmt-hydrology.yml
 The stack will be installed in '''$conda_dir/envs/wmt-analyst'''.
@@ Line 66: / Line 66: @@
 because of problems with HDF5 1.10.1 and ESMF, used in PyMT
 (see discussion [[TestingExecutorBlanca#HDF5_and_file_locks|here]]).
-Also note I used a tagged version of PyMT because there have been a lot of changes to it
+Also note I used an older version of PyMT.
-since I built the conda recipes for the permamodel models.
 ===Install executor software===
@@ Line 78: / Line 77: @@
   python setup.py develop
-Create a site configuration file that describes the executor and symlink it to the executor's '''etc/''' diectory.
+Create a site configuration file that describes the executor and symlink it to the executor's '''etc/''' directory.
-  work_dir="/rc_scratch/$USER/wmt/analyst"
+  work_dir="/rc_scratch/$USER/wmt/hydrology"
   python setup.py configure --wmt-prefix=$install_dir --launch-dir=$work_dir --exec-dir=$work_dir
-  ln -s "$(realpath wmt.cfg)" $conda_dir/envs/wmt-analyst/etc
+  ln -s "$(realpath wmt.cfg)" $conda_dir/envs/wmt-hydrology/etc
 Check that `$USER` didn't get expanded in the file. Lines 10-11 should be:
-  exec_dir = /rc_scratch/$USER/wmt/analyst
+  exec_dir = /rc_scratch/$USER/wmt/hydrology
-  launch_dir = /rc_scratch/$USER/wmt/analyst
+  launch_dir = /rc_scratch/$USER/wmt/hydrology
 Note that we're using '''/rc_scratch''' for the launch and execution directories instead of the default '''~/.wmt'''.
 Also note that we needed an [https://github.com/csdms/wmt-exe/pull/12 SbatchLauncher class]
-for wmt-exe because blanca uses [https://slurm.schedmd.com/overview.html Slurm]
+for wmt-exe because '''''blanca''''' uses [https://slurm.schedmd.com/overview.html Slurm]
 instead of Torque for job control.