PermafrostExecutorBlanca

wmt-permafrost executor on blanca

Instructions for installing and configuring a WMT executor on blanca.

--Mpiper (talk) 16:48, 19 March 2018 (MDT)

Set install directory

The install directory for this executor is /projects/mapi8461/wmt/permafrost.

install_dir=/projects/mapi8461/wmt/permafrost
mkdir -p $install_dir

Make sure my projects directory has read and execute bits set.

chmod 0775 /projects/mapi8461

Install Python

Install a Python distribution to be used locally by WMT. We like to use Miniconda.

cd $install_dir
curl https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -o miniconda.sh
bash ./miniconda.sh -f -b -p $(pwd)/conda
export PATH=$(pwd)/conda/bin:$PATH

If working with an existing Miniconda install, be sure to update everything before continuing:

conda update conda
conda update --all

Install the CSDMS software stack

A limiting factor in setting up this executor is the componentized version of ILAMB can only be run with `pymt=0.2`; in particular, the BMI I wrote for ILAMB is incompatible with the `model_metadata` package in newer versions of PyMT. Further, as of 2018-03-20, the `netcdf-fortran` package needs to be wound back so that HDF5 1.8.18 is used because of problems with HDF5 1.10.1 and ESMF, used in PyMT. To ensure all the packages meet the same dependencies (looking at you, numpy), simultaneously install `cca-tools`, `netcdf-fortran`, `pymt=0.2` and `csdms-ilamb` using the csdms-stack conda channel (the Bakery),

conda install cca-tools netcdf-fortran=4.4.4=5 pymt=0.2 csdms-ilamb -c csdms-stack -c defaults -c conda-forge

Install the `babelizer` for components that need to be built from source.

conda install -c csdms-stack babelizer

Optionally install IPython for testing.

conda install ipython

Note that when running IPython remotely on siwenna, it's helpful to set

export MPLBACKEND=Agg

Install executor software

Load blanca's `git` module.

module load git

Install the `wmt-exe` package from source.

mkdir -p $install_dir/opt && cd $install_dir/opt
git clone https://github.com/csdms/wmt-exe
cd wmt-exe
python setup.py develop

Create a site configuration file that describes the executor and symlink it to the executor's etc/ diectory.

python setup.py configure --wmt-prefix=$install_dir --launch-dir='/rc_scratch/$USER/wmt/permafrost' --exec-dir='/rc_scratch/$USER/wmt/permafrost'
#ln -s "$(realpath wmt.cfg)" $install_dir/conda/etc # "realpath" not installed on blanca :(
cd $install_dir/conda/etc
ln -s $install_dir/opt/wmt-exe/wmt.cfg

Note that we're using rc_scratch for the launch and execution directories instead of the default ~/.wmt. Also note that we needed an SbatchLauncher class for wmt-exe because blanca uses Slurm instead of Torque for job control.

Install and test CSDMS components

Each section below describes how to install and test a particular CSDMS component.

Currently installed components:

1. BenchmarkIngestTool
2. CMIP
3. CRUAKTemp
4. FrostNumberGeoModel
5. FrostNumberModel
6. ILAMB
7. KuGeoModel
8. KuModel
9. ModelIngestTool

ILAMB

We installed ILAMB above. Test it by starting a Python session in $install_dir/test in an interactive Slurm session and executing the setup and IRF methods.

from pymt.components import ILAMB

comp = ILAMB()
comp.get_component_name()
comp.setup('.')

At this point, stop and edit bmi_ilamb.yaml, updating the `ilamb_root` attribute with the appropriate location on blanca, which is temporarily /projects/mapi8461/ilamb. All other attributes can be left unchanged with their default values.

comp.initialize('bmi_ilamb.yaml')
comp.get_start_time()
comp.get_end_time()
comp.get_current_time()
comp.update()
comp.finalize()

ILAMB is run at the call to `update`. The default run takes about 75 min.

PBS

Download and install the `pbs-executor` software from source.

cd $install_dir/opt
git clone git@github.com:permamodel/pbs-executor.git
cd pbs-executor
python setup.py develop

Babelize the PBS components.

cd $install_dir
mkdir -p build && cd build
bmi-babelize $install_dir/opt/pbs-executor --prefix=$install_dir/conda

Test a component (here, ModelIngestTool) by starting a Python session in $install_dir/test in an interactive Slurm session and executing the setup and IRF methods.

from pymt.components import ModelIngestTool

comp = ModelIngestTool()
comp.get_component_name()
comp.setup('.')

At this point, suspend the IPython session and copy a sample config file and data from the `pbs-executor` repo. (This could be improved.)

cd $install_dir/opt/pbs-executor/pbs_executor/data
cp ingest.yaml sftlf_fx_PBS-test_historical_r9i0p0.nc $install_dir/test
cd -

Next, edit the ingest.yaml file to update the `ilamb_root` attribute to /projects/mapi8461/ilamb.

Restore the IPython session and continue.

comp.initialize('ingest.yaml')
comp.update()
comp.finalize()

View the index.html file that results.

Permamodel

Install the permamodel components from the Bakery using the `csdms-permamodel` metapackage.

conda install csdms-permamodel -c csdms-stack

This metapackage contains

• csdms-permamodel-ku
• csdms-permamodel-frostnumber
• csdms-permamodel-kugeo
• csdms-permamodel-frostnumbergeo
• csdms-cruaktemp

Test a component (here, FnM) by starting a Python session in $install_dir/test in an interactive Slurm session and executing the setup and IRF methods.

from pymt.components import FrostNumberModel

comp = FrostNumberModel()
comp.get_component_name()
# args = comp.setup('.')
# comp.initialize(*args)
comp.setup('.')
comp.initialize('frostnumber_model.cfg')
comp.get_start_time()
comp.get_end_time()
comp.get_current_time()
comp.update()
comp.finalize()

Check the ouput of the run

$ cat frostnumber_output.dat
Year: 2000  output=('0.396', '-1.000', '-1.000')

CMIP

Install the CMIP component through its conda recipe.

conda install csdms-cmip -c csdms-stack -c defaults -c conda-forge

Download the sample config file and test the CMIP component by starting a Python session in $install_dir/test and executing the setup and IRF methods.

from pymt.components import CMIP

comp = CMIP()
comp.get_component_name()
comp.initialize('sample_cmip-blanca.cfg')
comp.get_start_time()
comp.get_end_time()
comp.get_current_time()
comp.update()
comp.get_current_time()
comp.update_until(comp.get_end_time())
comp.get_current_time()
comp.finalize()