PbsExecutorSiwenna

PBS executor on siwenna

Instructions for installing and configuring a WMT executor for the PBS application on siwenna.

--Mpiper (talk) 14:15, 15 March 2019 (MDT)

--Mpiper (talk) 09:27, 30 July 2018 (MDT)

Set install directory

The install directory for this executor is /home/csdms/wmt/pbs.

install_dir=/home/csdms/wmt/pbs
mkdir -p $install_dir

Install Python

Install a Python distribution to be used locally by WMT. We like to use Miniconda.

cd $install_dir
curl https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -o miniconda.sh
bash ./miniconda.sh -f -b -p $(pwd)/conda
export PATH=$(pwd)/conda/bin:$PATH

If working with an existing Miniconda install, be sure to update everything before continuing:

conda update conda
conda update --all

Install the CSDMS software stack

A limiting factor in setting up this executor is the componentized version of ILAMB can only be run with `pymt==0.2`; in particular, the BMI I wrote for ILAMB is incompatible with the `model_metadata` package in newer versions of PyMT.

Start by installing an obsolete dependency from the anaconda channel:

conda install libgfortran-ng=7.3 -c anaconda

To (hopefully) ensure all the packages meet the same dependencies, install the base software for the PBS executor through a conda requirements file.

mkdir -p $install_dir/opt && cd $install_dir/opt
git clone https://github.com/csdms/wmt-executor-config-files
cd wmt-executor-config-files
nice conda install --file pbs.txt -c csdms-stack -c defaults -c conda-forge -c anaconda

(It might take awhile for conda to harmonize the dependencies.)

When running/testing interactively on siwenna, activate the base conda environment

source activate base

and set

export MPLBACKEND=Agg

Install executor software

Install the `wmt-exe` package from source.

cd $install_dir/opt
git clone https://github.com/csdms/wmt-exe
cd wmt-exe
git checkout mdpiper/activate-conda-env
python setup.py develop

Note that I'm installing from a branch that I haven't merged into the master branch.

Create a site configuration file that describes the executor and symlink it to the executor's etc/ diectory.

scratch_dir='/scratch/$USER/wmt/pbs'
python setup.py configure --wmt-prefix=$install_dir --launch-dir=$scratch_dir --exec-dir=$scratch_dir
ln -s "$(realpath wmt.cfg)" $install_dir/conda/etc

Here I've configured the executor to do stuff in the /scratch directory instead of the default ~/.wmt.

Install and test CSDMS components

Each section below describes how to install and test a particular CSDMS component.

Currently installed components:

1. BenchmarkIngestTool
2. ILAMB
3. ModelIngestTool

ILAMB

We installed ILAMB above. Test it by starting a Python session in $install_dir/test and executing the setup and IRF methods.

from pymt.components import ILAMB

comp = ILAMB()
comp.get_component_name()
comp.setup('.')

At this point, stop and edit bmi_ilamb.yaml, updating the `ilamb_root` attribute with the appropriate location on siwenna, /home/csdms/ilamb. All other attributes can be left unchanged with their default values.

comp.initialize('bmi_ilamb.yaml')
comp.get_start_time()
comp.get_end_time()
comp.get_current_time()
comp.update()
comp.finalize()

ILAMB is run at the call to `update`. The default run takes about 75 min.

PBS

Download and install the `pbs-executor` software from source.

cd $install_dir/opt
git clone https://github.com/permamodel/pbs-executor
cd pbs-executor
python setup.py install

Babelize the PBS components.

cd $install_dir
mkdir -p build && cd build
bmi-babelize $install_dir/opt/pbs-executor --prefix=$install_dir/conda

The first attempt to babelize fails because a file, libltdl.la, is missing. I copied it from another install, modified the path at the end, then re-babelized (be sure to remove existing babelization).

Another set of files is missing. Try again.

Test a component (here, ModelIngestTool) by starting a Python session in $install_dir/test and executing the setup and IRF methods.

from pymt.components import ModelIngestTool

comp = ModelIngestTool()
comp.get_component_name()
comp.setup('.')

At this point, suspend the IPython session and copy a sample config file and data from the `pbs-executor` repo. (This could be improved.)

cd $install_dir/opt/pbs-executor/pbs_executor/data
cp ingest.yaml sftlf_fx_PBS-test_historical_r9i0p0.nc $install_dir/test
cd -

Restore the IPython session and continue.

comp.initialize('ingest.yaml')
comp.update()
comp.finalize()

View the index.html file that results.