PermafrostServerDiluvium

wmt-permafrost server on diluvium

Instructions for installing and configuring a WMT server on diluvium for the wmt-permafrost instance.

--Mpiper (talk) 13:59, 27 February 2018 (MST)

Set install directory

The install directory for this server is /data/web/htdocs/wmt/api/permafrost.

install_dir=/data/web/htdocs/wmt/api/permafrost
mkdir -p $install_dir

Install Python

Install a Python distribution to be used locally by WMT. We like to use Miniconda.

cd $install_dir
curl https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -o miniconda.sh
bash ./miniconda.sh -f -b -p $(pwd)/conda
export PATH=$(pwd)/conda/bin:$PATH

If working with an existing Miniconda install, be sure to update everything before continuing:

conda update conda
conda update --all

Install and configure server

Get the server install script and run it.

curl -O https://raw.githubusercontent.com/csdms/wmt/master/scripts/install
python ./install --with-python=$(which python) $(pwd)

This makes a fresh set of database files. Users will have to create a new login. (This is something that needs to be changed. I can't copy over the users.db file from another WMT instance because the client sets up the user tag, used to distinguish public and private models, when the user creates a login.)

Set write permissions on the wsgi script:

chmod a+w $install_dir/opt/wmt-api/wmt/scripts/wmt_wsgi_main.py

Edit conf/wmt.ini to include paths for this server.

[paths]
bin = bin
templates = templates
files = files
uploads = files/uploads
downloads = files/downloads
static = static
logs = logs
database = %(db)s/wmt.db
user_db = %(db)s/users.db
submission_db = %(db)s/submission.db
pickup = /data/ftp/pub/users/wmt
db = db
opt = opt

[url]
scheme = https
netloc = csdms.colorado.edu
path = wmt/api-permafrost

[pickup]
scheme = https
netloc = csdms.colorado.edu
path = pub/users/wmt

[passlib]
schemes = sha512_crypt, sha256_crypt
sha256_crypt__default_rounds = 100000
sha512_crypt__default_rounds = 100000

[user]
name = <your-name>
email = <your-email>

Create files that describe the executor.

cd $install_dir
mkdir -p db/hosts/beach.colorado.edu/db
cd db/hosts/beach.colorado.edu/db
touch env.json info.json

Add info to the files.

# env.json
{
 "PATH": [ "/home/csdms/wmt/permafrost/conda/bin", "/bin", "/usr/bin" ]
}

# info.json
{
 "name": "beach.colorado.edu",
 "username": "",
 "password": "",
 "wmt_prefix": "/home/csdms/wmt/permafrost/conda",
 "launcher": "qsub"
}

I made a similar set of files for siwenna and blanca.

Install software used by the server

Install packages used by the WMT server software.

conda install ipython numpy  # for debugging
conda install passlib pyyaml
pip install wsgilog
pip install requests-toolbelt

Note that WMT server software has already been installed (at the install step above) into opt/wmt-api.

Install the PBS server package. I'm choosing to install with "develop" for this and all Python packages built from source.

cd $install_dir/opt
git clone git@github.com:permamodel/pbs-server.git
cd pbs-server
python setup.py develop

If permissions on the server have been set (see below), root permission will be needed to install in the Python site-packages directory. Use instead

sudo $(which python) setup.py develop

Install component metadata

Clone the `wmt-metadata` repository.

cd $install_dir/opt
git clone git@github.com:csdms/wmt-metadata.git
cd wmt-metadata

Carefully read the README for this repo. There are several environment variables that need to be set.

Ideally, retrieve component metadata from the executor and install it on the server with the install-components.sh script.

bash ./scripts/install_components.sh

To run this script, a login is required on the executor, and sudo privileges may be required on the server. This script needs to be updated. Instead, execute the following commands

export wmt_project=permafrost
export wmt_executor="siwenna.colorado.edu"
export wmt_executor_username=XXX
export wmt_executor_path=/home/csdms/wmt/$wmt_project
export wmt_server_path=/data/web/htdocs/wmt/api/$wmt_project
ssh $wmt_executor_username@$wmt_executor PATH=$wmt_executor_path/conda/bin:\$PATH cmt-config > wmt-config-siwenna.yaml
sudo rm -rf $wmt_server_path/db/components
sudo cp -r metadata/ $wmt_server_path/db/components
sudo chown -R $USER $wmt_server_path/db/components
./scripts/build-metadata ./wmt-config-siwenna.yaml --prefix=$wmt_server_path/db/components

I really need to fix the script.

After installing the components, any that aren't installed on the executor must be deleted. For example, if only FrostNumberModel has been installed on the executor, every other component has to be deleted.

cd $install_dir/db/components
rm -rf everything-except-for-FrostNumberModel
# rm -rf A* B* Ce* Ch* D* E* H* In* Me* Mu* P* R* S* V* W*  # for example

As of the writing of this article, the following components are installed:

• BenchmarkIngestTool
• CMIP
• CRUAKTemp
• FrostNumberGeoModel
• FrostNumberModel
• ILAMB
• KuGeoModel
• KuModel
• ModelIngestTool

Set write permissions on the db/components directory and the parameters file for ILAMB (because of its `refresh` method):

chmod a+w $install_dir/db/components
chmod a+w $install_dir/db/components/ILAMB/db/parameters.json

Activate the server

The WMT server needs almost all of its files to have `nobody` as the group owner. The only exception are pacakges installed into the opt directory, although everything under opt/wmt-api/wmt must also have `nobody` as the group owner, or models won't save correctly.

cd $install_dir
sudo chown -R mapi8461:nobody bin
sudo chown -R mapi8461:nobody conda
sudo chown -R mapi8461:nobody conf
(more)
sudo chown -R mapi8461:nobody opt/wmt-api/wmt

Edit the Apache config file, /usr/local/adm/config/apache/conf/httpd.conf, to add the new server URL.

# permafrost
WSGIDaemonProcess wmt-api-permafrost threads=16 maximum-requests=10000 python-path=/data/web/htdocs/wmt/api/permafrost/conda/lib/python2.7/site-packages
WSGIProcessGroup wmt-api-permafrost
WSGIScriptAlias /wmt/api-permafrost /data/web/htdocs/wmt/api/permafrost/bin/wmt_wsgi_main.py/ process-group=wmt-api-permafrost
Alias /wmt/api-permafrost/static /data/web/htdocs/wmt/api/permafrost/static
<Directory /data/web/htdocs/wmt/api/permafrost/>
 Require all granted
</Directory>

Restart the web server.

sudo /usr/local/httpd/bin/apachectl -k restart

If, at a later point, components are updated, or other changes are made, instead of restarting the web server, just touch the bin/wmt_wsgi_main.py file.

cd $install_dir
touch bin/wmt_wsgi_main.py