PermafrostServerDiluvium

wmt-permafrost server on diluvium

Instructions for installing and configuring a WMT server on diluvium for the wmt-permafrost instance.

--Mpiper (talk) 13:59, 27 February 2018 (MST)

Set install directory

The install directory for this server is /data/web/htdocs/wmt/api/permafrost.

install_dir=/data/web/htdocs/wmt/api/permafrost
mkdir -p $install_dir

Install Python

Install a Python distribution to be used locally by WMT. We like to use Miniconda.

cd $install_dir
curl https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -o miniconda.sh
bash ./miniconda.sh -f -b -p $(pwd)/conda
export PATH=$(pwd)/conda/bin:$PATH

If working with an existing Miniconda install, be sure to update everything before continuing:

conda update conda
conda update --all

Install and configure server

Get the server install script and run it.

curl -O https://raw.githubusercontent.com/csdms/wmt/master/scripts/install
python ./install --with-python=$(which python) $(pwd)

This makes a fresh set of database files. Users will have to create a new login. (This is something that needs to be changed. I can't copy over the users.db file from another WMT instance because the client sets up the user tag, used to distinguish public and private models, when the user creates a login.)

Set write permissions on the wsgi script:

chmod a+w $install_dir/opt/wmt-api/wmt/scripts/wmt_wsgi_main.py

Edit conf/wmt.ini to include paths for this server.

[paths]
bin = bin
templates = templates
files = files
uploads = files/uploads
downloads = files/downloads
static = static
logs = logs
database = %(db)s/wmt.db
user_db = %(db)s/users.db
submission_db = %(db)s/submission.db
pickup = /data/ftp/pub/users/wmt
db = db
opt = opt

[url]
scheme = https
netloc = csdms.colorado.edu
path = wmt/api-permafrost

[pickup]
scheme = https
netloc = csdms.colorado.edu
path = pub/users/wmt

[passlib]
schemes = sha512_crypt, sha256_crypt
sha256_crypt__default_rounds = 100000
sha512_crypt__default_rounds = 100000

[user]
name = <your-name>
email = <your-email>

Create files that describe the executor.

cd $install_dir
mkdir -p db/hosts/beach.colorado.edu/db
cd db/hosts/beach.colorado.edu/db
touch env.json info.json

Add info to the files.

# env.json
{
 "PATH": [ "/home/csdms/wmt/permafrost/conda/bin", "/bin", "/usr/bin" ]
}

# info.json
{
 "name": "beach.colorado.edu",
 "username": "",
 "password": "",
 "wmt_prefix": "/home/csdms/wmt/permafrost/conda",
 "launcher": "qsub"
}

I made a similar set of files for siwenna and blanca.

Install software used by the server

Install packages used by the WMT server software.

conda install ipython numpy  # for debugging
conda install passlib pyyaml
pip install wsgilog
pip install requests-toolbelt

Note that WMT server software has already been installed (at the install step above) into opt/wmt-api.

Install the PBS server package. I'm choosing to install with "develop" for this and all Python packages built from source.

cd $install_dir/opt
git clone git@github.com:permamodel/pbs-server.git
cd pbs-server
python setup.py develop

If permissions on the server have been set (see below), root permission will be needed to install in the Python site-packages directory. Use instead

sudo $(which python) setup.py develop

Install component metadata

Clone the `wmt-metadata` repository.

cd $install_dir/opt
git clone git@github.com:csdms/wmt-metadata.git
cd wmt-metadata

Carefully read the README for this repo. There are several environment variables that need to be set.

export wmt_project=permafrost
export wmt_executor="siwenna.colorado.edu"
export wmt_executor_username=wmtuser
export wmt_executor_path=/home/csdms/wmt/$wmt_project
export wmt_server_path=/data/web/htdocs/wmt/api/$wmt_project

Retrieve component metadata from the executor and install it on the server with the install-components.sh script.

bash ./scripts/install_components.sh

To run this script, a login is required on the executor, and sudo privileges may be required on the server.

As of the writing of this article, the following components are installed:

• CMIP
• CRUAKTemp
• FrostNumberGeoModel
• FrostNumberModel
• KuGeoModel
• KuModel

Activate the server

The WMT server needs almost all of its files to have `nobody:nobody` as the owner. The only exceptions are

• packages installed into the opt directory, although everything under opt/wmt-api/wmt must also have `nobody` as the group owner, or models won't save correctly;
• the db/components directory and everything underneath it may have `$USER:nobody` as owner.

cd $install_dir
sudo chown -R nobody:nobody bin
sudo chown -R nobody:nobody conda
sudo chown -R nobody:nobody conf
(more)
sudo chown -R nobody:nobody opt/wmt-api/wmt
sudo chown -R mapi8461:nobody db/components

Edit the Apache config file, /usr/local/adm/config/apache/conf/httpd.conf, to add the new server URL.

# permafrost
WSGIDaemonProcess wmt-api-permafrost threads=16 maximum-requests=10000 python-path=/data/web/htdocs/wmt/api/permafrost/conda/lib/python2.7/site-packages
WSGIProcessGroup wmt-api-permafrost
WSGIScriptAlias /wmt/api-permafrost /data/web/htdocs/wmt/api/permafrost/bin/wmt_wsgi_main.py/ process-group=wmt-api-permafrost
Alias /wmt/api-permafrost/static /data/web/htdocs/wmt/api/permafrost/static
<Directory /data/web/htdocs/wmt/api/permafrost/>
 Require all granted
</Directory>

Restart the web server.

sudo /usr/local/httpd/bin/apachectl -k restart

If, at a later point, components are updated, or other changes are made, instead of restarting the web server, just touch the bin/wmt_wsgi_main.py file.

cd $install_dir
touch bin/wmt_wsgi_main.py