PbsServerDiluvium
Instructions for installing and configuring a WMT server on diluvium for the PBS project.
--Mpiper (talk) 11:16, 30 July 2018 (MDT)
Set install directory
The install directory for this server is /data/web/htdocs/wmt/api/pbs.
install_dir=/data/web/htdocs/wmt/api/pbs mkdir -p $install_dir
Install Python
Install a Python distribution to be used locally by WMT. We like to use Miniconda.
cd $install_dir curl https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -o miniconda.sh bash ./miniconda.sh -f -b -p $(pwd)/conda export PATH=$(pwd)/conda/bin:$PATH
If working with an existing Miniconda install, be sure to update everything before continuing:
conda update conda conda update --all
Install and configure server
Get the server install script and run it.
curl -O https://raw.githubusercontent.com/csdms/wmt/master/scripts/install python ./install --with-python=$(which python) $(pwd)
This makes a fresh set of database files. Users will have to create a new login. (This is something that needs to be changed. I can't copy over the users.db file from another WMT instance because the client sets up the user tag, used to distinguish public and private models, when the user creates a login.)
Set write permissions on the wsgi script:
chmod a+w $install_dir/opt/wmt-api/wmt/scripts/wmt_wsgi_main.py
Edit conf/wmt.ini to include paths for this server.
[paths] bin = bin templates = templates files = files uploads = files/uploads downloads = files/downloads static = static logs = logs database = %(db)s/wmt.db user_db = %(db)s/users.db submission_db = %(db)s/submission.db pickup = /data/ftp/pub/users/wmt db = db opt = opt [url] scheme = https netloc = csdms.colorado.edu path = wmt/api-permafrost [pickup] scheme = https netloc = csdms.colorado.edu path = pub/users/wmt [passlib] schemes = sha512_crypt, sha256_crypt sha256_crypt__default_rounds = 100000 sha512_crypt__default_rounds = 100000 [user] name = <your-name> email = <your-email>
Create files that describe the executor.
cd $install_dir mkdir -p db/hosts/beach.colorado.edu/db cd db/hosts/beach.colorado.edu/db touch env.json info.json
Add info to the files.
# env.json
{
"PATH": [ "/home/csdms/wmt/permafrost/conda/bin", "/bin", "/usr/bin" ]
}
# info.json
{
"name": "beach.colorado.edu",
"username": "",
"password": "",
"wmt_prefix": "/home/csdms/wmt/permafrost/conda",
"launcher": "qsub"
}
I made a similar set of files for siwenna and blanca.
Install software used by the server
Install packages used by the WMT server software.
conda install ipython numpy # for debugging conda install passlib paramiko pyyaml pip install wsgilog pip install requests-toolbelt
Note that WMT server software has already been installed (at the install step above) into opt/wmt-api.
Install the PBS server package.
cd $install_dir/opt git clone https://github.com/permamodel/pbs-server.git cd pbs-server python setup.py install
If permissions on the server have been set (see below), root permission will be needed to install in the Python site-packages directory. Use instead
sudo $(which python) setup.py install
Install component metadata
Clone the `wmt-metadata` repository.
cd $install_dir/opt git clone https://github.com/csdms/wmt-metadata.git cd wmt-metadata
Carefully read the README for this repo. There are several environment variables that need to be set.
export wmt_project=pbs export wmt_executor="siwenna.colorado.edu" export wmt_executor_username=wmtuser export wmt_executor_path=/home/csdms/wmt/$wmt_project export wmt_server_path=/data/web/htdocs/wmt/api/$wmt_project
Retrieve component metadata from the executor and install it on the server with the install-components.sh script.
bash ./scripts/install-components.sh
To run this script, a login is required on the executor, and sudo privileges may be required on the server.
As of the writing of this article, the following components are installed:
- BenchmarkIngestTool
- ILAMB
- ModelIngestTool
Set write permissions on the db/components directory and the parameters file for ILAMB (because of its `refresh` method):
chmod a+w $install_dir/db/components chmod a+w $install_dir/db/components/ILAMB/db/parameters.json
Activate the server
The WMT server needs almost all of its files to have `nobody:nobody` as the owner. The only exceptions are
- packages installed into the opt directory, although everything under opt/wmt-api/wmt must also have `nobody` as the group owner, or models won't save correctly;
- the db/components directory and everything underneath it may have `$USER:nobody` as owner.
cd $install_dir sudo chown -R nobody:nobody bin sudo chown -R nobody:nobody conda sudo chown -R nobody:nobody conf (more) sudo chown -R nobody:nobody opt/wmt-api/wmt sudo chown -R mapi8461:nobody db/components
Edit the Apache config file, /usr/local/adm/config/apache/conf/httpd.conf, to add the new server URL.
# PBS WSGIDaemonProcess wmt-api-pbs threads=16 maximum-requests=10000 python-path=/data/web/htdocs/wmt/api/pbs/conda/lib/python2.7/site-packages WSGIProcessGroup wmt-api-pbs WSGIScriptAlias /wmt/api-pbs /data/web/htdocs/wmt/api/pbs/bin/wmt_wsgi_main.py/ process-group=wmt-api-pbs Alias /wmt/api-pbs/static /data/web/htdocs/wmt/api/pbs/static <Directory /data/web/htdocs/wmt/api/pbs/> Require all granted </Directory>
Restart the web server.
sudo /usr/local/httpd/bin/apachectl -k restart
If, at a later point, components are updated, or other changes are made, instead of restarting the web server, just touch the bin/wmt_wsgi_main.py file.
cd $install_dir touch bin/wmt_wsgi_main.py
