Mastr-MS Server

The Mastr-MS server is a Django application which provides a web interface for user management, sample management, and access to experiment data.

Mastr-MS is distributed as a RPM for Redhat systems and as a deb package for Debian and Ubuntu systems.

Installation on Debian/Ubuntu

The deb package is known to work on Debian 8 (Jessie) and Ubuntu 14.04 LTS (Trusty Tahr).

Apt repository setup

To get started, put the following lines in /etc/apt/sources.list.d/ccg.list:

# Centre for Comparative Genomics Ubuntu package repository
deb http://repo.ccgapps.com.au/repo/ccg/ubuntu ccg ccg
deb-src http://repo.ccgapps.com.au/repo/ccg/ubuntu ccg ccg

Then run (as root):

curl http://repo.ccgapps.com.au/repo/ccg/ubuntu/ccg-archive.asc | apt-key add -
apt-get update

Database Setup

The database server package isn’t a dependency of the Mastr-MS package, because it’s possible to run the database on another host. The database therefore needs to be installed manually:

apt-get install postgresql
service postgresql start
update-rc.d postgresql enable

In order to benefit from automatic database initialization, the database needs to be installed and running before installing Mastr-MS.

Install

To download and install Mastr-MS, run:

apt-get install mastr-ms

Your database and web server will be automatically configured using the Debconf system.

Manual database initialization

If using a remote database or otherwise manually setting up the database, you should edit /etc/mastr-ms/database.conf and enter the correct settings.

To then initialize an empty database for Mastr-MS, run:

mastr-ms migrate

Configuration

You can customize /etc/mastr-ms/mastr-ms.conf as needed for your site. Each setting is documented in settings.

After changing SELF_URL_PATH, you must run this to update the database:

mastr-ms set_site

The Apache config file is installed in /etc/apache2/conf-available/mastr-ms.conf and can be adjusted as required.

Mastr-MS generally requires SSL to be enabled on the web server. If this is not possible, add the following line to /etc/mastr-ms/settings.py:

SSL_FORCE = False

Testing

Reload the web server:

service apache2 restart

Mastr-MS is available at https://your-web-host/mastr-ms/. A login page should be visible at this URL.

Data Repository and Syncing

The default data repository location is /var/cache/mastr-ms/scratch. This can be changed in the settings file or redirected with a symlink.

For production use, there should be ample disk space on this filesystem and it should be backed up.

The package creates a user called mastr-ms which owns the data repository directories.

To support data syncing, the mastr-ms user should be enabled and a password-less SSH key pair generated for it.

Upgrading to a new version

New versions of Mastr-MS are made available in the CCG Apt repository.

Before upgrading, please check the Release Notes for any special information relating the new version.

Install the latest version, run:

apt-get update && apt-get install mastr-ms

If there are any database migrations required, they will be run automatically. A precautionary database backup will be created in /var/cache/dbconfig-common/backups.

If you would like to manually migrate the database, just run:

mastr-ms migrate

Installation on CentOS

Yum repository setup

The Mastr-MS RPM is made for on Centos 6.x (x86_64). To satisfy dependencies, Epel and IUS repos need to be enabled:

sudo rpm -Uvh http://repo.ccgapps.com.au/repo/ccg/centos/6/os/noarch/CentOS/RPMS/ccg-release-6-2.noarch.rpm
sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sudo rpm -Uvh http://dl.iuscommunity.org/pub/ius/stable/CentOS/6/x86_64/ius-release-1.0-11.ius.centos6.noarch.rpm

Dependencies

The database server package isn’t a dependency of the Mastr-MS RPM, so it has to be installed manually:

sudo yum install postgresql-server

Note

These instructions are written for PostgreSQL. With minor alterations (issue MAS-32) Mastr-MS could work with MySQL/MariaDB, but at present only PostgreSQL is supported.

Install

Install the Mastr-MS RPM, replacing X.X.X with the desired version:

sudo yum install mastrms-X.X.X

Database Setup

If starting from a fresh CentOS install, you will need to configure PostgreSQL:

service postgresql initdb
service postgresql start
chkconfig postgresql on

To enable password authentication in PostgreSQL, you need to edit /var/lib/pgsql/data/pg_hba.conf. As described in the documentation, add the following line to pg_hba.conf:

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
host    all         all         127.0.0.1/32          md5

Then restart postgresql.

Database Creation

Create the database in the normal way for Django/PostgreSQL:

sudo su postgres
createuser -e -DRS -P mastrms
createdb -e -O mastrms mastrms
exit

The default database, username, password are all set to mastrms. These settings can be changed, see (Django Settings File).

Database Population

Run the Django database migration:

sudo mastrms migrate

Django will prompt to create a superuser. If you choose to create a superuser, ensure the username and e-mail address are exactly the same, otherwise you won’t be able to log in.

Alternatively, you can use the preconfigured user admin@example.com with password admin to log in. Once you have set up your own users, the admin@example.com user can be deleted.

Apache Web Server

The Mastr-MS RPM installs an example Apache config file at /etc/httpd/conf.d/mastrms.ccg. This config is designed to work out of the box with an otherwise unconfigured CentOS Apache installation. All that is needed is to rename mastrms.ccg to mastrms.conf so that Apache will pick it up.

If you have already made changes to the default Apache configuration, you may need to tweak mastrms.conf so that the existing setup is not broken. It may be necessary to consult the Apache and mod_wsgi documentation for this.

User Creation

It is a good idea to create a special user and group for data syncing:

SYNCUSER=maupload
adduser $SYNCUSER
su $SYNCUSER -c "mkdir --mode=700 /home/$SYNCUSER/.ssh"

Data Repository and Permissions

By default, the data repository is located at /var/lib/mastrms/scratch.

There should be ample disk space on this filesystem and some data redundancy would be desirable. If this is not the case, then you could mount a suitable file system at this path. If the data repository needs to be at another location, its path can be configured in the settings file.

The data sync user needs to be able to write to this directory, and the web server user needs to be able to read from this directory, so:

DATADIR=/var/lib/mastrms/scratch
mkdir -p $DATADIR/files $DATADIR/quotes
chown maupload:maupload $DATADIR $DATADIR/*
chmod 2770 $DATADIR $DATADIR/*

Also add the web server user to the maupload group so that it can read/write the data which maupload has rsynced in:

usermod -a -G maupload apache

Django Settings File

The config file for Mastr-MS is installed at /etc/mastrms/mastrms.conf. It contains basic settings that need to be changed for most sites, for example the database password.

SELinux and Mastr-MS

Mastr-MS does not yet ship with a SELinux policy (issue MAS-21). For Mastr-MS to function correctly on a CentOS server, SELinux must be disabled.

The CentOS wiki contains instructions on how to disable SELinux.

Upgrading to a new version

New versions of Mastr-MS are made available in the CCG yum repository.

Warning

Some versions will require “database migrations” to update the database. While every care is taken to ensure smooth upgrades, we still advise to make a backup of the database before upgrading. This can be done with a command such as:

su - postgres -c "pg_dump mastrms | gzip > /tmp/mastrms-$(date +%Y%m%d).sql.gz"

Before upgrading, please check the Release Notes for any special information relating the new version.

Install the Mastr-MS RPM, replacing X.X.X with the desired version:

sudo yum install mastrms-X.X.X

Run Django database migration:

sudo mastrms migrate

Testing

After changing the configuration or upgrading, start/restart the web server with:

service httpd restart

Mastr-MS is available at https://your-web-host/mastrms/. A login page should be visible at this URL.

Mastr-MS Settings File

The Mastr-MS settings file is called mastr-ms.conf or mastrms.conf depending on the system.

The following settings are customizable. There are also comments and example values for each setting within the settings file.

Option Description
dbtype Database backend – always use pgsql.
dbname The rest are standard database connection options.
dbuser
dbpass
dbserver Optional settings for remote database connection.
dbport
memcache Optional connection string(s) for memcached. Multiple servers are separated by spaces.
allowed_hosts Space-separated list of address permitted to access the server. Wildcards can be used as in the ALLOWED_HOSTS docs.
server_email “From” e-mail address for server-generated error mails.
email_host Details for SMTP server. User and password are optional.
email_port
email_host_user
email_host_password
alert_email Where error messages are sent.
return_email The “From” address on e-mail sent by mastr-ms.
logs_to_email E-mail address to receive datasync client log notifications.
keys_to_email E-mail address to receive datasync key upload notifications.
registration_to_email E-mail address to receive registration requests.
repo_user Mastr-MS will attempt to change ownership of data files to this user and group.
repo_group
repo_files_root Location of data files for experiments and quotes.
quote_files_root
secret_key Needs to be a secret random string, can be generated by a key generator program.
self_url_path This is the public URL which Mastr-MS is served from. It is used for links and redirects.

More advanced options appear in settings.py. Any of the Django Settings can be changed in this file.

Administration

There are two levels of administration necessary for Mastr-MS.

  • Management

    This involves administrating users, projects, quotes, experiments, etc. The URL for management is the normal Mastr-MS address, but only users who are in the admin group can see the interface.

    https://your-web-host/mastrms/

    The management interface is described in Using the System.

  • Django Admin

    This involves manipulation of database objects to configure the data sync system. Only admin users can access the address:

    https://your-web-host/mastrms/repoadmin/

    The Django Admin site can also be accessed from Dashboard → Repository → Admin.

Data Sync Node Client Configuration

Configuration of a new site is done by adding a Node client using the Django Admin. The fields should be set as follows.

Field Description
Organisation name These values determine how the node is visible in the data sync client.
Site name
Station name
Default data path This should be a subdirectory of $DATADIR (see Data Repository and Permissions).
Username This should be the data sync user (see User Creation).
Hostname The hostname or IP address of the Mastr-MS server.
Flags This controls the options the data sync client will pass to rsync. They should always be set to --protocol=30 -rzv --chmod=ug=rwX.

Instrument Method Configuration

Before runs can be created, an Instrument method must be created using the Django Admin. At present, the Instrument Method object isn’t used, but it must be set. The fields should be set as follows.

Field Description
Title Default Method
Method path A folder path on the lab machine, e.g. D:\mastrms
Method name Default Method
Version 1
Creator Your own username
Template CSV
The other fields Blank

Standard Operating Procedure Documents

If you would like to make SOP documents available for viewing, you can create objects in the Django Admin within the Repository / Standard operation procedures page.

Once the documents are uploaded, they can be attached to experiments and viewed through the Experiment Sample Preparation screen.

SSH Key Management

When the data sync clients hit Send Key, it sends the client’s public key via a HTTP post to a URL at the Mastr-MS site, and a view handles this, saving it to the publickeys directory on the server. It then sends an e-mail to the admins configured for the site, telling them that a new key has been uploaded, and they should append it on to the authorized_keys for the data sync user.

To install the key, run:

cat $DATADIR/files/publickeys/$ORG.$SITE.$STATION_id_rsa.pub \
    >> /home/$SYNCUSER/.ssh/authorized_keys

(Replace $DATADIR, $SYNCUSER and $ORG.$SITE.$STATION with your actual settings and the information from the e-mail.)

Once the key is added, the client should be able to “Handshake” with the server (see Configuration).

If the key isn’t working, try checking the authorized_keys permissions.