Installation

This is an installation guide to get PyRate running on various platforms. Follow the instructions to install PyRate and run a small toy example using the test dataset included in the repository.

Dependencies

The following dependencies need to be on your system (or in your working environment) prior to PyRate installation:

  • Python, versions 3.7, 3.8 or 3.9.

  • GDAL, versions 3.0.2 or 3.0.4

The following optional dependency should be on your system if you want to make use of MPI processing:

  • Open MPI, versions 2.1.6, 3.0.4, 3.1.4 or 4.0.2

The versions of each package stated above have been tested to work using GitHub Actions continuous integration testing.

Other Python dependencies that will be installed are listed in PyRate/requirements.txt.

Platforms

PyPI

PyRate and its Python dependencies can be installed directly from the Python Package Index (PyPI):

pip install Py-Rate

Linux

These instructions have been tested using Ubuntu 18.04. If using another Linux distribution, you will have to install packages equivalent to those listed in PyRate/scripts/apt_install.sh.

Clone the repository, install required packages and install PyRate:

git clone git@github.com:GeoscienceAustralia/PyRate.git
./PyRate/scripts/apt_install.sh

# Add GDAL includes to C/CPLUS paths before building Python GDAL bindings.
export C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/include/gdal
export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:/usr/include/gdal

python3 setup.py install

These commands will compile the executable program pyrate.

Docker

Docker can be used to run PyRate on Linux, Windows and MacOS systems.

Note

  • Docker is the recommended method to use PyRate under Windows.

The only system pre-requisites for using docker are Git and Docker Desktop (e.g. for Windows https://docs.docker.com/docker-for-windows/ ). The other system dependencies required by PyRate are installed in to the docker container itself during the build operation. Once Git and Docker Desktop are installed, and Docker Desktop is running, execute the following at a command line prompt (e.g. PowerShell in Windows 10):

git clone git@github.com:GeoscienceAustralia/PyRate.git
cd PyRate
docker build -t pyrate-image .
docker run -it pyrate-image

Note

  • The image name “pyrate-image” is not mandatory. You are free to name the docker image whatever you choose.

The docker build command builds the docker container described in the Dockerfile in the PyRate repo. It first starts by setting up an Ubuntu linux system image, and then installs the required system and python dependencies.

Once the docker container is running (you will see a different-looking command prompt in the PowerShell), execute the following commands:

source /usr/local/bin/virtualenvwrapper.sh
workon pyrate
cd PyRate
python3 setup.py install
pyrate --help
pyrate workflow -f input_parameters.conf

The pyrate executable program is built as part of the docker build step. If the pyrate executable is reported as “not found”, re-run the compilation:

cd PyRate
python3 setup.py install

HPC

These instructions have been written for the Gadi supercomputer of the National Computational Infrastructure (NCI). The process for other HPC platforms may differ.

Login to Gadi and clone the PyRate repository:

ssh <user_name>@gadi.nci.org.au
cd ~
git clone https://github.com/GeoscienceAustralia/PyRate.git

Load the required Gadi modules (this will also remove the default NCI GDAL Python bindings so we can build and use our own):

source ~/PyRate/scripts/nci_load_modules.sh

Create a Python virtual environment:

python3 -m venv ~/PyRateVenv
source ~/PyRateVenv/bin/activate

Install PyRate:

cd ~/PyRate
python3 setup.py install

Following this, PyRate will be available for PBS jobs. To verify the installation, first run an interactive session:

qsub -I -q express -l walltime=01:00:00,mem=16Gb,ncpus=4,wd

Once the session has started, you will need to reactivate your virtual environment and reload the required modules. There is a call to activate the virtual environment built into the script below so this can be completed with a single command:

source ~/PyRate/scripts/nci_load_modules.sh

Verify Installation

To verify PyRate has been successfully installed, run the full processing workflow with the example config file and the small dataset included in the repository. If you compiled the pyrate executable program:

pyrate workflow -f input_parameters.conf

If you installed from the Python Package Index (PyPI), you won’t have a pyrate executable program. Instead use:

python3 pyrate/main.py workflow -f input_parameters.conf

If the installation has been successful, this workflow will complete without errors and output geotiff files will be generated in:

~/PyRate/out