Installation¶
This is an installation guide to get PyRate running on various platforms. Follow the instructions to install PyRate and run a small toy example using the test dataset included in the repository.
Dependencies¶
The following dependencies need to be on your system (or in your working environment) prior to PyRate installation:
The following optional dependency should be on your system if you want to make use of MPI processing:
Open MPI, versions 2.1.6, 3.0.4, 3.1.4 or 4.0.2
The versions of each package stated above have been tested to work using GitHub Actions continuous integration testing.
Other Python dependencies that will be installed are listed in
PyRate/requirements.txt
.
Platforms¶
PyPI¶
PyRate and its Python dependencies can be installed directly from the Python Package Index (PyPI):
pip install Py-Rate
Linux¶
These instructions have been tested using Ubuntu 18.04. If using another
Linux distribution, you will have to install packages equivalent to those
listed in PyRate/scripts/apt_install.sh
.
Clone the repository, install required packages and install PyRate:
git clone git@github.com:GeoscienceAustralia/PyRate.git
./PyRate/scripts/apt_install.sh
# Add GDAL includes to C/CPLUS paths before building Python GDAL bindings.
export C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/include/gdal
export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:/usr/include/gdal
python3 setup.py install
These commands will compile the executable program pyrate
.
Docker¶
Docker can be used to run PyRate on Linux, Windows and MacOS systems.
Note
Docker is the recommended method to use PyRate under Windows.
The only system pre-requisites for using docker are Git and Docker Desktop (e.g. for Windows https://docs.docker.com/docker-for-windows/ ). The other system dependencies required by PyRate are installed in to the docker container itself during the build operation. Once Git and Docker Desktop are installed, and Docker Desktop is running, execute the following at a command line prompt (e.g. PowerShell in Windows 10):
git clone git@github.com:GeoscienceAustralia/PyRate.git
cd PyRate
docker build -t pyrate-image .
docker run -it pyrate-image
Note
The image name “pyrate-image” is not mandatory. You are free to name the docker image whatever you choose.
The docker build
command builds the docker container described in the
Dockerfile
in the PyRate repo. It first starts by setting up an Ubuntu linux
system image, and then installs the required system and python dependencies.
Once the docker container is running (you will see a different-looking command prompt in the PowerShell), execute the following commands:
source /usr/local/bin/virtualenvwrapper.sh
workon pyrate
cd PyRate
python3 setup.py install
pyrate --help
pyrate workflow -f input_parameters.conf
The pyrate
executable program is built as part of the docker build
step.
If the pyrate
executable is reported as “not found”, re-run the compilation:
cd PyRate
python3 setup.py install
HPC¶
These instructions have been written for the Gadi supercomputer of the National Computational Infrastructure (NCI). The process for other HPC platforms may differ.
Login to Gadi and clone the PyRate repository:
ssh <user_name>@gadi.nci.org.au
cd ~
git clone https://github.com/GeoscienceAustralia/PyRate.git
Load the required Gadi modules (this will also remove the default NCI GDAL Python bindings so we can build and use our own):
source ~/PyRate/scripts/nci_load_modules.sh
Create a Python virtual environment:
python3 -m venv ~/PyRateVenv
source ~/PyRateVenv/bin/activate
Install PyRate:
cd ~/PyRate
python3 setup.py install
Following this, PyRate will be available for PBS jobs. To verify the installation, first run an interactive session:
qsub -I -q express -l walltime=01:00:00,mem=16Gb,ncpus=4,wd
Once the session has started, you will need to reactivate your virtual environment and reload the required modules. There is a call to activate the virtual environment built into the script below so this can be completed with a single command:
source ~/PyRate/scripts/nci_load_modules.sh
Verify Installation¶
To verify PyRate has been successfully installed, run the full processing
workflow with the example config file and the small dataset included
in the repository. If you compiled the pyrate
executable program:
pyrate workflow -f input_parameters.conf
If you installed from the Python Package Index (PyPI), you won’t have a
pyrate
executable program. Instead use:
python3 pyrate/main.py workflow -f input_parameters.conf
If the installation has been successful, this workflow will complete without errors and output geotiff files will be generated in:
~/PyRate/out