Troubleshooting¶
Some common/known issues with PyRate that users may encounter are described below.
If your issue is not covered below, please contact the Geoscience Australia InSAR Team for help.
ValueError: too many values to unpack (expected 2)¶
Problem: During prepifg
step, the following error is encountered:
Traceback (most recent call last):
File "pyrate/main.py", line 131, in <module>
main()
File "pyrate/main.py", line 103, in main
prepifg.main(params)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/pyrate/prepifg.py", line 57, in main
do_prepifg(gtiff_paths, params)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/pyrate/prepifg.py", line 88, in do_prepifg
_prepifg_multiprocessing(gtiff_path, xlooks, ylooks, exts, thresh, crop, params)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/pyrate/prepifg.py", line 112, in _prepifg_multiprocessing
coherence_path=coherence_path, coherence_thresh=coherence_thresh)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/pyrate/core/prepifg_helper.py", line 187, in prepare_ifg
op = output_tiff_filename(raster.data_path, out_path)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/pyrate/core/shared.py", line 1222, in output_tiff_filename
fname, ext = os.path.basename(inpath).split('.')
ValueError: too many values to unpack (expected 2)
Reason: This is caused by multiple double dots being used in the file names.
Solution: Ensure only a single dot is given in the file names, delimiting the file extension.
For example, rename 20151219-20160112.unw.tif
to 20151219-20160112_unw.tif
.
Stack Rate map appears to be blank/empty¶
Problem: Output of Stack Rate algorithm contains NaN values causing “merge“ to fail:
Traceback (most recent call last):
File "~/PyRateVenv/bin/pyrate", line 11, in <module> load_entry_point('Py-Rate==0.4.0', 'console_scripts', 'pyrate')()
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/main.py", line 154, in main merge_handler(args.config_file)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/main.py", line 87, in merge_handler merge.main(config.__dict__)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/merge.py", line 64, in main create_png_from_tif(output_folder_path)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/merge.py", line 130, in create_png_from_tif minimum, maximum, mean, stddev = srcband.GetStatistics(True, True)
File "/apps/gdal/3.0.2/lib/python3.7/site-packages/osgeo/gdal.py", line 2610, in GetStatistics return _gdal.Band_GetStatistics(self, *args)
RuntimeError: ~/out/stack_rate.tif, band 1: Failed to compute statistics, no valid pixels found in sampling.
Traceback (most recent call last):
File "~/PyRateVenv/bin/pyrate", line 11, in <module> load_entry_point('Py-Rate==0.4.0', 'console_scripts', 'pyrate')()
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/main.py", line 154, in main merge_handler(args.config_file)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/main.py", line 87, in merge_handler merge.main(config.__dict__)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/merge.py", line 64, in main create_png_from_tif(output_folder_path)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/merge.py", line 130, in create_png_from_tif minimum, maximum, mean, stddev = srcband.GetStatistics(True, True)
File "/apps/gdal/3.0.2/lib/python3.7/site-packages/osgeo/gdal.py", line 2610, in GetStatistics return _gdal.Band_GetStatistics(self, *args)
RuntimeError: ~/out/stack_rate.tif, band 1: Failed to compute statistics, no valid pixels found in sampling.
Traceback (most recent call last):
File "~/PyRateVenv/bin/pyrate", line 11, in <module> load_entry_point('Py-Rate==0.4.0', 'console_scripts', 'pyrate')()
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/main.py", line 154, in main merge_handler(args.config_file)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/main.py", line 87, in merge_handler merge.main(config.__dict__)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/merge.py", line 64, in main create_png_from_tif(output_folder_path)
File "~/PyRateVenv/lib/python3.7/site-packages/Py_Rate-0.4.0-py3.7.egg/merge.py", line 130, in create_png_from_tif minimum, maximum, mean, stddev = srcband.GetStatistics(True, True)
File "/apps/gdal/3.0.2/lib/python3.7/site-packages/osgeo/gdal.py", line 2610, in GetStatistics return _gdal.Band_GetStatistics(self, *args)
RuntimeError: ~/out/stack_rate.tif, band 1: Failed to compute statistics, no valid pixels found in sampling.
Reason: The maxsig
parameter is too low, resulting in stack rate values being replaced by NaNs. maxsig
is a threshold for masking stack rate pixels according to the corresponding stack error estimate saved in out/tmpdir/stack_error_*.npy
.
Solution: Increase maxsig
, then re-run the merge
step. Maximum permittable value for maxsig
is 1000 mm.
Failure of APS spatial low pass filter¶
Problem: Atmospheric corrections during correct
step fails due to the interpolated grid used for correction being empty:
+8s pyrate.aps:INFO Applying spatial low pass filter
Traceback (most recent call last):
File "~/PyRateVenv/bin/pyrate", line 11, in <module>
load_entry_point('Py-Rate==0.3.0.post3', 'console_scripts', 'pyrate')()
File "~/PyRateVenv/lib/python3.6/site-packages/Click-7.0-py3.6.egg/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "~/PyRateVenv/lib/python3.6/site-packages/Click-7.0-py3.6.egg/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "~/PyRateVenv/lib/python3.6/site-packages/Click-7.0-py3.6.egg/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "~/PyRateVenv/lib/python3.6/site-packages/Click-7.0-py3.6.egg/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "~/PyRateVenv/lib/python3.6/site-packages/Click-7.0-py3.6.egg/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "~/PyRateVenv/lib/python3.6/site-packages/Py_Rate-0.3.0.post3-py3.6.egg/pyrate/scripts/main.py", line 69, in linrate
run_pyrate.process_ifgs(sorted(dest_paths), params, rows, cols)
File "~/PyRateVenv/lib/python3.6/site-packages/Py_Rate-0.3.0.post3-py3.6.egg/pyrate/scripts/run_pyrate.py", line 391, in process_ifgs
_wrap_spatio_temporal_filter(ifg_paths, params, tiles, preread_ifgs)
File "~/PyRateVenv/lib/python3.6/site-packages/Py_Rate-0.3.0.post3-py3.6.egg/pyrate/aps.py", line 63, in _wrap_spatio_temporal_filter
spatio_temporal_filter(tsincr, ifg, params, preread_ifgs)
File "~/PyRateVenv/lib/python3.6/site-packages/Py_Rate-0.3.0.post3-py3.6.egg/pyrate/aps.py", line 86, in spatio_temporal_filter
ts_aps = mpiops.run_once(spatial_low_pass_filter, ts_hp, ifg, params)
File "~/PyRateVenv/lib/python3.6/site-packages/Py_Rate-0.3.0.post3-py3.6.egg/pyrate/mpiops.py", line 54, in run_once
f_result = f(*args, **kwargs)
File "~/PyRateVenv/lib/python3.6/site-packages/Py_Rate-0.3.0.post3-py3.6.egg/pyrate/aps.py", line 192, in spatial_low_pass_filter
_interpolate_nans(ts_lp, params[cf.SLPF_NANFILL_METHOD])
File "~/PyRateVenv/lib/python3.6/site-packages/Py_Rate-0.3.0.post3-py3.6.egg/pyrate/aps.py", line 208, in _interpolate_nans
_interpolate_nans_2d(a, rows, cols, method)
File "~/PyRateVenv/lib/python3.6/site-packages/Py_Rate-0.3.0.post3-py3.6.egg/pyrate/aps.py", line 224, in _interpolate_nans_2d
method=method
File "~/PyRateVenv/lib/python3.6/site-packages/scipy-1.3.0-py3.6-linux-x86_64.egg/scipy/interpolate/ndgriddata.py", line 226, in griddata
rescale=rescale)
File "interpnd.pyx", line 846, in scipy.interpolate.interpnd.CloughTocher2DInterpolator.__init__
File "qhull.pyx", line 1836, in scipy.spatial.qhull.Delaunay.__init__
File "qhull.pyx", line 276, in scipy.spatial.qhull._Qhull.__init__
ValueError: No points given
Solution: Use more interferograms as input and/or reduce the threshold parameters ts_pthr
, pthr
, tlpfpthr
in the configuration file.
In general, users are advised to input a network of small-baseline interferograms
that has at least 2 interferometric connections per SAR image epoch. Furthermore,
make sure that ts_pthr
, pthr
and tlpfpthr
are smaller than the number
of image epochs. To check that the spatio-temporal filters worked correctly, users can check that
the numpy arrays saved at /<outdir>/aps_error/*aps_error.npy
contain numeric values and not NaNs.
Out of memory errors¶
Problem: PyRate is memory intensive. You may receive various out of memory errors if there is not enough memory to accommodate the images being processed.
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker. The exit codes of the workers are {EXIT(1), EXIT(1), EXIT(1)}
Solution: Increase the amount of memory available. On HPC systems this can be done by increasing the value provided to the mem
argument when submitting a PBS job, e.g.:
mem=32Gb
If no more memory can be called upon, users can try running the job in serial, or reducing the size of the interferograms by increasing the multi-looking factors applied during prepifg
(parameters ifglksx
and ifglksy
).
Incorrect modules loaded on Gadi¶
Problem: PyRate requires certain versions of Python, GDAL and Open MPI to be loaded on Gadi and other HPC systems. While sourcing the PyRate/scripts/nci_load_modules.sh script will load the correct modules, you may need to unload previously unloaded modules.
Example of errors caused by module conflicts:
ERROR:150: Module 'python3/3.7.2' conflicts with the currently loaded module(s) 'python3/3.4.3-matplotlib'
ERROR:150: Module 'gdal/2.2.2' conflicts with the currently loaded module(s) 'gdal/2.0.0'
Solution: Purge the loaded modules and source the nci_load_modules.sh
script:
module purge
source ~/PyRate/scripts/nci_load_modules.sh