Add a dummy run to the Pipeline
Summary
Add an initial time-step "dummy" run of the pipeline with a known output to check for parametrisation bugs.
Current behaviour/setbacks
If a run fails, there is no straight-forward way to know whether it is a parametrisation problem or a deeper bug. This makes debugging difficult especially with an incomplete knowledge of the pipeline.
Desired behaviour/advantages
Before fully running a pipeline, we would run a few steps with spoofed "dummy" data, with a predictable output. If the pipeline fails during the dummy run, it should be due to a missing or incorrect parameter. The obtained output can be compared to the expected output for easier debugging.
Implementation sketch
- Create a sample dataset that can be used for calibration; this could be the same as the unit test dataset
- Create way to modify the dataset to match the requested parameters (e.g. fake channels, resize images)
- Create a
calibrate
method in the pipeline that saves results to a temporary file- if the results are correct: delete the temporary file and run the rest
- if the results are incorrect: save the temporary file (for manual inspection) and error out