Picker fails, as part of postprocessor pipeline run
Summary
When postprocessor
is run as part of the pipeline, the picker
process seems to fail due to an illegal assignment.
Steps to reproduce
aliby
version used: 0.1.49 (latest published on PyPI) -- this affects 0.1.48 too.
Chose any segmented experiment, and use the filepath as the folder
argument in https://git.ecdf.ed.ac.uk/swain-lab/aliby/skeletons/-/blob/alan/scripts/essential/postprocess_folder.py
What is the current bug behavior?
Running the full segmentation pipeline (aliby
0.1.48) succeeded without major errors on experiment staffa:297, and I was able to get signals e.g. extraction/Flavin_bgsub/max/mean
-- heatmaps showed that these look as expected. However, when using signal
, all the postprocessing
HDF5 datasets (e.g. postprocessing/buddings/extraction_general_None_volume
) were missing.
When running postprocessor
again, the picker
process seems to fail due to an illegal assignment. See traceback. This is strange given that it seemed to function fine as part of the complete pipeline.
What is the expected correct behavior?
picker
should run.
Logs/Traceback
/home/jupyter-arin/.conda/envs/aliby/lib/python3.8/site-packages/pandas/core/indexing.py:1732: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
self._setitem_single_block(indexer, value, name)
/home/jupyter-arin/.conda/envs/aliby/lib/python3.8/site-packages/pandas/core/indexing.py:723: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
iloc._setitem_with_indexer(indexer, value, self.name)
/home/jupyter-arin/.conda/envs/aliby/lib/python3.8/site-packages/pandas/core/frame.py:4906: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
return super().drop(
Traceback (most recent call last):
File "postprocess_folder.py", line 35, in <module>
process_file(file)
File "postprocess_folder.py", line 26, in process_file
pp.run()
File "/home/jupyter-arin/.conda/envs/aliby/lib/python3.8/site-packages/postprocessor/core/processor.py", line 299, in run
self.run_prepost()
File "/home/jupyter-arin/.conda/envs/aliby/lib/python3.8/site-packages/postprocessor/core/processor.py", line 200, in run_prepost
indices = self.picker.run(
File "/home/jupyter-arin/.conda/envs/aliby/lib/python3.8/site-packages/postprocessor/core/reshapers/picker.py", line 99, in run
self.mothers, self.daughters = self.cells.mothers_daughters
ValueError: too many values to unpack (expected 2)
Possible fixes
agora.io.cells.Cells
has a mothers_daughters
property which returns a single array with three columns: trap, mothers, and daughters (https://git.ecdf.ed.ac.uk/swain-lab/aliby/aliby/-/blob/b206c48a180a9a04eeebbe7f06ea0fa8fefaa4af/src/agora/io/cells.py#L284). However, picker
excepts two values (https://git.ecdf.ed.ac.uk/swain-lab/aliby/aliby/-/blob/b206c48a180a9a04eeebbe7f06ea0fa8fefaa4af/src/postprocessor/core/reshapers/picker.py#L99) rather than an array.
Naïvely, perhaps a fix could be (in picker
):
...
self.mothers, self.daughters = self.cells.mothers_daughters[:,1], self.cells.mothers_daughters[:,2]
or something similar.