From 71150227e28a0b2f2623da979733b304601468a3 Mon Sep 17 00:00:00 2001
From: arinwongprommoon <arin.wongprommoon@ed.ac.uk>
Date: Tue, 11 Jul 2023 10:40:01 +0100
Subject: [PATCH] fix(postproc): detrend process only removes columns with all
NaNs
WHY IS THIS CHANGE NEEDED?:
- detrend process overzealously removes columns if input dataframes
have NaNs near the beginning or end.
- this is because by default, df.dropna() uses how="any", i.e. it drops
columns with at least one NaN
HOW DOES THE CHANGE SOLVE THE PROBLEM?:
- drops columns with at least one NaN is not the intention -- the
intention is to remove column filled with NaNs created by rolling
window operations.
- so, specified the how argument
WHAT SIDE EFFECTS DOES THIS CHANGE HAVE?:
- i honestly don't know why i didn't catch it in the 1.5 years this
process existed -- this seems so obvious.
- this is a deprecated signal processing post-process (i use the
butterworth filter instead), so i don't expect it to break things much
EVIDENCE THAT COMMIT WORKS:
---
src/postprocessor/core/processes/detrend.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/src/postprocessor/core/processes/detrend.py b/src/postprocessor/core/processes/detrend.py
index a2be9aec..7bbe744c 100644
--- a/src/postprocessor/core/processes/detrend.py
+++ b/src/postprocessor/core/processes/detrend.py
@@ -54,4 +54,6 @@ class detrend(PostProcessABC):
).mean()
# Detrend: subtract normalised time series by moving average
signal_detrend = signal.subtract(signal_movavg)
- return signal_detrend.dropna(axis=1) # Remove columns with NaNs
+ # Rolling window operations create columns that are all NaNs at the left
+ # and right edges because of the maths. This line removes these columns.
+ return signal_detrend.dropna(axis=1, how="all")
--
GitLab