helperWPHarmonicFilter function performs poorly on larger datasets

I'm trying to use the helperWPHarmonicFilter function to filter 60Hz interference from my data. The function seems to work well when I input a subset of my data ~4000 data points, but progressively gets less effective as I increase the number of data points (see power spectrums below). Is there something I can manipulate in the function that will allow it perform better on larger datasets?

Thanks!

4000 data point

25000 data points

Answer 1

[Yatharth]

Hi Chris,

I understand that you are trying to filter 60Hz interference from your data using "helperWPHarmonicFilter" which is working fine for a sample size of <4000 but doesn’t give the expected output for a large sample size.

I was able to find the helper function you mentioned in one of the examples from Wavelet Packet Harmonic Interference Removal.

https://www.mathworks.com/help/wavelet/ug/wavelet-packet-harmonic-interference-removal.html#mw_rtc_WPHarmonicInterferenceRemovalExample_M_A29A58D1:~:text=The%20helper%20function-,helperWPHarmonicFilter,-implements%20the%20baseline

Could you help me with the model and data that you are running so that I can reproduce the issue at my end?

I have few other approaches that you can try from your end and see if it works.

To improve the performance of the function on larger datasets, you can try the following approaches:

1. Down sampling: If your data has a high sampling rate, you can consider down sampling it to reduce the number of data points. This can help reduce the computational load on the function.
2. Windowing: You can divide your data into smaller windows or segments and apply the "helperWPHarmonicFilter" function to each window individually. This can help mitigate the performance degradation by processing smaller chunks of data at a time.
3. Optimization: Review the implementation of the "helperWPHarmonicFilter" function and check if there are any opportunities for optimization. Look for any unnecessary calculations or redundant operations that can be removed or simplified.
4. Parallelization: If your programming environment allows it, you can consider parallelizing the filtering process. By distributing the computation across multiple processors or threads, you can potentially improve the performance on larger datasets.

Alternatively, to filter out 60Hz interference from your data you could also look into:

1. Notch Filter https://www.mathworks.com/discovery/notch-filter.html
2. Fast Fourier Transform https://www.mathworks.com/discovery/fft.html#

I hope this helps.

[Chris Patrick]

Thank you Yatharth! I switched to windowing the data as you suggested and that has worked well.