How to smoothen the noisy part of the data?

I have taken out 10 examples of data sets. These data sets show gradual variation except towards the end where they show haphazard behavior. I have tried to correlate the problem with a previous answer related to signal smoothing (link): I have a very surface knowledge of which they have used.

In an ideal situation I would like to retain the original data as long as it varies gradually. Gradually varies refers to data which:

1. has sudden jump but then it varies smoothly (can be seen initially in plot 3 and 7).
2. does not have spikes where a spike is sudden up and down.

Keeping in mind the above results, I have adopted the reference code in the link and made some modifications e.g. movmean to movmedian and reduced the sample size of mov to 5 because data set is small. This can mostly work for me but it shows some inconsistent behavior at the locations encircled in the screenshot attached. Sometimes at the start the points do not match. Sometimes in the midway, it chooses the noisy points. I am not much proficient in the data filtering. I would like to smoothen the disturbed data without disturbing the acceptable data at all. I Will appreciate your help in this regard.

If it does not work, I would simply have to remove the noisy data.

clc
clear all
close all

%% Data have been sorted in x and y
load('chk.mat')
x = chkdta(:,1);
dta = chkdta(:,2:end);
figure

for i= 1:10
nexttile
hold on; grid on; box on;

% data
y = dta(:,i);

y_out = y;

% create the low pass filtered / smoothed version of y 
k = [5 5];
ys = smoothdata(y,'movmedian',k); % smoothed version of y 
% let's detect when we have too much noise content and replace y with ys in
% those sections
[B,A] = butter(2,0.03,'high');
yhp = filtfilt(B,A,y);
% logic signal 
ls = abs(yhp);
k = [5 5];
ls = movmedian(ls.^2,k);
ls = ls./max(ls);
idx = (ls>0.1);
y_out(idx) = ys(idx);

plot(x,y,':.r',DisplayName='original')
plot(x,y_out,'--ob',DisplayName='filtered')
end

[John D'Errico]

Unfortunately, your eye/brain is very good at knowing what you want those curves to look like. But writing an automatic scheme that willl be robust, and consistently perfect when faced with a variety of outliers is not oing to be trivial. Good luck.

Answer 1

[Star Strider]

See if using the filloutliers function will do what you want —

clc
clear all
close all

%% Data have been sorted in x and y
load('chk.mat')
x = chkdta(:,1);
dta = chkdta(:,2:end);

dta = filloutliers(dta, 'linear');                          % <— ADDED

figure

for i= 1:10
nexttile
hold on; grid on; box on;

% data
y = dta(:,i);

y_out = y;

% create the low pass filtered / smoothed version of y 
k = [5 5];
ys = smoothdata(y,'movmedian',k); % smoothed version of y 
% let's detect when we have too much noise content and replace y with ys in
% those sections
[B,A] = butter(2,0.03,'high');
yhp = filtfilt(B,A,y);
% logic signal 
ls = abs(yhp);
k = [5 5];
ls = movmedian(ls.^2,k);
ls = ls./max(ls);
idx = (ls>0.1);
y_out(idx) = ys(idx);

plot(x,y,':.r',DisplayName='original')
plot(x,y_out,'--ob',DisplayName='filtered')
end

This runs in MATLAB Online and seems to produce tthe result you want. I can’t run this here, since I get some sort of weird ‘Authentication failed’ error. MathWorks must be doing site maintenance. (The problem didn’t exist a few hours ago.)

.

[Muha]

I have tried this but with other parameters. The linear seems to work better. Thank you. Should have found this out. Do'h! Thank you for providing the solution.

Cheers

[Star Strider]

As always, my pleasure!