How to generate a smooth derivative after fitting a curve to the data?

Hi all,

I have a set of data which is attached. I used spline function to fit the data and then fnder to take the second derivative and my results ended to be very noisy. I tried fit(x,y,'smoothinhgspline') and I got a noisy derivative. I don't know how to find a smooth derivative. I'd appreciate your suggestions.

Here's my code:

load Data %Matt J added

% using cubic spline
pp = spline(x,y);
derX= fnder(pp,2);
yy = fnval(derX,x);
% using fit
fit1 = fit( x, y, 'smoothingspline' );
[d1,d2] = differentiate(fit1,x);

when I plot yy and d2, none are smooth, they are very noisy.

plot(x,yy,x,d2) %Matt J added
legend('Non-Smoothed','Smoothed'); %Matt J added

[Matt J]

We need a demonstration of the problem. We don't know what you did or how you examined the results.

[azarang asadi]

Done

[Torsten]

The usual way is to filter your signals first:

https://de.mathworks.com/help/signal/ref/sgolayfilt.html

[azarang asadi]

The thing is that the signal is filtered and pretty smooth.

[Torsten]

Yes, I see now: In my opinion, also the 2nd derivative is quite smooth. What do you need it for that you think it's too noisy ?

[azarang asadi]

so I attached the second derivative to make sure that you are getting the same thing. if so, it is very noisy as it supposed to look like a smooth curve.

[Torsten]

I get this graph for the second derivative in OCTAVE:

[Bruno Luong]

% using cubic spline
pp = spline(x,y)

This is NOT fitting, this is interpolation. So it's not surprise that you get noisy derivatives.

[azarang asadi]

@Torsten I'm using MATLAB.

@Bruno Luong well okay, then what do you suggest me to do? I have attached the data as well.

[Matt J]

I've added some lines of code and RUN output to the original post.

[Torsten]

I get this graph for the second derivative in OCTAVE:

Sorry, it's the first derivative that I included as graphics.

Answer 1

[Bruno Luong]

You have to FIT the spline, not interpolate.

For example I use my own tool BSFK avalable in FEX.

load Data.mat;
pp = BSFK(x,y); % FEX 
% Check the spline model
xq = linspace(min(x),max(x),100);
pp1 = ppder(pp); pp2 = ppder(pp1); % You might use fnder, I don't have the toolbox

subplot(2,1,1)
plot(x,y,'.r',xq, ppval(pp,xq),'b')
legend('data', 'spline fitting')
subplot(2,1,2)
plot(xq,ppval(pp2,xq),'b')
ylabel('second derivative')

[azarang asadi]

pp2 = ppder(pp), this should be pp2 = ppder(pp1), but yeah seems like it's giving me what I need! Thank you

[Bruno Luong]

Ops I correct the error of pp2. Thank you

[azarang asadi]

The first derivative was looking good and close to what I expected it to be but the second derivative is way off. Cause I'm trying to compare it to the sources available in literature.

[Bruno Luong]

OK attached is the script, graphical plot and MATLAB derivative data, if you find anything incoherent with the second derivative (computed by 2 ways) please let me know.

If not you can contact the author of whatever the literature you read and ask him/her to correct.

[azarang asadi]

Thank you so much for all the help. appreciated

Answer 2

[Matt J]

You can try some different choices of the smoothing parameter,

load Data

% using cubic spline
pp = spline(x,y);
derX= fnder(pp,2);
yy = fnval(derX,x);

plot(x,yy,'--'); hold on

for p=[0.9999,0.999,0.95]
% using fit
fit1 = fit( x, y, 'smoothingspline' ,SmoothingParam=p);
[d1,d2] = differentiate(fit1,x);
legend(string(p))
plot(x,d2);

end; hold off
legend(["Non-Smoothed","p="+string([0.9999,0.999,0.95])])

[azarang asadi]

Thank you so much for all the help. appreciated