How to draw a line of best fit on filtered discrete data sets

How can i draw a line of best fit on a subset of the full data set?

I want a line of best fit but only using the data in blue below

So ignore the green, yellow, orange, red, only use the blue part

Now that is easily filtered down to, so know which parts are the blue part is easy, but getting to somehow draw a line of best fit to only use the blue is the challeging part

I can filter the data set to only show the data i want to use for the line of best fit, as shown in graph below

Now all i need it to find a way to draw a line of best fit on this data set, but when i try polyfit and other best fit functions, it gets all messed up

I'm assuming because of the gaps in the dataset.

So how can i draw a line of best fit for the chart above?

[Mathieu NOE]

hello

so the gaps correspond of some data exceeding a given threshold ... did you simply remove the data or you replaced it with NaN's ?

polyfit cannot work with data containing NaN's - they must be removed , this is one trick :

If the NaNs occur in the same locations in both the X and Y matrices, you can use a function call like the following, your_function( X(~isnan(X)), Y(~isnan(X)) ). If the NaNs don't occur in the same locations, you will have to first find the valid indices by something like, `X(~isnan(X)| isnan(Y))'

[Scott MacKenzie]

Please provide the data

[Rizwan Khan]

Dear Sir,

I have not used NaNs i have removed the data.

The first graph has many more data points while the second graph has less data points.

The gaps are because i've removed some of the data from the first graph

How can i share the data?

[Rizwan Khan]

Here is some code i use, hoepfully this can make sense, particulalarly how i filter the data

%get data, detrend, normalise, plot

[ClosePrice, t] = GetCryptoDataFromCoingGeckoAPI(cell2mat(symbols(1)), BaseCurrency);

detrend_sdata = CalculateDetrendedDataByTrendDiff(ClosePrice, PolyDegree);

normalised_detrendedData = normalize(detrend_sdata, 'range');

PlotWithDetrendedColorMap(s, ClosePrice, t, normalised_detrendedData, [], 'Daily', numColourBands, PolyDegree);

y = linspace(0,1,numColourBands + 1);

filter = normalised_detrendedData <= y(3) & normalised_detrendedData > y(2) ;

normalised_detrendedDataFiltered = normalised_detrendedData(filter);

closePriceFiltered = ClosePrice(filter);

tFiltered = t(filter);

%x = datenum(tFiltered);

%p = polyfit(x,closePriceFiltered,1);

%p = fit(x,closePriceFiltered,'poly2');

%f = polyval(p,x);

figure;

scatter(tFiltered,closePriceFiltered);

xlabel('Time');

ylabel('Price');

set(gca,'yscale','log');

grid on;

[Scott MacKenzie]

You post data via the Attachments button when you are writing your question:

If you insert code in your question, it's always best to have it properly appear as code. To do this, select the code and click the code button to get the code propertly formatted in your question.

Answer 1

[Mathieu NOE]

I used this simple example and created a gap in the data (not NaN's)

there is no problem to use polyfit in that situation

%     Example: simple linear regression with polyfit
 
      % Fit a polynomial p of degree 1 to the (x,y) data:
        x = 1:100;
        x(x>25 & x <75) = [];
        y = 0.3*x + 2*randn(1,length(x));
        p = polyfit(x,y,1);
 
      % Evaluate the fitted polynomial p and plot:
        f = polyval(p,x);
        plot(x,y,'o',x,f,'-')
        legend('data','linear fit')

[Rizwan Khan]

Dear Sir,

Your code helped me and i was able to make it work, however, i don't like the fit.

How can i fit a log curve?

When i try using p = fit(x,y,'exp1'); i get an error that it doesnt like NAN

Secondly, how can i extend the best fit line out further ?

[Mathieu NOE]

hello

to do it on a y log scale , the trick is to do the fit not on y but on log10(y) and then convert back the fitted model by doing y_fitted = 10.^(a x + b) , where a and b are the two parameters (from the p vector) obtained with polyfit

demo :

%     Example: simple log regression with polyfit
 
      % Fit a polynomial p of degree 1 to the (x,y) data:
        x = 1:100;
        x(x>25 & x <75) = [];
        y = exp(0.3*x + 2*randn(1,length(x)));
        figure(1), semilogy(x,y,'o');
        p = polyfit(x,log10(y),1);
        
 
      % Evaluate the fitted polynomial p and plot:
        f = polyval(p,x);
        fl = 10.^(f); % convert back from log to lin
        
        figure(2), semilogy(x,y,'o',x,fl,'-')
        legend('data','log fit')

[Rizwan Khan]

Thank You Sir,

This has helped resolve my issues

I now wanted to draw some confidence intervals above and below the line, the challenge is that it seems to not draw them right, potentially because i'm getting confused if i also need to 1log10 it and reverse it or not.

If you could assist with that, it would be highly appreciated

[Mathieu NOE]

hello

just found this :

95% confidence interval on a linear regression with polyfit? - MATLAB Answers - MATLAB Central (mathworks.com)

[Star Strider]

@Mathieu NOE — The ‘polyparci’ function calculates the parameter confidence intervals. To get confidence intervals on the fitted line, see the related polypredci funciton.

[Rizwan Khan]

Dear Sir,

I used polypredci, however, it doesn't seem to work.

It simply draws the same polynomial, and so i tried to change its input alpha value from 0.95 to 0.5 then to 0.1 and the output is the same, it just gives the same polynomial not an confidence interval around the polynomial.

So i send the same input into the polypredci as i did into the polyfit

for eg

[p_lower, S_lower] = polyfit(x_lower,log10(y_lower),3);

f_lower = polyval(p_lower,xx_lower);

fl_lower = 10.^(f_lower); % convert back from log to lin

plot(inputTime,fl_lower,'b');

[ p, yhat, ci ] = polypredci( xx_lower, fl_lower, 3, 0.10);

plot(inputTime,yhat,'r');

The two plots are prety much the same, and regardless of if you change the alpha to be 0.95 or 0.1 the plot is the same

[Mathieu NOE]

Thank you @Star Strider for your help

@Rizwan Khan try ths , hopefully I didn't make any wrong usage of polypredci

this is the plot associated with this demo code :

%     Example: simple log regression with polyfit
 
      % Fit a polynomial p of degree 1 to the (x,y) data:
        x = 1:100;
        x(x>25 & x <75) = [];
        y = exp(0.03*x + 0.5*randn(1,length(x)));
        yl = log10(y);
        
        
%%%%%%%%%%%%%%%%%%%%
n = 1; % Polynomial Order
alfa = 0.99; % Desired Significance
xv = linspace(min(x), max(x))'; % Optional High-Resolution Vector
[ p, yhat, ci ] = polypredci( x, yl, n, alfa, xv ); % Define Both ‘alfa’ & ‘xv’

yhat = 10.^(yhat); % convert back from log to lin

figure(1)
semilogy(x, y, 'bp'); 
hold on
if length(yhat) == length(x)
semilogy(x, yhat, '--r')
semilogy(x, yhat+ci, '--g')
semilogy(x, yhat-ci, '--g')
elseif length(yhat) == length(xv)
semilogy(xv, yhat, '--r')
semilogy(xv, yhat+ci, '--g')
semilogy(xv, yhat-ci, '--g')
end
hold off
grid on