Removing Unwanted data from a bunch of data

Hey there everyone

I have a set of data of an eye digram (which I attached its .mat file) and clustered it by color as shown below. However, there are a disturbing line crossing the eye diagram and made it untidy and I intend to remove it (as I indicated in the second shape with black highlighter) but I do not know how to do it. I attached the code by the way to get the main output but removing those unwanted data and make the diagram looks clean is my problem

Can anyone help me do this?

Answer 1

[Image Analyst]

One way to do it is to fit a quadratic to 3 points, the points closest to x=-0.2, x=0.3, and x=0.6

coefficients = polyfit([x1,x2,x3], [y1,y2,y3], 2); % FIt quadratic.

If the coefficient of the squared term is small enough, then it's a straight line between the left and right point and that means it's the segment you want to ignore.

if coefficients(1) < someValue
    % Don't include this segment
end

First go through and print out what all the coefficients are to see what most values are. Take the histogram of all the coefficients to see what are normal values and what are bad values that you want to exclude.

[Proman]

Thank you for your answer but where in my code should I apply your approach?

[Image Analyst]

At some point in your code do you have all the x and y into distinct rows or columns of a 2D matrix? I mean can we at some point do this (assuming each row of x or y is a single plot line):

[rows, columns] = size(x);
keeperRows = true(rows, 1);
allCoeffs = true(rows, 1);
for row = 1 : rows
    thisX = x(row, :);
    thisY = y(row, :);
    index1 = 1;
    index2 = find(x > 0.2, 1, 'first');
    index3 = find(x > 0.8, 1, 'first');
    coefficients = polyfit([x(index1),x(index2),x(index3)], [y(index1),y(index2),y(index3)], 2); % FIt quadratic.
    allCoeffs(row) = coefficients(1);
    if coefficients(1) < 0.1 % or whatever.... I have no idea
        keepers(row) = false;
    end
end
% Extract only the keepers
x = x(keepers, :);
y = y(keepers, :);
% Let's see what the coefficients of the squared terms are.
histogram(allCoeffs); % Let's visualize what all the slopes were so we can see what value an outlier would have.

[Proman]

Yes they are already distinguished. In fact, I already uploaded my code together with my data in my question. My problem is, I already filtered some of my data (line 6 to 30 in my code) but filtering this disturbing line in my graph is really tricky. I understand what your algorthim is but I can not apply it to my code or at least I do not figure out where should it be placed in my own code