Averaging varying dimension (x y z) data over n cycles

Hi there,

So I have some collected data. The data is split in runs and each run has points defined by x y z coordinates. I have n number of runs and I want to average the data over the n cycles. The problem is that I am missing some of the data points/the dimension of the runs is not consistent (eg.Run1 has 150 data points, Run2 has 154 data points). I don't know where in the run the data is missing from. Is there any straightforward way of doing this?

Answer 1

[Ayush]

Hi Daniela,

I understand that you are trying to average the data given in (x,y,z) coordinates over ‘n’ runs, where each run is having different data points due to missing data.

To solve this, you can use the interpolation method which generates missing data points. MATLAB provides several methods to interpolate, linear interpolation being one of them. After interpolating you can take the average over the ‘n’ runs. Follow the below steps to interpolate the data and calculate the average:

1. Find maximum number of data points in a run by using: “max(cellfun(@(x) size(x, 1), data))”
2. Create an empty matrix to store the interpolated values: “zeros(maxPoints, 3);”
3. Determine number of data points using: ”size(data{i}, 1);”
4. Linearly interpolate missing data using: “interp1(1:numPoints, data{i}, linspace(1, numPoints, maxPoints));”
5. Store the data and calculate the average

An example code for the above steps is given below:

% Example data
n = 3; % Number of runs
data = cell(n, 1);
data{1} = [1 2 3; 4 5 6; 7 8 9]; % Run 1
data{2} = [10 11 12; 13 14 15]; % Run 2
data{3} = [16 17 18; 19 20 21; 22 23 24; 25 26 27]; % Run 3

% Step 1: Identify the maximum number of data points
maxPoints = max(cellfun(@(x) size(x, 1), data));


% Step 2: Loop through each run
for i = 1:n
    % Step 2: Generate empty matrix
    averagedData = zeros(maxPoints, 3);

    % Step 3b: Determine the number of data points
    numPoints = size(data{i}, 1);
    
    % Step 3c: Interpolate missing data points
    interpolatedData = interp1(1:numPoints, data{i}, linspace(1, numPoints, maxPoints));
    
    % Step 3c: Store interpolated data
    averagedData(1:maxPoints, :) = averagedData(1:maxPoints, :) + interpolatedData;
    data{i} = averagedData;
    disp(data{i})
end

    1.0000    2.0000    3.0000
    3.0000    4.0000    5.0000
    5.0000    6.0000    7.0000
    7.0000    8.0000    9.0000

    10    11    12
    11    12    13
    12    13    14
    13    14    15

    16    17    18
    19    20    21
    22    23    24
    25    26    27

% Step 4: Calculate the average
dim = ndims(data{1});
M = cat(dim+1,data{:});
meanArray = mean(M,dim+1);
disp(meanArray)

     9    10    11
    11    12    13
    13    14    15
    15    16    17

For more information on the “interp1” function, please refer to the following link:

https://www.mathworks.com/help/releases/R2020b/matlab/ref/interp1.html

Hope this helps!

Regards,

Ayush.

[Walter Roberson]

See also the File Exchange contribution "interparc" https://www.mathworks.com/matlabcentral/fileexchange/34874-interparc

[Daniela Sawyer]

Thank you all for this, I am aiming to trial Ayush solution soon. Back in the day I looked at different solutions for this in MATLAB but struggled to find one that works when you don't know where the missing data points are exactly. We use MATLAB a lot here to process data and it is the preferred method but over 1 year later there still doesn't seem to be a straightforward way. I will let you know if it works.

[Daniela Sawyer]

I finally found some time to test this and unfortunatley it does not seem to work for my problem. I have tested it with some data I had that was collected appropriately. I ran a range of calculations on the full set of data and then I removed some of the data points randomly (as in just random rows of data) and ran the interpolation method to get a full set of data points followed by running the same calculations. The results were significantly different so whatever the interpolation does, it does not do what I need it to do. Not sure if it's the interpolation method that's the issue or what. In the example you kindly provided, all data was missing at the end of the matrix. In my data, the data is missing from the middle, end, it's random.

[Steven Lord]

Please show us a small sample of your data with which we could experiment, and show us what you've already tried (adapting the solution in the answer at the top of this discussion.)