Extracting and summing data from a 3d matrix

Hi there,

I have some data in a 3d matrix 11*32*19. I want to a) identify all columns where there is a non-NaN and b) identify the blocks of non-NaNs (from 1 row long to 11 rowns long) and sum together the blocks in each of the 32 columns, giving a total value for each column against the column name for all the 19 'pages' combined. I have some code to do that for a 2d matrix (below), although i am just realising that it will miss the blocks that start at row 1 :/ so any suggestions on that front would be awesome too.

filename = '1001_1_2';
[locAll,typeAll]=find(isfinite(mVocs));  %extracting the locations of all the non NaNs (finite numbers)
indVocsAll=any(isfinite(mVocs));     
colindx = find(indVocsAll==1);
colres = zeros(length(colindx),1);
for i =1:length(colindx)
    tmpdat = mVocs(:,colindx(i));
    blocklength = zeros(length(tmpdat),1);
    for fi = 1:length(tmpdat)-1
        if isnan(tmpdat(fi)) && ~isnan(tmpdat(fi+1)) %finding where the column goes from NaN to 1
            count = fi+1;
            count2 = 1;
            while ~isnan(tmpdat(count)) %also count Non-Na
                count = count+1;
                count2 = count2+1;
            end
            %tmpdat2 = tmpdat(fi+1:fi+count2-1);
            blocklength(fi) = count2;
        end
    end
    blocklength = nonzeros(blocklength);
    savename = strcat(savedir, 'filename.mat');
    save(savename, 'blocklength');
    colres(i) = length(blocklength);
    colres(:,2) = colindx;
end

by counting the start of the number of blocks in each column - i am just realising that it will miss the blocks that start at row 1 :/ so any suggestions on that front would be awesome too.

[dpb]

" a) identify all columns where there is a non-NaN"

What does "column" mean in this context of a 3D array? Are we talking by plane so there are 32*19 columns or what?

I'd note that in

[locAll,typeAll]=find(isfinite(mVocs));

I think you again don't follow what find is returning -- if mVocs is the 3D array, then by the <documentation find> for the output argument col, "then col is a linear index over the N-1 trailing dimensions of X".

[Emu]

Hi there, this is the script for a 2d array. So column is the 2nd dimension (header?) (32). I want to convert this script to extract the same information across 3 dimensions (so different pages/planes yes). So identify all the values in each 2d column across all 3d pages. So if there are values in column 3 (header) in page/plane 1 and in column 3(header) in page/plane 7 - extrct that information and sum together. So using the code you gave me the other day I've extracted and just need to know how to sum together across multiple dimensions (I think).

mVocs is a 2d array with my original data (X*32). (time*variable). I have multiple of these. Within this I extracted some time blocks around events for multiple datasets - so i selected x number of rows and looked in these rows for data within the 32 header columns , and need to save this event related data seperately to compare between events in future which is why I saved it as 3d. I also need to look at the sum total for all events across each individual dataset.

So you mean I need to use something different to find - maybe ind2sub with find?

[dpb]

If this is for a 3D array and can find what is needed on a 2D basis, then turn the 2D code into a function to which you can pass each plane of the 3D array in turn is one way.

I would still suggest that it isn't at all clear just what the end result is to be -- one can easily sum in MATLAB across any dimension by providing the optional 'DIM' parameter -- and use logical addressing across the array to select the elements needed/wanted. find itself is generally not needed for such operations; only if it is needed to return those locations themselves, not to address the elements.

But, one can generally arrange computations such that the results can be computed and then the locations of elements in the result obtained by their location rather than having to keep those first.

It would make things a whole lot easier to try to help here if you would provide a sample (small) dataset that fully illustrates the input conditions and then the desired output and describe the algorithm by which the output is obtained so that folks here can visualize the problem trying to be solved rather than having to try to figure that out from trying to debug non-working code.

[Matt J]

@Emily Greenwood How is the result you are looking for different from doing, simply,

result=sum(mVocs,[1,3],'omitnan');

[dpb]

What I've been trying to ascertain... :)

The descriptions have been so convoluted can't really tell -- there's something she's tried to do about strings of consecutive elements mixed in there as well that simply haven't been able to figure out just what is the desired end result...or even if she can produce an example manually or not (or whether hasn't yet defined the logic sufficiently that there even is a solution)

[Emu]

Attached is some example data. I want to know a) the indexes of all non-NaN locations and b) how many strings of consecutive non-NaNs in each index position; summed across 3 dimensions.

[Matt J]

Your demo.mat includes example input data, but it doesn't complete the example by showing the desired output for that data.

[Emu]

oh apologies - here is a desired output attached. So this is the summed number of events where there is at least one partial string of consecutive non-NaNs, what i want is the total number of all strings & partial strings too :) \sorry for being so unclear I am really new to this.

Answer 1

[Nipun]

Hi Emu,

I understand that you have a three dimensional matrix with dimensions: 11,32,19 with few NaN values. And, you are trying to get a sum-block output against each column which has a non-NaN value. I assume that the output will be a 32 x 19 matrix with one summation for each column across all rows.

In the example code below, I have generated a random matrix with the required dimensions. Please refer to the documentation for sum function in MATLAB for any clarifications.

M = rand(11,32,19);


% Generate 100 random indices to make them nan
idx = [];
for i=1:12000
    x = randi(size(M,1),1);
    y = randi(size(M,2),1);
    z = randi(size(M,3),1);
    idx = [idx [x,y,z]];
    M(x,y,z) = NaN;
end

% sum over all rows for each column across all 'pages' (3rd dimension)
A = sum(M,1,"omitnan");
size(A)

Hope this helps.

Regards,

Nipun