Find first non NaN element in a column of a matrix

Hi,

I have a realtively large Matrix with time series data. each column stands for one series. given a variable that defines which column I am interested in, I want to extract all non NaN element into a new Array.

Example

lets say myMatrix is 3000X100 and I am looking at column 15. In this column the first 15 elements are NaN and then 300 elements contain data and the rest of the column is NaN again.

I want to extract now exactly those 300 elements in a new Array, let's call it myVec and in Addition make sure it is sorted backwards (last element in myMatrix becomes first element in myVec).

any ideas?

thy

Answer 1

[Sebastian Castro]

Sure, there's lots of functionality that can help you. Here is an example.

First, I made up some "fake" data for myself. I have a 100x100 matrix where, in the 15th column, I set elements 1 through 15 and 75 through 100 to NaN. You can try it with your data if you'd like:

x = rand(100);
x([1:15,75:100], 15) = NaN;

Next you can pick out the "valid" indices of this column by logical indexing using the "isnan" function. Note that you want the values that are NOT NaN, hence the "~".

validIndices = ~isnan(x(:,15))

You can use these to index into the 15th column of your data. Just to verify, you can display the size of the vector.

myVec = x(validIndices,15)
size(myVec)

Finally, if you want to flip the matrix, you can use the "flipud" (flip up-down) function:

myVec = flipud(myVec)

If you're one of those people who prefers borderline illegible one-liners to show off their MATLAB prowess, you could always do this:

myVec = flipud(x(~isnan(x(:,15)),15));

- Sebastian

[Guillaume]

"If you're one of those people who prefers borderline illegible one-liners to show off their MATLAB prowess"

Actually, if I was reviewing your code and I saw this expression split over several lines I would ask what would be the purpose of splitting it in to and creating a variable part way through. Is the variable going to be reused later in the code? Is it just a temporary? Why then is there no comment stating its purpose? etc.

It's nothing about showing off, it's about writing code that cleary express your intent.

[Locks]

thanks guys, exactly what I was looking for, works perfectly!

[Sebastian Castro]

No problem!

@ Guillaume: I was joking with that comment, and I typed it even before I saw your response! Maybe it came out weird because of that, so my apologies if it seemed offensive.

Indeed, it's nice to eliminate temporary variables especially for large matrices that eat up a lot of memory. I just wanted to explain things step-by-step first so it's easier to understand, and then provide the efficient way at the end :)

Answer 2

[Guillaume]

This will get you all the non-nan elements of column col in a reverse order:

coldata = flipud(myMatrix(~isnan(myMatrix(:, col)), col))

This will do it for all the columns at once:

nonanmat = arrayfun(@(col) flipud(myMatrix(~isnan(myMatrix(:, col)), col)), ...
               1:size(myMatrix, 2), ...
               'UniformOutput', false);

Answer 3

[Tim Jackman]

Making an example matrix:

>> MyMatrix = zeros(3000,100);
>> MyMatrix(1:15,15) = NaN;
>> MyMatrix(16:315,15) = rand(300,1);
>> MyMatrix(316:end,15) = NaN;

Now pulling out only the 15th column:

>> ExtractVec = MyMatrix(:,15);

Using logical indexing, I can extract all values from this array that are not NaN:

>> myVec = ExtractVec(isnan(ExtractVec) == 0);

They will be sorted by their original index in the column. Use flipud to reverse the order:

>> myVec = flipud(myVec);

Hopefully this will help.

[Guillaume]

Traditionally,

isnan(x) == 0

is written

~isnan(x)

I don't know how good matlab's JIT is at optimising the former, but theoretically it involves a conversion from double to logical before a logical comparison while the latter is just a logical inversion.