Fast alternative to assign 0 to NaNs

Hi all,

is there a astest way to assign to zeros the value NaN? From the Profiler I am noticing that this operation is taking more than a second in my code and would like it to be fastest.

I have some inefficiencies in the code (e.g. the fact that there are several for loop rather a unique one). Here is my code and where I would like it to be faster:

for i = 1:N_RIPETIZIONI
    for j = 1:size(colonna_i,2)
        %MATRICI_SIMULATE_nonan{i,j}=MATRICI_SIMULATE{i,j}.*A_nan;
        MATRICI_SIMULATE_nonan{i,j}(MATRICI_SIMULATE_nonan{i,j} == 0) = NaN; %%here
    end
end

MATRICI_SIMULATE_DUMMY = cell(N_RIPETIZIONI,size(colonna_i,2)); 


for i = 1:N_RIPETIZIONI
    for j = 1:size(colonna_i,2)
        %MATRICI_SIMULATE_DUMMY{i,j}=double(MATRICI_SIMULATE_DUMMY{i,j});
        MATRICI_SIMULATE_DUMMY{i,j}(A_nan == 0) = NaN; %%here
    end
end

Is this slowing up in speedd the result of the fact that I am operating with cell arrays? That I am operating in different for loops?

Thank you

[Stephen23]

"Is this slowing up in speed the result of the fact that I am operating with cell arrays? That I am operating in different for loops?"

Possibly. Amongst other things it depends on the sizes and classes of the content of the cell arrays, which you did not tell us. It also depends on what kind of processing you are doing on this data, which you did not tell us.

If all of your numeric data are the same sizes, then probably using one ND numeric array would make this particular operation faster: A(A==0) = NaN. But there are too many unknowns here.

[federico nutarelli]

Oh I see.

@Stephen23 thank you for the reply and clarrifications.

"it depends on the sizes and classes of the content of the cell arrays."

The cell array contains 1229x222 matrices and it's a 10x119 cell array. So we have a totale of 1190 matrices in there. The code is basically telling that I want the elements of the matrices to be double and to put NaN wherever another matrix, A, has zeros. I did not want to use "find" since it slows down the code.

"It also depends on what kind of processing you are doing on this data"

The whole sstory would be too long. However, say I have a 1229x222 matrix called A, a vecotr called colonna_i and MATRICI_SIMULATE which iss, similarly a cell arrray of size 10x119 containing 1229x222 matrices. Then what I am doing is simply this:

A_nan = A~=0;
N_RIPETIZIONI=10;

MATRICI_SIMULATE_nonan = cell(N_RIPETIZIONI,size(colonna_i,2)); % preallocate!

for ii = 1:N_RIPETIZIONI
    for jj = 1:size(colonna_i,2)
        tmp = MATRICI_SIMULATE{ii,jj}.*A_nan;
        tmp(tmp==0) = NaN;
        MATRICI_SIMULATE_nonan{ii,jj} = tmp;
    end
end

for i = 1:N_RIPETIZIONI
    for j = 1:size(colonna_i,2)
        MATRICI_SIMULATE_nonan{i,j}=MATRICI_SIMULATE{i,j}.*A_nan;
        MATRICI_SIMULATE_nonan{i,j}(MATRICI_SIMULATE_nonan{i,j} == 0) = NaN;
    end
end

MATRICI_SIMULATE_DUMMY = cell(N_RIPETIZIONI,size(colonna_i,2)); % preallocate!

%setting the non NaN values to 0-1 matrices:
for i = 1:N_RIPETIZIONI
    for j = 1:size(colonna_i,2)
        MATRICI_SIMULATE_DUMMY{i,j}=MATRICI_SIMULATE_nonan{i,j}>0;
    end
end


for i = 1:N_RIPETIZIONI
    for j = 1:size(colonna_i,2)
        MATRICI_SIMULATE_DUMMY{i,j}=double(MATRICI_SIMULATE_DUMMY{i,j});
        MATRICI_SIMULATE_DUMMY{i,j}(A_nan == 0) = NaN;
    end
end

[Matt J]

I agree with Stephen. Based on what you've shown us, it is not clear why your matrices are split into cells, when they could easily be a 3D array.

Answer 1

[Walter Roberson]

Asking about the fastest way to do something is almost always a mistake.

Suppose that the fastest way to do this particular sequence involved making sure that each of the cell contents is aligned in memory on a 4 Kb physical boundary, and using exactly 8 simultaneous cores. So you rewrite your code with a bunch of mex calls to ensure that memory is allocated the way you want (asking for aligned memory requires an operating-system-specific call but you need to use the MATLAB-supplied memory allocator routines...) And you test it all out, and after only three weeks of performance improvement work, you have saved at least 17 clock cycles in your calculation.

And then you go to run the code in real life... and your system goes to check email and you lose access to one of the 8 cores... and your code ends up taking 29 cycles longer than it otherwise would have.

And then you hand the code over to someone else, and on their system, your carefully tuned fastest code is always slower than the alternatives. Because to get the fastest code you had to take advantage of the exact microcode timings of the various instructions, and on their system the fine details of the microcode timing are slightly different and a cycle is lost each iteration instead of gaining a cycle.

Going for the fastest possible version of something is almost always a net loss (unless you are a chip manufacturer and getting the highest possible calculation rate is terribly important for advertising purposes.)

[federico nutarelli]

Thank you for the clarification. What would you suggest as being the right terminology for the task that I would like to achieve? Maybe the most "efficient" way to do what I would like?

Thank you. I am trrying to improve but am yet quite a beginner :)

[Walter Roberson]

You need to question any time you use a superlative such as "fastest" or "most" efficient. How do you measure efficiency? How do you handle the fact that making one aspect better might make another aspect worse?

Answer 2

[Jan]

Avoid calculate A_nan==0 repeatedly inside the loop. Do this once before the loop:

C = cell(10, 119);
for iC = 1:numel(C)
    C{iC} = rand(1229, 222);
end
A = randi([0,1], 1229, 222);

tic;  % Original version:
A_nan = (A ~= 0);
D = cell(size(C));
for i1 = 1:10
    for i2 = 1:119
        tmp        = C{i1, i2};
        tmp(A_nan == 0) = NaN;
        D{i1, i2}  = tmp;
    end
end
toc

Elapsed time is 2.895987 seconds.

tic;  % Move A_nan==0 out of the loop:
A_nan = (A == 0);   % A==0 instead of A~=0
D = cell(size(C));
for i1 = 1:10
    for i2 = 1:119
        tmp        = C{i1, i2};
        tmp(A_nan) = NaN;  % A_nan instead of A_nan==0
        D{i1, i2}  = tmp;
    end
end
toc

Elapsed time is 2.144534 seconds.

As suggested already, the masking is much faster, if you work with a [10 x 119 x 1229 x 222] array.