How to apply this for-loop containing '<', designed for a scalar array, to several scalar arrays in a cell array?

Hello, i have the code below which was designed to be applied on a 100x6 scalar array called x and contains '>'. I wish to apply it on 4 scalar arrays 100x6 which are in a column as a cell array. Example: Cell :{ [100,6]

[100,6]

[100,6]

[100,6] }

for j=1:6
  q=quantile(x(:,j),0.97);
  id=find(x(:,j)>(q));
  x(id,j)=NaN;
end

Thanks.

Answer 1

[dpb]

There was no need for the loop in the original code -- quantile is already vectorized as are almost all ML-supplied functions. The first code should have been written as

Q=0.97;                 % use variables don't bury magic numbers inside argument lists
q=quantile(x,Q);        % find the limit
x(x>q)=nan;             % set the exceeding locations

If it is needed to know/have the values of q, then then iterate over the cell array with the above two lines, dereferncing each cell content in turn.

If the values of q aren't needed later, then the whole thing can be turned into a one liner...

x(x>quantile(x,Q))=nan;

and you can iterate over the cell array with the single expression.

But, NOTA BENE: it takes only the loop over the cell array, NOT a double loop as would wrapping the original code in another loop.

Assume the cell array containing the arrays is C --

Q=0.97;
for i=1:numel(C)
  q{i}=quantile(C{i});      % compute the quantiles by column
  C{i}(C{i}>q{i})=nan;      % fixup the cell array
end

NB: the dereferencing of the cell array C with "the curlies" {} to obtain the underlying array, then the subscripting expression inside "regular" parentheses () to address elements within the array.

NB Second: the above overwrites the original cell array; if need to keep the original data, then use another variable to make a copy first before running the loop and use it instead is simplest and fastest (although the original also overwrites so this may be moot point).

See the documentation on cell arrays for the details on the syntax for using and addressing cell arrays.

Answer 2

[Voss]

Wrap that loop in another loop:

Cell = { rand(100,6); rand(100,6); rand(100,6); rand(100,6) };

for k = 1:numel(Cell)
    x = Cell{k};
    for j=1:6
        q=quantile(x(:,j),0.97);
        id=find(x(:,j)>(q));
        x(id,j)=NaN;
    end
    Cell{k} = x; % store the result back in Cell
end

By the way, if something is scalar, then it can't be 100x6; scalar means 1x1.

[Ziad Sari El Dine]

Hi, thanks for the note, but how would this be applied to the cell array? The result i need is that for each cell in the cell array, the values of the numerical array that exceed the 0.97 quantile, in each column seperately, be replaced by nan.

[dpb]

Wrong approach -- see <Answer_993850> below for "the MATLAB way"...

[Ziad Sari El Dine]

Hi, i believe i need the loop since i need the quantile for each column separately in the array in my application.

[Voss]

@Ziad Sari El Dine: I have added a line that stores each x back in Cell, so that Cell has the updated values.

[dpb]

"...i, i believe i need the loop since i need the quantile for each column separately..."

NO!!! READ THE DOCUMENTATION for quantile -- as I point out below, it is already vectorized. Use MATLAB as it is intended to be; that's why it's named MATrixLABoratory.

Did you even test it to see???

>> X = normrnd(0,1,4,6)
X =
   -1.1665    0.9708    0.0230   -0.1692   -0.7118    2.9043
   -0.4522    1.0586   -0.9011    0.7256    1.2125   -0.1707
   -0.2237   -1.5278    1.4810    0.0304    0.3300    0.1707
    0.5736   -0.3764    0.7949   -1.1602   -0.1762    0.0304
>> q=quantile(X,0.7)
q =
    0.0155    0.9972    1.0007    0.2390    0.5947    0.9908
>> X(X>quantile(X,0.7))=nan
X =
   -1.1665    0.9708    0.0230   -0.1692   -0.7118       NaN
   -0.4522       NaN   -0.9011       NaN       NaN   -0.1707
   -0.2237   -1.5278       NaN    0.0304    0.3300    0.1707
       NaN   -0.3764    0.7949   -1.1602   -0.1762    0.0304
>> 

QED. No loop in sight; it's already done for you inside quantile

[Voss]

@Ziad Sari El Dine: @dpb makes a good point about replacing your original loop with a single call to quantile, but if you do that, you better use the dim argument in quantile in case you run into an x with one row:

% more than one row:
X = normrnd(0,1,4,6);
Q = quantile(X,0.7)     % without dim=1

Q = 1×6

0.2556 -0.3720 0.6476 0.7030 1.2387 0.2020

Q = quantile(X,0.7,1)   % with dim=1 (same as without)

Q = 1×6

0.2556 -0.3720 0.6476 0.7030 1.2387 0.2020

% one row:
X = normrnd(0,1,1,6);
Q = quantile(X,0.7)     % without dim=1 ("wrong")

Q = 0.4386

Q = quantile(X,0.7,1)   % with dim=1    ("right")

Q = 1×6

-0.7910 0.1657 -0.1221 0.5556 1.5881 -0.4429

[Ziad Sari El Dine]

Thanks! I thought it did not work for me after checking my real data (1,256,585x6). Turns out it did (my bad) and thanks for your explanation and example. Also, thanks for both answers.

[dpb]

I'd argue the first is the correct solution for a single row -- or that there being only one row violates the user contract for the original code... :)

It probably can't hurt in OP's case to use the dimension, but we don't know that for sure and it also probably is not needed because there can't ever be the case in practice.

Would need more info to know for certain the intent and/or range of possible input array sizes.

[Voss]

I guess I should've said: to be consistent with the original (loop) code, set the dim argument to 1.

[dpb]

Understand/understood -- I was just being me! :)

It would be consistent with the original coding as done; agreed; I had simply observed that from that code it seemed the case of a single row would have no meaning and so not ever arise in practice but, logically, if it were the case then the only way in which any quantile would have any chance of meaning anything would be over the vector...but it is probably also true that the various columns represent different variables and so mixing across them is also not a reasonable thing to do in practice...

My coding habits would be to have uncovered there being insufficient data on entry into the routine and aborted then before ever getting to this code section.

But, if the intent is only to duplicate the original code logic and make no changes even to edge cases, then the optional argument would do that, yes.