How can I get the indices of values meeting a condition in a tall array?

Hi everybody,

I have a tall array containing voltage data. I want to thresholding the trace to receive the signals. So with thresh is the threshold, data is the tall double array (Mx1), I perform

data = abs(data);
loc = tall(data > thresh);

to get loc, a tall logical, containing ones for samples above threshold and zeros for samples below, after evaluating.

I just want to have the crossings, so I apply

location = diff(loc,1,1);

So locations where the signal crosses the threshold are 1, all others are 0 or -1;

But now I need a vector listing the position values (indices), and i use

starts = find(gather(location == 1));

Is there any possibility to realize that without gather to leave the array unevaluated?

Many thanks in advance!!!!

Eva

[madhan ravi]

Eva-Maria Weiss's answer moved here:

I found that in the MATLAB documentation:

https://de.mathworks.com/help/matlab/import_export/index-and-view-tall-array-elements.html

"The find function locates nonzero elements in tall column vectors, and can be useful to generate a vector of indices for elements that meet particular conditions. For example, k = find(X<0) returns the linear indices for all negative elements in X."

So I tried to convert a tall variable (location) of type tall double (unevaluated) to a Mx1 tall double (unevaluated) - to get a column vector,

location = location(:,1);

but still I got the error message

"undefined function 'find' for input arguments of type'tall' "

Isn't oit a column vector? How can I convert my variable to a column vector thats's a real one?

I really appreciate any help or hint!

Thank you very much in advance!

Eva

-----

Maybe find is not working on tall since I use 2017b?

So far I come up with some proceedings:

I aim to solve the issue by logical indexing;

so first I create this tall lgocal vector containing the crossings

loc = data(data > thresh); 
location = diff(loc,1,1);
location = location == 1;

Then I creat a tall column vector containing indices:

index = location;
index(:,1) = tall([1:1:numbSamples -1]');

Then I select indices meeting the condition

starts = index(location,1);
starts(starts == 0) = [];

To see if that approach is working, I want evaluate the result by using

starts(starts == 0) = [];

Evaluating starts properly, but after finishing pass 1 of 2, I was again left with an error message saying:

Incompatible tall array arguments. The tall arrays must be based on the same datastore and

be indexed in the first dimension using the same logical index vector.

Learn more about errors encountered during GATHER.

How can I overcome that? Could anyone please help, meanwhile I really feel like circling around!

Answer 1

[Guillaume]

Maybe find is not working on tall since I use 2017b

Indeed, support for using find on tall arrays was added in R2018a.

As for your error with your workaround, there's a big difference between

[1:1:numbSamples -1]

and

[1:1:numbSamples-1]  
%or
[1:1:numbSamples - 1]

The former is equivalent to

[1:1:numbSamples, -1]  %concatenation! not subtraction

and will therefore have numbSamples + 1 elements when you meant to have numbSamples - 1 elements. So, of course, the length doesn't match that of your logical array and you get an error.

To avoid this sort of issue:

• be consistent with your spacing. Either no space at all on both sides of the operator, or spaces on both sides.
• Don't use [] when you don't mean concatenation. If you want to transpose a vector surround it with (). This is (negligibly) faster to process as well.

So:

index(:,1) = tall((1:numbSamples-1)');

One potential problem with your workaround is that if numbSamples is sufficiently large, the temporary non-tall array 1:numbSamples-1 may not fit in memory.

[Eva-Maria Weiss]

Thank you Guillaume for your answer. You are so right, regarding the dimension thing. OMG, I am a bit embarrassed, since it's something I really should know. Ok, but I'm still learning...

Unfortunatelly correcing the dimension doesn't fix the original prolem, I still receive the same error message. But I will check now all sizes and will go through the problem again step by step keeping your last note in mind! I'll let you know when I've done that. Thank you again.

[Guillaume]

Indeed, I see the same issue in the latest version of matlab.

I don't know enough about tall arrays to explain why it errors and the error would appear to contradict the documentation so I'll raise a bug report with Mathworks to get an explanation,

Digging a bit in the tall array code, the incompatibitly is caused because location stems from a Mx1 tall array (even thought it is now (M-1)x1) whereas index starts as as (M-1)x1 array. That's enough to cause an error, but I don't know why.

So, a workaround is to create |index as a Mx1 tall array and then reduce ot to (M-1)x1. The following works:

assert(numbSamples <= 134217728, 'flintmax overflow, see explanation below');
index = diff(tall(cumsum(0:numbSamples-1)'), 1, 1);
starts = index(location);

The major limitation with that is that if numbSamples is greater than 134,217,728, the cumulative sum will go over flintmax and you won't be storing exact numbers anymore.

If numbSamples is greater than that, you can use 64-bit integers instead to push the limit to 6,074,000,999:

assert(numbSamples <= 6074000999, 'this is going to use too much memory! (and it will overflow uint64)'
index = diff(tall(cumsum(uint64(0:0:numbSamples-1)'), 1, 1);
starts = index(location);

Since it would take about 45 GB of memory to store the temporary non-tall array with 6,074,000,999 elements, I don't think that limit is a problem.

[Eva-Maria Weiss]

Hi Guillaume,

thank you a lot for your answer. You really provided a sophisticated solution for reducing the Mx1 vector to a M-1x1;

Unfortunately it still doesn't solve the issue. I'm starting to get really desparated. I didn't thought that's such a hugh problem to get the indices from a tall array.

Maybe it has something to do with the datastore backing the tall arrays? Since index is not derived from the datastore like location, perhapts this might cause the problems? According to the error message: The tall arrays must be based on the same datastore

Everything is fine when I evaluate location by applying gather; But this takes time, I really would like to avoid that.

Thank's again Guillaum, and at least, I've learned some things. That's always a good thing.

[Guillaume]

Indeed, I initially tried with in-memory tall arrays and my workaround worked fine for that. Having no tried with a datastore backed tall array, the workaround doesn't work.

There appears to be some restrictions on logical indexing that are not documented. Unfortunately, this is beyond my knowledge of tall arrays, so I don't think I can help further. If you can, I suggest you raise a support request with mathworks (or upgrade to 2018a so you can use find).

Note that I've raised my own bug report to at least get the documentation explaining why this does not work.

[Eva-Maria Weiss]

Well, my next try is to expand the datastore with this index vector. It's far away from being elegant, since you have to create tall arrays, then write them back to disk and then again create new tallDatastores and tall arrays.

And 'write' takes a time, too. But my guess is, that I will just do it one time per experiment. Evaluation 'location' will be for each electrode...380 times per experiment.

But still I don't know if it's working at all. So lets see, I will report.

And yes, I guess I should also contact the matlab support. Version 2018 would be great, but I'm lmited to what I can get from university :).

Thank you again for your help and ideas!

Eva