Faster way of loading data than loading mat file with array? / Faster way than logical indexing or find?

Hello everybody,

I need to load a really big amount of data into matlab. I could speed it up by moving from a table to an array and by using v6 to save it. But still it takes almost 50% of the time of the program. Is there an even faster method?

After that, I need to find certain values in this array. This takes about 40% of the time. Is there a faster emthod than logical indexing?

[Star Strider]

Depending on the options available to you, installing a solid-state drive (SSD) in your computer or as an external drive would be the best option. Lacking that (and depending on how large the file is), saving it to an SD card (that currently can have reasonably large capacities) would be something to experiment with.

[Guillaume]

What format is the data stored as? text file? excel? If you mention -v6 does that mean it's stored as a (binary) mat file? If it's a mat file, I doubt that the storage method has much of an impact.

"I need to find certain values in this array" This is too vague for us to answer. Find according to what rule/criteria? To do what with the values afterward?

[Gerrit]

You are right, but I need to speed it up on every computer.

[Gerrit]

Yeah, it's stored as a binary mat file. At first, I used tables for my program. There I had a loading time of about 2.4s. Now I use arrays and the loading time decreased to 1.1s. Thats why I ask.

This is what I meant by I need to find certain values. For my 4D Interpolation afterwards, I only need values in a specific area.

new_upper_airfoil_all=new_upper_airfoil_all(new_upper_airfoil_all(:,4) < (Ma+delta_ini) & new_upper_airfoil_all(:,4) > (Ma-delta_ini) & new_upper_airfoil_all(:,5) < (C_l+delta_ini) & new_upper_airfoil_all(:,5) > (C_l-delta_ini),2:7);
new_lower_airfoil_all=new_lower_airfoil_all(new_lower_airfoil_all(:,4) < (Ma+delta_ini) & new_lower_airfoil_all(:,4) > (Ma-delta_ini) & new_lower_airfoil_all(:,5) < (C_l+delta_ini) & new_lower_airfoil_all(:,5) > (C_l-delta_ini),2:7);

[Guillaume]

If you're using plain arrays, that's as fast as you'll get with matlab. Similarly, your current indexing is the best you can do.

On the other hand, the problem may come from your algorithm itself, which may be (very) inefficient regardless of the storage method. Do you know the complexity of your algorithm? If it's exponential or worse, then you may need to find something more efficient.

[Gerrit]

Ok, thank you. What do you mean by algorithm? The rest of my code? Yeah I still try to vectorize it, but I can't do it till now. But this would only get me some millisecounds. But this programm needs to run for 7500 cases, so it would add up.

[Walter Roberson]

If you have plain numeric arrays then the fastest way to load them is to use fread() from a binary file.

[Walter Roberson]

The data that you are currently doing the searching for ranges: is that data in sorted order? Because if it is, then binary search can be used to find the boundary in log2 time instead of linear time. I seem to recall that there is a File Exchange contribution to do binary search at the mex level.

[James Tursa]

A mex routine could avoid extracting the temporary arrays such as new_upper_airfoil_all(:,5) etc, but I'm not sure how much time that would save overall. Is the data in the arrays sorted? If so, a mex routine could use a binary search to find the edges and speed up the sub-array extraction.

[Gerrit]

I use load at the moment, I will try fread tomorrow. The data is sorted by the AeroDataId. For every AerodataId there are 70 points in the array. Is plain numeric array something else than a normal array?

[Walter Roberson]

In MATLAB, many kinds of arrays are "normal", including cell arrays. "plain numeric array" is emphasizing on a simple real-valued double array especially one that is vector or 2d.

[Guillaume]

If the data is saved in a mat file, fread won't help (unless you're planning to write your own mat file parser).

[Gerrit]

Yeah, I just tested that. I think, the way I have it, is the best way to do it. Now I try to focus on vectorizing my loops.

If you want, you can take a look at my question. Link: https://de.mathworks.com/matlabcentral/answers/501586-can-i-vectorize-one-of-the-loops