I'm curious about methods for storing extremely large data.

I am working with complex double data of size approximately 5000 x 100 x 100 x 4000. Not only does this data take a long time to generate, but the time taken to save it is also extremely long, which is causing issues. Using the -v7.3 option, saving the data takes more than a day, and although using parfor reduces the time somewhat, it is still far too slow. I considered using C++ code to manually assign threads for faster saving instead of using parfor, but the compiled mex file causes MATLAB to crash with an error sound as soon as it runs.

Could someone please share a method for quickly saving extremely large data like mine?"

Additionally, I have tried saving with an h5 file, and although it is faster than using v7.3, it is still slow...

[Stephen23]

https://www.mathworks.com/help/matlab/large-files-and-big-data.html

[Walter Roberson]

One thing to keep in mind with saving files to hard drive, is that the optimum limits are typically at two to three file write requests per controller channel; and it is not common for there to be more than 2 controller channels per controller. (You might get in more simultaneous write requests if the block sizes involved are small.)

If each write request is a file being written into, the implication is that you need to plan ahead to split the files across different drives. Or possibly to use a RAID device.

These rules of thumb possibly need to be modified for SSD; I do not know how channel management is done for SSD.

Answer 1

[Andreas Goser]

The resource Stephen shared is very good. Besides that, I like to share my experience having looked at many real-life applications: Often, much of the data is not needed for the specific research or development goal. Example is having DOUBLE measurements with a 1ms rate, but looking into this closer, SINGLEs every 10ms is good enough. If you find this obvious guidance based on your experience: If have seen this with beginners as well as highly experienced users so I "dare" to mention this.