What is the fastest way to swap large blocks of data between RAM and disk storage?

Hi all. I need to work with 4 large matrix variables repeatedly in sequence. But only one of them can fit in RAM at a time. (70 GB each, 128 GB of RAM). The operations are time-sensitive, so I need to be able to load one block of data, do some computations, release the memory, load the next block...and so on, as fast as possible. I have been looking at data store & tall arrays -- I could concatenate the 4 into one tall array -- but that seems intended for operations that have to span more rows than memory can hold, like taking a max across all of them. Here I just need access to one contiguous block that -will- all fit in memory, but to cycle through multiple of them. What is the preferred approach? I am using the parallel processing toolbox, but am not sure how to apply parallelization to this issue in a way that doesn't just add more overhead. Currently I save data into uncompressed v7.3 .mat files, but I can convert to whatever helps. Thanks in advance for your insights!

Answer 1

[dpb]

The fastest would be straight binary transfer with fwrite and fread

function n=writeVariable(x,where)
  fid=fopen(where,'w');
  n=fwrite(fid,x);
  fclose(fid)
end

function x=readVariable(where)
  fid=fopen(where,'r');
  x=fread(fid);
  fclose(fid)
end

The variable will be on disk in sequential order by MATLAB internal storage order of by column so on reading will return a 1D vector, but reshape() is quick. For convenience/factoring, putting the shape in the file is the user-friendly way.

If it isn't mandatory for precision, you can cut down the size of the data by 2 if just save single instead of double precision. This may save some i/o time, might not help or even might hurt compute time.

[Patrick Ramsey]

Thanks, this makes sense.