load very large data file in 32bit matlab 2013 and windows 7 32bit

Hi,

i have a very large data file (.txt), 15 coloumns and several rows. (600MB). figures are plotted successfully but have to enlarge the cache memory. is there any way to transform this file to other format so that it can be read quickly and to reduce size . e.g .mat etc,

thanks

Answer 1

[Walter Roberson]

Generally speaking, you can use

fid_in = fopen('YourInput.txt', 'rt');
fid_out = fopen('BinaryVersionOfTxt.dat', 'w');   %no 't'
while true
  inputline = fgetl(fid_in);
  if ~ischar(input_line); break; end   %end of file
  input_num = sscanf(inputline, '%f');
  fwrite(fid_out, input_num(:), 'double');
end
fclose(fid_in);
fclose(fid_out);

BinaryVersionOfTxt.dat will now be a binary data file of doubles, with the values saved reading across in row order from the input file. This code will handle any number of inputs on each line and does not put in any kind of end-of-row marker, so be careful about interpreting the file.

If the values are integer then you can reduce storage by changing the program slightly.

You would do the above conversion once and then you can read the binary file any number of times.