Matlab freeze when fprintf to file

Hi

I have a large amount of data, many years. It is given in daily files.

For reasons irrelevant for the question I have to combine these files to yearly data files ~ 3Gb.

The daily files are of .cdf format and contain more variables than needed and also lack some that has to be derived.

The following shows an example that makes my computer freeze for around 5 min. But that data datafile is complete with seconds.

fid = fopen('test.m', 'w');
data = randi(5, 40000, 10);
for i=1:2 %5
    fprintf(fid, '%d %d %d %d %d %d %d %d %d %d\n', data);
end
fprintf(fid, '%d %d %d %d %d %d %d %d %d %d\n', zeros(1, 10)); %Newly added line
fclose(fid)

Does anyone know why it freezes for so long when the files is written so fast?

Using Matlab R2018a on a remote desktop with Ubuntu OS

[Guillaume]

How do you know that the data is written fast if matlab is frozen?

[Michael Madelaire]

I have added a new line to the code above. It creates a row of zeros at the end of the file.

Seconds after running the code I can open the .m file with a text reader program. At the end of the file there is a row of zeros. This indicates that the file is complete.

Also the fact that it is able to open and does not send an error message saying "The file is open in another program".

But it takes MatLab a minute or two to become responsive. This increases with the increase of iterations.

[Stephen23]

@Michael Madelaire: this sounds similar to when many files are opened but not closed. When there are too many open file then MATLAB can become slow or unresponsive. Run fclose('all') and check your code carefully that it really closes all files properly.

How many files do you have open at once?

[Michael Madelaire]

@Stephen Cobeldick: Thanks for the comment. Although the code above is an example it is pretty much what is going on.

One file is opened.

Within the following loop data is loaded using cdfread:

data = cdfread(fullfile(base, filename{1}), ...
       'Variables', { 'Timestamp', ...
       'Latitude', 'Longitude', 'Radius', ...
       'F', 'B_VFM', 'q_NEC_CRF', ...
       'Flags_F', 'Flags_B', 'Flags_q', ...
       'dB_Sun' }, ...
       'ConvertEpochToDatenum', true, ... 
       'CombineRecords', true);

This data is written to the .m file using fprintf.

When the loop is complete the file is closed with fclose and a file is opened for the next year.

Answer 1

[OCDER]

Instead of 'test.m', write to 'test.txt' or something else. The fprintf is fast, but having a "test.m" file in the current directory seems to freeze Matlab.

fid = fopen('test.txt', 'w'); %Don't write to .m files, unless it's a matlab file type
data = randi(5, 40000, 10);
for i=1:2 %5
    fprintf(fid, '%d %d %d %d %d %d %d %d %d %d\n', data);
end
fprintf(fid, '%d %d %d %d %d %d %d %d %d %d\n', zeros(1, 10)); %Newly added line
fclose(fid)

My Matlab also freezes when there is a "test.m" file in the current directory that Matlab cannot determine if it's a script or function - it just says it's a "Matlab Code File". I'm guessing this step for determining what file type this "test.m" file is causes Matlab to freeze. I think this should go into the bug report for 2017 versions - not sure if 2018 version fixed this.

[Stephen23]

@OCDER: well spotted!

@Michael Madelaire: note that .m files are not intended for storing data, but for storing code. There are much better formats for data, such as binary files (e.g. .mat) and text files (e.g. .csv). MATLAB does all kinds of things with .m files, which are not very useful if the file is just being used to store rows of numeric data.

[Michael Madelaire]

@OCDER: You are absolutely correct. I actually did this on purpose, but did not realize the consequence.

I saved in .m format because when saving in .mat I would get the following error when using the load command:

Unable to read MAT-file C:\Users\s144117\Desktop\test_mat\test2.mat. Not a binary MAT-file. Try load
-ASCII to read as text.

But I am getting a bit tired of this issue and am just going to use textscan.

Thank you all for the many inputs!

[Stephen23]

@Michael Madelaire: please show us how you call save. Are any of the variables instances of custom classes?

[Michael Madelaire]

@Stephen Cobeldick: I do not call save. As shown in the example code

fid = fopen('test.mat');
data = randi(5, 40000, 10);
for i=1:2 %5
  fprintf(fid, '%d %d %d %d %d %d %d %d %d %d\n', data);
end
fprintf(fid, '%d %d %d %d %d %d %d %d %d %d\n', zeros(1, 10)); %Newly added line
fclose(fid);
data = load('test.mat');

[Stephen23]

"But I am getting a bit tired of this issue and am just going to use textscan."

Or you could just follow the instructions given in the error message.

Or use dlmread.

Both would be much simpler than using textscan.

Or use save and load (these are designed to be used together).

[OCDER]

do you want to save/load matlab variables?

data = randi(5, 40000, 10)
save('test.mat', 'data', '-v7.3', '-nocompression') %works for > 2GB files
S = load('test.mat', 'data') 
S.data = %your 'data'.

https://www.mathworks.com/help/matlab/ref/save.html