MATLAB Answers

Improve speed reading in a .dat file

20 views (last 30 days)
Aaron Crawford
Aaron Crawford on 4 Mar 2021
Edited: Stephen Cobeldick on 4 Mar 2021
Im attempting to process some rather heafty .dat files with 500k+ lines. The bulk of my code runtime is being taken up by two functions
fgetl and str2double
Does anyone know a faster way to read in a dat file or functions that run faster
  2 Comments
Aaron Crawford
Aaron Crawford on 4 Mar 2021
This is essentially what the file looks like.
The example file is included as a .txt however they are actually .dat files.
TITLE="Solution flow field"
VARIABLES="X" "Y" "Z" "RHO" "RHOU" "RHOV" "RHOW" "RHOE" "RHOK" "RHOOMEGA" "MUT"
ZONE T="10005_16", I= 1 , J= 195 , K= 265 ,
DATAPACKING=BLOCK
VARLOCATION=([4-11]=CELLCENTERED)
0.42820862965225337
0.42820862965225337
0.42820862965225337
0.42820862965225337
0.42820862965225343
0.42820862965225337
It continues on with a new number every line untiil the file ends. After the first 5 lines of text there is no more text. I honestly dont even need the test at the top.
Thank you for the input. I will definitley look into the fscanf function.

Sign in to comment.

Answers (2)

Image Analyst
Image Analyst on 4 Mar 2021
You could try fileread() to read in the whole file into one variable in one shot, then go through it parsing it.

Stephen Cobeldick
Stephen Cobeldick on 4 Mar 2021
Edited: Stephen Cobeldick on 4 Mar 2021
Some version of this is probably the fastest you can get:
[fid,msg] = fopen('example.txt','rt');
assert(fid>=3,msg)
for k = 1:5 % 5 header lines to ignore
fgetl(fid);
end
vec = fscanf(fid,'%f'); % fast!
fclose(fid);
fprintf('%.17f\n',vec)
0.42820862965225337 0.42820862965225337 0.42820862965225337 0.42820862965225337 0.42820862965225343

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by