reading multiple text file from two different folders using nested for loop

3 visualizaciones (últimos 30 días)
i am using the following lines to read multiple text files and do some comparison between them. Baiscally the first loop will read text files of certain format (we have 30 files of this type). the second loop will read text files of another format (we have 33000 of them) and we do some comparison between them.
I have my code working and it is fine but it is extremely slow because of the nested loop, and its important to mention that i dont really need these loop for anything else, only for reading the files from folders.
So is there a more effecient way of doing this.
for n = 1 :nfiles2
fprintf('.......... File - %d of %d\n',n, nfiles2)
fullFileName2 = fullnames2{n};
t2 = readtable(fullFileName2); %t2 stores the ionosonde profiles
for m = 1 :10
fprintf('.......... insideFile - %d of %d\n',m, 10)
fullFileName1 = fullnames1{m};
t1 = readtable(fullFileName1); %t1 stores the ISR profiles

Respuestas (1)

Konrad
Konrad el 31 de Mzo. de 2022
Hi Salma
It depends on what exactly you want to do with the data in the text files, but reading the file line by line seems to be considerably faster compared to readtable()
tic;
for k = 1:100
t = readtable('test.csv');
end
toc
Elapsed time is 1.449934 seconds.
tic;
for k = 1:100
fid = fopen('test.csv');
clear tl;
tl{1} = fgetl(fid);
while ischar(tl{end})
tl{end+1} = fgetl(fid);
end
fclose(fid);
end
toc
Elapsed time is 0.633903 seconds.
Of course readtable() converts numbers to double while fgetl() returns char and if you have to do type conversion the speed gain might disappear. But I think its worth trying.
Best, Konrad
  2 comentarios
Salma fathi
Salma fathi el 31 de Mzo. de 2022
thank you for your reply, I have tried what you suggested, it is much more faster than the way I used to do it.
but I get the content of the textfiles in the following form, each row of the file is stored in a cell
while we want it to be in the following form
Would there be a way to do that?
Konrad
Konrad el 31 de Mzo. de 2022
Editada: Konrad el 31 de Mzo. de 2022
Yes, thats what I ment with type-conversion ;)
I assume your data is comma delimited.
get the header in the first row:
fid = fopen('test.csv');
tline = fgetl(fid);
header = strsplit(tline,','); % <- the variable names as cellstring
If the data is all numeric (with variable names in the first row) you can do the following:
tic;
data = csvread('test.csv',1,0);
toc
Elapsed time is 4.558796 seconds.
Else: you can read the text data, store it in a cellstring and convert the numeric part (you will have to know which columns you need). This takes much longer due to the type conversion (and the growing array, which could be avoided by preallocating), but could be made faster if you only need a small fraction of the data for your comparison. But I don't know what you want to compare, so here's how you get everything from the test.csv:
tic;
dataCellstr = cell(0);
selectedCols = 1:3;
tline = fgetl(fid);
while ischar(tline)
cols = strsplit(tline,',');
dataCellstr(end+1,:) = cols(selectedCols);
tline = fgetl(fid);
end
data2 = str2double(dataCellstr);
fclose(fid);
toc
Elapsed time is 15.281463 seconds.

Iniciar sesión para comentar.

Categorías

Más información sobre String Parsing en Help Center y File Exchange.

Productos


Versión

R2021b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by