Alternatives for dir command

25 visualizaciones (últimos 30 días)
Linus Dock
Linus Dock el 10 de Oct. de 2021
Comentada: Linus Dock el 11 de Oct. de 2021
Hello dear masterminds!
I have a working code for collecting strings from several txt.files (appr. 53000 files) in a network folder. I have one problem though. It is quite time consuming (17-19 seconds) using the dir command to identify the filenames. Is there a better way of doing this? I will attach a part of the code and some of the filenames.
Thank you!
addpath \\winfs\data\prod\maprod\arkivering\TAF;
DS = StartDTstr;
DE = StopDTstr;
TI = duration(timein,'InputFormat','hh:mm');
TO = duration(timeout,'InputFormat','hh:mm');
P = '\\winfs\data\prod\maprod\arkivering\TAF';
S = dir(fullfile(P,'*.txt'));
D = regexp({S.name},'\d{10}','match','once');
T = datetime(D, 'InputFormat','yyMMddHHmm');
X = isbetween(T,DS,DE);
tod = timeofday(T);
Y = tod>=TI & tod<=TO;
Z = S(X&Y);
Z.name;
numfiles = length(Z(:,1)); %create empty cell
for h=1:numfiles
filename=Z(h).name;
fileID=fopen(filename); %open filename to create fileID
Data{h}=textscan(fileID,'%s','delimiter','=','headerlines',1);
fclose(fileID); %close fileID
end
rmpath \\winfs\data\prod\maprod\arkivering\TAF;
Utcell = cell(1,length(Data)); %output data of all TAF
  6 comentarios
Stephen23
Stephen23 el 11 de Oct. de 2021
"Is it the network and the size of that folder that's the issue then?"
If the files are being accessed over a network then that is likely to be a bottleneck.
But only measuring the properties of your storage+network+local machine would answer that question.
Linus Dock
Linus Dock el 11 de Oct. de 2021
Ok thanks!
Really great to have your help!
/Linus

Iniciar sesión para comentar.

Respuesta aceptada

Jan
Jan el 10 de Oct. de 2021
Editada: Jan el 10 de Oct. de 2021
addpath \\winfs\data\prod\maprod\arkivering\TAF;
Why do you append this folder to Matlab's path? Are the files stored in this folder or some required Matlab functions? If this folder does not contain Matlab functions, there is no reason to append it to the path. Simply omit the addpath/rmpath commands. See below.
What is the purpose of this:
DS = StartDTstr;
DE = StopDTstr;
TI = duration(timein,'InputFormat','hh:mm');
TO = duration(timeout,'InputFormat','hh:mm');
Are this variables or functions?
What exactly takes 17-19 seconds?
Is Data pre-allocated? If the final size is known, allocate it instead of letting the array grow iteratively:
numfiles = size(Z, 1); % Better than: length(Z(:,1));
Data = cell(1, numfiles);
for h=1:numfiles
filename = fullfile(P, Z(h).name); % Absolute path
fileID = fopen(filename);
Data{h} = textscan(fileID, '%s', 'delimiter', '=', 'headerlines', 1);
fclose(fileID);
end
It is more efficient to use the absolute path name because with fopen('file.txt') Matlab's complete path is searched.
Reading 53'000 files over a potentially slow network connection takes time. How large are the files? Maybe the runtime is limited by the network connection. Then there is no way to accelerate it magically.
  4 comentarios
Stephen23
Stephen23 el 10 de Oct. de 2021
"Is this a wrong way of doing it"
Yes, that is the wrong way of doing it.
"Could you give me an example?"
Jan's answer already shows you an example of how to use an absolute filename.
Linus Dock
Linus Dock el 10 de Oct. de 2021
I see that now, sorry :)
/Linus

Iniciar sesión para comentar.

Más respuestas (0)

Categorías

Más información sobre Debugging and Analysis en Help Center y File Exchange.

Etiquetas

Productos


Versión

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by