Can't open anymore files with fopen above some threshold
1 visualización (últimos 30 días)
Mostrar comentarios más antiguos
Hi guys!
I am trying to open all files contained in a folder, with something like
folder = 'some/folder/path'
files = dir(folder);
for i = 1:length(files)
filenames{i} = files(i).name;
output(i).name = filenames{i};
output(i).file = sprintf('%s/%s',folder,filenames{i});
output(i).fid = fopen(output(i).file);
if (output(i).fid == -1)
output(i).status = 'not read';
else
output(i).status = 'read';
end
end
The folder contains few thousands of very light text files.
The problem is that, after having read some number of this files (in my case always 1020) the files in the list are not read anymore. So I get something like
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/982620/image.png)
Now, the problem is not the limit of files which can't be opened by MATLAB, because for my system it is apparently 10240.
If, in this situation, I try to open again manuallt that qc.dat, for example, I get -1 and 'Operation not permitted'. BUT if I try to read again manually that q_plates.dat, i.e. the last file which was read, it is read again with new fid 1021.
And, it seems also not a problem related with the files. If i close everything and only after that I try to manually read that 'qc.dat' and so on, I succeed.
What's the problem? The only thing I can think is a limit of the number of files opened in the same moment which sit in the same folder... is it the case?
I use MATLAB R2021b on a MacBook 10.15.7.
2 comentarios
Rik
el 29 de Abr. de 2022
I don't have a solution for your problem, so let me try to figure out the deeper motive here: why exactly do you want to open that many files at once? I try to keep my file interactions as short-term as possible, so I hardly ever have more than 1 active FID.
Steven Lord
el 29 de Abr. de 2022
Having two files open simultaneously (one for reading, one for writing) is a common pattern. But I agree with @Rik; why do you need thousands of files open at once?
Open the file, do whatever processing you need to do with it, then close it. Repeat until you get to the end of the list of files. [Depending on the details of what processing you're performing, you may even be able to parallelize it using a parfor with this approach if you have Parallel Computing Toolbox installed.]
Respuestas (2)
Walter Roberson
el 29 de Abr. de 2022
There is a per-process limit on the number of open files,
!ulimit -n
The limit for the operating system running MATLAB Answers is 65535, but the limit on my Mac is 10240 by default.
If you wanted to change the limit for your system, see the "Sierra and newer systems" description at https://wilsonmar.github.io/maximum-limits/
For systems before Sierra, see https://docs.riak.com/riak/kv/latest/using/performance/open-files-limit/index.html
Image Analyst
el 29 de Abr. de 2022
Why do you need to have them ALL open simultaneously? What I'd do is open one, get it's contents and do something with it, then close it and repeat for all the other files. So you'd have only one oepn at a time.
0 comentarios
Ver también
Categorías
Más información sobre Startup and Shutdown en Help Center y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!