Borrar filtros
Borrar filtros

fprintf for ~million row string matrix

2 visualizaciones (últimos 30 días)
Erik Johannes Loo
Erik Johannes Loo el 26 de Jul. de 2022
Editada: Jon el 26 de Jul. de 2022
Hi,
I'm trying to write data to csv files, but using fprintf (or the way I am using it) seems to be extremely slow
The data is formatted in columns containing fields such as: {ID (string), date (string), numeric data, flag (string), ...}.
I get the data as a cell array, with each cell having a column of data.
What I've done is cast everything into a string and take the transpose of the data (it seems MATLAB does things in a column-major order).
Also note I've added a bunch of commas without data inbetween. That's the expected format of the data, so that has to stay.
fid = fopen(fullfile(output, replace(file.name, '.dat', '.csv')), 'w');
fprintf(fid, [repmat('%s,', 1, nCols), repmat(',', 1, 86-nCols), '\n'], T);
fclose(fid);
It looks as if saving a single file takes several minutes (haven't managed to save even one file yet - and I need to do this for 365 files).
Is there a way to speed up this process?
Kind Regards,
Erik

Respuestas (1)

Jon
Jon el 26 de Jul. de 2022
You could try writecell and see if that is more efficient.
  2 comentarios
Erik Johannes Loo
Erik Johannes Loo el 26 de Jul. de 2022
Thanks. I don't have writecell on my version (R2018b) but I tried the code on a different computer (with R2020b) and my original approach (fprintf) worked much faster without modification.
I'm assuming fprintf was improved from R2018b to R2020b which has led to dramatic performance improvements.
Jon
Jon el 26 de Jul. de 2022
Editada: Jon el 26 de Jul. de 2022
Or maybe the computer with R2020b was just faster? Also I have found that MATLAB is sometimes very slow writing to network drives. If possible write it to a file on your local hard drive.

Iniciar sesión para comentar.

Productos


Versión

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by