How to deal with a single tsv-file whose size is out of the memory?

2 visualizaciones (últimos 30 días)
ziyou teng
ziyou teng el 17 de Oct. de 2020
Comentada: ziyou teng el 10 de Nov. de 2020
Hi, guys.
I have a tsv file whose size is 20GB while my pc's memory is only 16GB.
When I read the file, it always shows errors.
I tried tall array as follows but fails due to tsv file cannot be recognized.
So I am looking for advice.
ds = tabularTextDatastore('D:\database1\scipatlinkage\paperauthoridaffiliationname.tsv');
ds.TreatAsMissing = 'NA';
ds.SelectedVariableNames = {'paperid','authorid','affiliationame'};
ds.SelectedFormats(2:3) = {'%s','%s'};
pre = preview(ds)
My matlab is R2020a
  1 comentario
Athul Prakash
Athul Prakash el 24 de Oct. de 2020
What is the exact error you're getting? If it s a File Not Found issue, you may use the 'isfile' function to confirm the existence of your file before running the script. See this doc for 'isfile': https://in.mathworks.com/help/matlab/ref/isfile.html

Iniciar sesión para comentar.

Respuestas (1)

Athul Prakash
Athul Prakash el 24 de Oct. de 2020
I have noticed that you have not created the tall array in the code attached. Perhaps . . .
t = tall(ds);
might be missing.
.As alternatives to tall array, you may try using this datastore with 'mapreduce'. See this doc:
Alternatively, you may also write your own code to 'read' from the datastore iteratively and process the data in chunks that can fit on your RAM.
Hope it Helps!
  1 comentario
ziyou teng
ziyou teng el 10 de Nov. de 2020
Thank you!It‘s really helpful. My problem is the file is too large that goes out of my pc's memory. Your answers have gaven me useful hints on that. Thanks again!

Iniciar sesión para comentar.

Categorías

Más información sobre Large Files and Big Data en Help Center y File Exchange.

Etiquetas

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by