Python-like generator for reading large CSV file?

4 visualizaciones (últimos 30 días)
roudan el 16 de Abr. de 2018
Respondida: roudan el 16 de Abr. de 2018
I have a large csv file, typically I just read everything into a array like this
num =csvread(filename,2,2)
if csv file is big, the num array will be big as well and sometime it blowout my memory.
In python, there is a way to read the file but without holding the data in the memory. the methods use generator object. For example,
Base case(w/o using generator object):
value=[len(x) for x in open('filename.tx')] this case will hold the data in the memory
another case(use generator object:
it=(len(x) for x in open('filename.tx'))
the it is a generator object and it only hold the memory size for the generator expression without holding memory for all the data, later, we just extract the data by using next(it).
I am wondering if Matlab has similar generator object as Python when reading csv file to save the memory? Thanks

Respuesta aceptada

Guillaume el 16 de Abr. de 2018

Más respuestas (1)

roudan el 16 de Abr. de 2018
Thank you, Guillaume. That is awesome! I appreciate it.


Más información sobre Standard File Formats en Help Center y File Exchange.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by