Reading huge number of files quickly (I/O)

For analysis of EOD data for stocks, I need to read 3650 .csv files, but using a for loop takes around 30 seconds for 15 files, which would mean around 2 hours just to read data. Is there any way I can do it faster?

Answer 1

[Walter Roberson]

Are you currently reading using csvread(), or import(), or load() ?

If each file has exactly the same structure (same number of fields, some number of headers, same fields are character vs numeric) then you might be able to do better by using textscan()

If you are currently using xlsread() on MS Windows then you can almost certainly do better than you are doing now (but textscan() might be better still). xlsread() opens and closes Excel each time; Image Analyst has posted code several times showing how to open Excel just once and do several operations with it.

[Vishnu Swaroop]

yeah, I'm using xls read with a for loop, Will look into your suggestions. Thanks!