How do I process big datasets ??

I have data of on file per day , that goes for each day of the month and then each month of the year. I now process daily files separately for daily results. How can I do this for complete analysis of full one year file. My file is like this YYYYMMDD.loc ; this means YYYY- Years, MM- months, DD-day. What if I want to take data of particular month and analyse ??

Answer 1

[Ced]

Hi

Do you now have each day in a separate file, or everything in one file?

Personally, if you want to evaluate different sections of time, I would keep each day in a separate file. Then, you can always just read the files you need.

E.g. Let's say you want to read the months of June and July of years 2013-2015:

(I am assuming that you want to read the same days and months for each year here)

years = 2013:2015;
months = 6:7;
days = [ 1 1 ; 30 31 ]; % first and last day to read, one month per column

N_years = length(years);
N_months = length(months);
N_days = sum(diff(days)+1); % Note: 1 is added to EACH month
N_files = N_years*N_months*N_days;
filenames = cell(N_files,1);

current_index = 1; % makes indexing easier
for i = 1:N_years
  for j = 1:N_months
    next_index = current_index + days(2,j)-days(1,j);
    filenames(current_index:next_index,1) = ...
       arrayfun(@(x)sprintf('%i%02i%02i.loc',years(i),months(j),x), days(1,j):days(2,j), 'UniformOutput',0);
    current_index = next_index+1;
  end
end

Then, you can load all the data you need

for i = 1:N_files
  % load data, process, combine.... whatever you need
end

Cheers

[Add]

Where does it load the data ?

[Walter Roberson]

As Ced says,

Then, you can load all the data you need

for i = 1:N_files
  % load data, process, combine.... whatever you need
end

http://matlab.wikia.com/wiki/FAQ#How_can_I_process_a_sequence_of_files.3F