Download all hours of all days of the year from CMORPH

This script below wants to download all hours of all days of the year from 2000 to 2005, but the following error appears:

I want to download all time data for 2000 2005 from this link: Index of /data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km (noaa.gov)

clc
clearvars

% Download the CPC data used in the script below
which_years = (2000:2005); % years to download

httpsUrl = "https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km/";
% filename_tmp="temp.nc"; % temporary filename (for loop); data will be always stored under this temp name


%% main loop
mm = (1:12); % all monthes in year
% mm = [1 6]; % process only some specific monthes in year

for cc = 1:numel(which_years) % loop over years
    
    this_year = which_years(cc);

    for ci = 1:numel(mm) % loop over monthes

        days_per_month = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % compute how many days this month/year

        for ck = 1:days_per_month % loop over days

           ymd_str = datenum(this_year, mm(ck)+1, 1) - datenum(this_year, mm(ck), 1); % year month day string
           
           for ch = 1:ymd_str
               
               hora_str = strcat(sprintf('%04d',this_year), sprintf('%02d', mm(ci)), sprintf('%02d', ch));
               
               Url = strcat(httpsUrl, sprintf('%04d', this_year), "/", sprintf('%02d', mm(ci)),"/");
               filename = strcat("CMORPH_V1.0_ADJ_8km-30min_", hora_str ,".nc"); % create filename according to list above
               disp(strcat(filename, "  is processed"));
           
                dataUrl = strcat(Url, filename);
                data = webread(dataUrl);
                filename_tmp=strcat("data_5", ymd_str ,".nc"); % temporary filename (for loop); data will be always stored under this temp name
                filename_out = websave(filename_tmp,dataUrl); % save downloaded data

                %     ncdisp(filename_tmp,'/','min');      % commented, just to avoid filling the command window
                precip=ncread(filename_tmp,'cmorph');  % Dimensions: lon,lat,time
                long=ncread(filename_tmp,'lon');
                lat=ncread(filename_tmp,'lat');
                time=ncread(filename_tmp,'time');

           end 
           
        end

    end 

end

The error reported is this:

Error using matlab.internal.webservices.HTTPConnector/copyContentToByteArray (line 396)
The server returned the status 404 with message "Not Found" in response to the request to
URL
https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km/2000/01/CMORPH_V1.0_ADJ_8km-30min_20000101.nc.

Error in readContentFromWebService (line 46)
        byteArray = copyContentToByteArray(connection);

Error in webread (line 125)
[varargout{1:nargout}] = readContentFromWebService(connection, options);

Error in Untitled (line 36)
                data = webread(dataUrl);

[Peter Perkins]

Two things:

1) You probably want to look at using a datastore for this. They are specifically designed for this kind of multiple file scenario.

2) You are not doing yourself any favors by using datenums. Use datetimes.

Answer 1

[Augusto Gabriel da Costa Pereira]

Script to download CMORPH reanalysis data that is inserted in the following link: https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/daily/0.25deg/

%% Part 1
clc
clearvars

% Download the CMORPH data used in the script below
which_years = (2000:2021); % years to download

httpsUrl = "https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km/";
% filename_tmp="temp.nc"; % temporary filename (for loop); data will be always stored under this temp name

%% main loop
mm = (1:12); % all monthes in year
% mm = [1 6]; % process only some specific monthes in year

for cc = 1:numel(which_years) % loop over years

      this_year = which_years(cc);

      for ci = 1:numel(mm) % loop over monthes

          days_per_month = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % compute how many days this month/year

          for ck = 1:days_per_month % loop over days

             ymd_str = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % year month day string

             for ch = 1:ymd_str

                 hora_str = strcat(sprintf('%04d',this_year), sprintf('%02d', mm(ci)), sprintf('%02d', ch));

                 Url = strcat(httpsUrl, sprintf('%04d', this_year), "/", sprintf('%02d', mm(ci)),"/",sprintf('%02d',ch),'/');

                 for i = 0:23

                 filename = strcat("CMORPH_V1.0_ADJ_8km-30min_", hora_str ,sprintf('%02d',i),".nc"); % create filename according to list above
                 disp(strcat(filename, "  is processed"));

                  dataUrl = strcat(Url, filename);
                  data = webread(dataUrl);
                  filename_tmp=strcat("data", num2str(hora_str), num2str(sprintf('%02d',i)) ,".nc"); % temporary filename (for loop); data will be always stored under this temp name
                  filename_out = websave(filename_tmp,dataUrl); % save downloaded data

                  %     ncdisp(filename_tmp,'/','min');      % commented, just to avoid filling the command window
                  precip=ncread(filename_tmp,'cmorph');  % Dimensions: lon,lat,time
                  long=ncread(filename_tmp,'lon');
                  lat=ncread(filename_tmp,'lat');
                  time=ncread(filename_tmp,'time');
                 end

             end 

          end

      end 

end

[VBBV]

How is this different from my answer ?

[Augusto Gabriel da Costa Pereira]

Here is:

hora_str = strcat(sprintf('%04d',this_year), sprintf('%02d', mm(ci)), sprintf('%02d', ck(ch)));

filename_tmp=strcat("data", num2str(hora_str), num2str(sprintf('%02d',i)) ,".nc"); % temporary filename (for loop); data will be always stored under this temp name

[VBBV]

Ok, The problem was with error located in the line

 data = webread(dataUrl);

which was resolved, Regarding the download of data, did you look at my updated comment ? . You need to use { } to store all files for years as shown below

 filename_out{i+1} = websave(filename_tmp,dataUrl); % save /downloaded all data

Answer 2

[VBBV]

Note the following changes

clc
clearvars

% Download the CPC data used in the script below
which_years = (2000:2005); % years to download

httpsUrl = "https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km/";
% filename_tmp="temp.nc"; % temporary filename (for loop); data will be always stored under this temp name


%% main loop
mm = (1:12); % all monthes in year
% mm = [1 6]; % process only some specific monthes in year

for cc = 1:numel(which_years) % loop over years
    
    this_year = which_years(cc);

    for ci = 1:numel(mm) % loop over monthes

        days_per_month = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % compute how many days this month/year

        for ck = 1:days_per_month % loop over days

           ymd_str = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % year month day string
           
           for ch = 1:ymd_str
               
               hora_str = strcat(sprintf('%04d',this_year), sprintf('%02d', mm(ci)), sprintf('%02d', ch));
               
               Url = strcat(httpsUrl, sprintf('%04d', this_year), "/", sprintf('%02d', mm(ci)),"/",sprintf('%02d',ch),'/');
               for i = 0:23
               filename = strcat("CMORPH_V1.0_ADJ_8km-30min_", hora_str ,sprintf('%02d',i),".nc"); % create filename according to list above
               disp(strcat(filename, "  is processed"));
           
                dataUrl = strcat(Url, filename);
                data = webread(dataUrl);
                filename_tmp=strcat("data_5", num2str(ymd_str) ,".nc"); % temporary filename (for loop); data will be always stored under this temp name
                filename_out = websave(filename_tmp,dataUrl); % save downloaded data

                %     ncdisp(filename_tmp,'/','min');      % commented, just to avoid filling the command window
                precip=ncread(filename_tmp,'cmorph');  % Dimensions: lon,lat,time
                long=ncread(filename_tmp,'lon');
                lat=ncread(filename_tmp,'lat');
                time=ncread(filename_tmp,'time');
               end

           end 
           
        end

    end 

end

[VBBV]

When i looked at the weburl link provided, for each day there are 23 datasets and the filenames ended with dataset number as shown below

https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates ...
    /access/30min/8km/2000/01/01/CMORPH_V1.0_ADJ_8km-30min_2000010100.nc
                                                                  %^^ here

So, you need to create a for loop to read each dataset and add or strcat addional '/01/' in the filename line as below

filename = strcat("CMORPH_V1.0_ADJ_8km-30min_", hora_str ,sprintf('%02d',i),".nc"); % create filename according to list above

Further, when you refer to year month day string in the line

 ymd_str = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % year month day string
                                %^^                              ^^

you need to use index for month and not for days since number of days are usually more than number of months as variable days_per_month refer to number of days but mm is vector for months with a limit of 12. Alternately, i think the above line should be

ymd_str = datenum(this_year, mm(ci), days_per_month(ck)+1) - datenum(this_year, mm(ci), days_per_month(ck)); % year month day string

Also, in the below line

filename_tmp=strcat("data_5", num2str(ymd_str) ,".nc");

you need to convert the variable ymd_str to string

[Augusto Gabriel da Costa Pereira]

I tried to do what you recommended, but to no avail.

[VBBV]

Do you get any error? What is problem you are facing ? There was no error when I ran the modified program

[Augusto Gabriel da Costa Pereira]

sorry, VBBV

it worked correctly here yes. I was running another totally different script. thanks for your help.

[Augusto Gabriel da Costa Pereira]

however, the script has a small error. when the file is downloaded, the name is not inserted the last character referring to the time. in a day it downloads only one file instead of 24 as a day has 24 hours.

when the file is being saved it is subscribing and not saving one file at a time but saving one on top of the other.

[Augusto Gabriel da Costa Pereira]

see in the current folder how the file is downloaded. the fourth file is being processed (referring to the third hour of the day), but they are saved and overwritten over the data20000101.nc file in the current folder

[Augusto Gabriel da Costa Pereira]

I was able to modify the code and find the error

I modified the following command below

filename_tmp=strcat("data", num2str(hora_str), num2str(sprintf('%02d',i)) ,".nc"); % temporary filename (for loop); data will be always stored under this temp name

The script that is working is this right after:

clc
clearvars

% Download the CMORPH data used in the script below
which_years = (2000:2005); % years to download

httpsUrl = "https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km/";
% filename_tmp="temp.nc"; % temporary filename (for loop); data will be always stored under this temp name


%% main loop
mm = (1:12); % all monthes in year
% mm = [1 6]; % process only some specific monthes in year

for cc = 1:numel(which_years) % loop over years
    
    this_year = which_years(cc);

    for ci = 1:numel(mm) % loop over monthes

        days_per_month = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % compute how many days this month/year

        for ck = 1:days_per_month % loop over days

           ymd_str = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % year month day string
           
           for ch = 1:ymd_str
               
               hora_str = strcat(sprintf('%04d',this_year), sprintf('%02d', mm(ci)), sprintf('%02d', ck(ch)));
               
               Url = strcat(httpsUrl, sprintf('%04d', this_year), "/", sprintf('%02d', mm(ci)),"/",sprintf('%02d',ch),'/');
               
               for i = 0:23
                   
               filename = strcat("CMORPH_V1.0_ADJ_8km-30min_", hora_str ,sprintf('%02d',i),".nc"); % create filename according to list above
               disp(strcat(filename, "  is processed"));
           
                dataUrl = strcat(Url, filename);
                data = webread(dataUrl);
                filename_tmp=strcat("data", num2str(hora_str), num2str(sprintf('%02d',i)) ,".nc"); % temporary filename (for loop); data will be always stored under this temp name
                filename_out = websave(filename_tmp,dataUrl); % save downloaded data

                %     ncdisp(filename_tmp,'/','min');      % commented, just to avoid filling the command window
                precip=ncread(filename_tmp,'cmorph');  % Dimensions: lon,lat,time
                long=ncread(filename_tmp,'lon');
                lat=ncread(filename_tmp,'lat');
                time=ncread(filename_tmp,'time');
               end

           end 
           
        end

    end 

end

[VBBV]

If you want to save all the files, change this line

filename_out = websave(filename_tmp,dataUrl); % save downloaded data

to

filename_out{i+1} = websave(filename_tmp,dataUrl); % save downloaded data

Which downloads and saves data for all files