How to regrid the spatial data in netcdf format directly without reading it

I am dealing with a large size data (aprox 3 Gb) in netcdf data with 0.004 degree resolution, trying to regrid it to 0.05 degree resolution. The problem I am facing here is the size of data (Out of memory(10902x6818x846)).

Is there any way to regrid (0.004 to 0.05) this data directly without reading (ncread) in matlab and any other to deal it within matlab?

I have gone through interp3 and get idea to execute it but the size of data is curtailing me to do so.

Thank you in advance for any kind of help.

[Walter Roberson]

Is it a 3D regriding you need to do, or can you regrid slice by slice?

Which NetCDF format are the files in? In particular if they are NetCDF 3 or below then it looks like direct binary access into the file is hypothetically possible, whereas NetCDF 4 and above use HDF5 underneath and have the possibility of compression making it more difficult to extract arbitrary data without using the routines.

[Shakir Hussain]

Actuall it is gridded preciptiation data in netcdf4 (nc4). I am seeking any possible way to regrid it either with slice or combinely. I prefer to slice on 3rd dimension (846).

[Walter Roberson]

One thing I have noticed about precipation data is that it is not uncommon for there to be a bunch of missing data. Data marked as missing in netcdf files will typically show up as NaN at the MATLAB level (no matter how it is stored in netcdf.) Do you have the possibility of nan in what you read in? If you do then you need to take more care in the regridding process, as NaN often "poison" the calculations.

The default for interp2 is bilinear interpolation: the new values for any location that is not at an exact vertex is determined by interpolation from the left, right, up, down neighbours of the requested location. If one of those is NaN, then the result will be NaN. How would you like that situation handled?

Answer 1

[Walter Roberson]

You could process one slice at a time. nc4 format is HDF5 underneath, which could be compressed, but nc4 defines that when compression is in place, the compressed data must be chunked in order to make it more efficient to access parts of it. There is the question of how the chunking is done; I suspect that probably layers are kept separate.

oldsize = [10902, 6818, 8456);
factor = 0.004/0.05;
%we read and process one slice first in order to get accurate dimensions for
%preallocating the overall data.
first_slice = ncread(filename, name_of_variable, [1 1 slice], [inf inf 1]);
newdata = imresize(firstslice, factor);
newdata(end, end, oldsize(3)) = 0;       %resize to hold it all
for slice = 2 : oldsize(3)
    thisslice = ncread(filename, name_of_variable, [1 1 slice], [inf inf 1]);
    newslice = imresize(thisslice, factor);
    newdata(:,:,slice) = newslice;
end

[Shakir Hussain]

Yes, you can say its 846 layers of 10902x6818.

This code says:

Error using slice (line 38)

Not enough input arguments.

and line 38 says

error(nargchk(4,8,nargs,'struct'));

[Walter Roberson]

oldsize = [10902, 6818, 8456);
factor = 0.004/0.05;
%we read and process one slice first in order to get accurate dimensions for
%preallocating the overall data.
first_slice = ncread(filename, name_of_variable, [1 1 1], [inf inf 1]);
newdata = imresize(firstslice, factor);
newdata(end, end, oldsize(3)) = 0;       %resize to hold it all
for slice = 2 : oldsize(3)
    thisslice = ncread(filename, name_of_variable, [1 1 slice], [inf inf 1]);
    newslice = imresize(thisslice, factor);
    newdata(:,:,slice) = newslice;
end

[Shakir Hussain]

Undefined function 'imresize' for input arguments of type 'double'.

[Walter Roberson]

imresize is Image Processing Toolbox. An alternative:

factor = 0.004/0.05;
%we read and process one slice first in order to get accurate dimensions for
%preallocating the overall data.
first_slice = ncread(filename, name_of_variable, [1 1 1], [inf inf 1]);
oldsize = [size(first_slice), 846];
%careful about x vs y
newy = linspace(1, oldsize(1), ceil(factor * oldsize(1)));
newx = linspace(1, oldsize(2), ceil(factor * oldsize(2)));
newdata = interp2(first_slice, newx, newy);
newdata(end, end, oldsize(3)) = 0;       %resize to hold it all
for slice = 2 : oldsize(3)
    thisslice = ncread(filename, name_of_variable, [1 1 slice], [inf inf 1]);
    newslice = interp2(thisslice, newx, newy);
    newdata(:,:,slice) = newslice;
end

[Shakir Hussain]

Thank you so much Sir Walter