Issue with ncread when _FillValue, add_offset, and scale_factor are present

I have run into an issue when trying to concatenate multiple netcdf files with the same variables into one file, combining these variables.

First, I create a new netcdf file using the header information from one of the files I wish to concatenate using ncinfo and ncwriteschema, changing the length of the variable along which I wish to concatenate to unlimited.

When I read a variable that does not have the attributes _FillValue, add_offset, and scale_factor using ncread, the variable reads into my workspace as I would expect, and writes as expected into the new file.

However, when I read a variable that does have those attributes, the variable reads into my workspace seemingly without applying those attributes, contrary to ncread's documentation. I checked this using netcdf.getVar, and the results are the same between the two methods, which seems to indicate that ncread is failing to apply the attributes.

Strangely, however, when I use ncwrite to store the variable data in the new netcdf file, and then read it back in with either ncread or netcdf.getVar, I get something different! The variable data now oscillates between the _FillValue and -1*_FillValue.

Hypothesis: To me, this seems to indicate that ncwrite is trying to apply the attributes, but since ncread or netcdf.getVar did not apply them inversely, it's now applying them a second time! This causes the values to be larger in magnitude than the _FillValue, and confuses the netCDF file.

An easy workaround would be to apply the attributes myself using ncreadatt and ncwriteatt, but I'm not sure my hypothesis is correct, so I'm hesitant to brute force the process in that way.

Working on attaching code and small enough netCDF files, will update when those are ready.

UPDATE: netCDF files are uploaded as "netcdf_files.zip", matlab code uploaded as "forumpost_ncread.m".

[Daniel Holstein]

This is a few years old, but I am having a similar issue. When reading a .nc into matlab using netcdf.getVar the fill values are correct as defined in the file. If loading with ncread, all fill values are changed to NaN. I believe that netcdf.getVar is reading the matrices as they are stored, but ncread converts any value that equals a fill value to NaN... this can be an issue for manipulating these files, as the fill values can change...

Answer 1

[meghannmarie]

Ncread is working, put a breakpoint on line 108. When you use ncread and have a scale factor/add offset, it returns a double. In your code, you are taking your double variable and recasting it an int16 which is turning everything into integers. You need to delete lines 108 and 109.

vname = info.Variables(kn).Name;                                   
var = ncread(fget,vname);    %variable that is double precision when there is a scale factor/add offset                                        
dtype = info.Variables(kn).Datatype;                                  
var = cast(var,dtype); %You are now recasting double to int16, delete this 

[Sam Kastner]

Ah, right, thanks. I've implemented this by commenting lines 108-109, but it didn't change the problem for me.

[meghannmarie]

It looks like it made my files correctly. Only thing I see wrong is it puts February before January. To fix that, you need to sort your flist variable.

What problem are you still having?

When I try this the numbers seem to be correct:

files = {'ECMWF_1hr_2018_Jan.nc';'ECMWF_1hr_2018_Feb.nc'};

for f = 1:numel(files)
    d = ncread(file,'u10',[1 1 1],[1 1 1])
    ncid = netcdf.open(file);
    varid = netcdf.inqVarID(ncid,'u10');
    i = netcdf.getVar(ncid,varid,[0 0 0],[1 1 1]);
    add_offset = netcdf.getAtt(ncid,varid,'add_offset');
    scale_factor = netcdf.getAtt(ncid,varid,'scale_factor');
     d == double(i) * scale_factor + add_offset
     netcdf.close(ncid)
end

[Sam Kastner]

Sorry for the vagueness! I'm still having issues with ncread, but using netcdf.getVar and its accomplices seems to work just fine. I'll just have to do the offset/scale factor stuff manually as you do above. Thanks for the help!

[meghannmarie]

If you add the lines below to your code, you will see ncread is working just fine. Apply the offset and scale factor and then look at your comparison pictures. Without applying those, you are comparing apples to oranges (int16 numbers to doubles).

ncid = netcdf.open(flist(1).name);                                          % open original netcdf file for reading
varid = netcdf.inqVarID(ncid,'u10');                                        % get wind speed variable id, should be 3
u_longway = netcdf.getVar(ncid,varid);                                      % get wind speed variable data
add_offset = netcdf.getAtt(ncid,varid,'add_offset');
scale_factor = netcdf.getAtt(ncid,varid,'scale_factor');
 u_longway = double(u_longway) * scale_factor + add_offset;
netcdf.close(ncid);

ncid = netcdf.open(fname);                                                  % open new netcdf file for reading
varid = netcdf.inqVarID(ncid,'u10');                                        % get wind speed variable id, should be 3
u_longway_new = netcdf.getVar(ncid,varid);                                  % get wind speed variable data
add_offset = netcdf.getAtt(ncid,varid,'add_offset');
scale_factor = netcdf.getAtt(ncid,varid,'scale_factor');
 u_longway_new = double(u_longway_new) * scale_factor + add_offset;
netcdf.close(ncid);