RMS error and Mean absolute error from text files

There are 24 text files name from interpot_linear_00.txt to interpot_linear_24.txt in a folder. Each file consist on three columns( First is latitude, second is longitude and third column is temperature). I want to check best of my interpolation technique for 324 samples of temperature.

That why i randomly select 30% of samples and gives them -9999.0. Now using scatter data interpolation i consider -9999.0 as bad data values and based upon my 70% remaining samples interpolate -9999.0. After interpolating values of -9999.0 i cross check its value with its original one and calculate overall R.M.S error and Absolute mean error for each text file.

Here i am just try, but little task to remains which need assistance as in my comments

methods = {'natural'};

S = dir('interpot_linear_*.txt');
N = sort({S.name});

for K = 1 : length(N)
  infile = N{K};
  whichfile = sscanf(infile, 'interpot_linear_%c%c');

      % Load the data
      data = load(infile);
    % separate the data columns, just to make the code clear
      Lat = data(:,1); % Column 1 is Latitude
      Lon = data(:,2); % Column 2 is Longitude
      Tmp = data(:,3); % Column 3 is Temperature

     % Creating 30% of sample as -9999.0
      nr=round(n*0.3);
      TE=Tmp(randperm(n,nr));
      good_temp = find(TE);
      TE(:)=-9999.000;
      %makes another temperature column (say Tmp1) with same indices of randomly
      %seclected but -9999.0 value.
      % Find the "good" data points. It should be 70% of remaing samples?
      good_temp = find(Tmp > -9999.000);

      % find the "bad" data points
      bad_temp = find(Tmp == -9999.000);

      for midx = 1 :length(methods) 
        method = methods{midx};
        outfile = ['interpot_' method '_' whichfile '.txt'];

        % creating vector 
        T = scatteredInterpolant(Lat(good_temp), Lon(good_temp), Tmp(good_temp), method);

        % use the interpolation object to interpolate temperature values
        interp_values = T(Lat(bad_temp), Lon(bad_temp));
        % replace the bad values with the interpolated values
        Tmp(bad_temp) = interp_values;

        % Here i will calculate RMS error with formula for each file
        RMS=sqrt(mean(Tmp(:).^2 - Tmp_new(:).^2));
        %Here i am calculating Mean Absolute Error for each file with formula
        MAE=sum(abs(Tmp(:)-Tmp_new(:)))/n;

        % Here i will export results in a excel file.
      end
  end      %files

After calculating RMS error and AM error for 24 text file. I want summery statistics in excel file( File no, RMS, AME etc.) My some text files has been attached with this post.

Please help!

[Stephen23]

The code in the question is derived from these answers (questions also from same OP)

http://www.mathworks.com/matlabcentral/answers/265845-1d-interpolation-from-multiple-text-files

http://www.mathworks.com/matlabcentral/answers/270450-scatter-data-interpolation-from-text-files-in-matlab

http://www.mathworks.com/matlabcentral/answers/272129-bilinear-interpolation-of-2d-matrix

Answer 1

[dpb]

...
% Creating 30% of sample as -9999.0
nr=round(n*0.3);
TE=Tmp(randperm(n,nr));
good_temp = find(TE);
...

In the code above you've not defined n which, it appears, should be length of the vectors or

n=size(data,1);

Then randperm will produce a pseudo-random sample of those indices as you've used it. BUT, what you've called "good_temp" is that 30% subset, not the remaining 70% so you're backwards on which group is which it would appear.

BTW, using just the indices is simpler; you don't need find there...

idx30=randperm(n,nr);  % index of 30% for substitution
tmp(idx30)=nan;        % use Nan instead of "magic number" for missing value

The remaining 70% are those which aren't members of the above set. It's simpler to use logical addressing than absolute for such cases; to generate use

ibad=ismember(1:n,idx30);  % logical vector of 30% bad locations
igood=~ibad;               % complementary "good" locations vector

Using the above index arrays, you don't need to do searching for the missing values; simply use the indexing array for the subset desired in expressions or, the isnan function or the builtin functions such as nanmean and friends which automagically handle NaN as missing value (latter may require Statistics Toolbox depending on release).

[Muhammad Usman Saleem]

Thanks @dpb...

Here i am trying this to one text file with name interpot_linear_00.txt

% Load the data
data = load('interpot_linear_00.txt');
  % separate the data columns, just to make the code clear
    Lat = data(:,1); % Column 1 is Latitude
    Lon = data(:,2); % Column 2 is Longitude
    Tmp = data(:,3); % Column 3 is Temperature

     % Creating 30% of sample as -9999.0
       n=size(data,1);
      nr=round(n*0.3);
      idx30=randperm(n,nr);
      Tmp(idx30)=-9999.0;
      % Find the "good" data points
      good_temp = find(Tmp > -9999.000);
       % find the "bad" data points
      bad_temp = find(Tmp == -9999.000);
       % creating vector 
        T = scatteredInterpolant(Lat(good_temp), Lon(good_temp), Tmp(good_temp), 'natural');
      % use the interpolation object to interpolate temperature values
        interp_values = T(Lat(bad_temp), Lon(bad_temp));
        % replace the bad values with the interpolated values
        Tmp(bad_temp) = interp_values;
        %recall oringinal temperature column
      Tmp_original=data(:,3);
      % Here calculate RMS error
      RMS=sqrt(mean(Tmp_original(:).^2 - Tmp(:).^2));
      % Here calculate mean absolute error
      MAE=sum(abs(Tmp_original(:)-Tmp(:)))/n;
      % Here calculate correlation coefficient
      correl=corrcoef(Tmp_original(:),Tmp(:));
      %making results matrix
      results=[RMS;MAE;correl];
      %here i am exporting my results
      xlswrite(Statistics,results);

Please check it and tell me whether this approach is fine??

As there are 24 text files. I want to export final excel file with summery of statistics of each file. Like first column is file no, second column is RMS error, third column is MAE and fourth column of correl.

One excel file will contains all these information??

[dpb]

Not unless you save it all before writing to Excel or write successive rows in the spreadsheet in the loop, no...as is, it'll overwrite the previous iteration each pass.

[Muhammad Usman Saleem]

Not able to understand , getting confusion please? Please do these two work for me

(1) update this code in order to read each file and calculate its statistics and export excel file as result please?

[dpb]

Not a consulting service, sorry...

It should be clear you'll need to either save all the summary results in the loop over the files and then write those to an external file (Excel or other makes on difference) after the loop completes or write each result to the same file during the loop. If you do the latter, in a sequential file it'll automagically add a record; in Excel you'll have to specify a location (row/column area) for each row as the default is always located at A1.

See the FAQ for the former and there's sufficient info on xlswrite in the doc's for your purposes there...

How_can_I_process_a_sequence_of_files?

[Muhammad Usman Saleem]

In understanding of "you'll need to either save all the summary results in the loop over the files and then write those to an external file (Excel or other makes on difference) after the loop completes or write each result to the same file during the loop."

Here i am doing

   S = dir('Output_*.txt');
  N = sort({S.name});

for K = 1 : length(N)
  infile = N{K};
  whichfile = sscanf(infile, 'Output_%c%c');
% Load the data
data = load(infile);
  % separate the data columns, just to make the code clear
    Lat = data(:,1); % Column 1 is Latitude
    Lon = data(:,2); % Column 2 is Longitude
    Tmp = data(:,3); % Column 3 is Temperature
    %Creating 30% of sample as -9999.0
     n=size(data,1);
    nr=round(n*0.3);
    idx30=randperm(n,nr);
    Tmp(idx30)=-9999.0;
     % Find the "good" data points
    good_temp = find(Tmp > -9999.000);
     % find the "bad" data points
    bad_temp = find(Tmp == -9999.000);
      % creating vector 
      T = scatteredInterpolant(Lat(good_temp), Lon(good_temp), Tmp(good_temp), 'natural');
    % use the interpolation object to interpolate temperature values
      interp_values = T(Lat(bad_temp), Lon(bad_temp));
      % replace the bad values with the interpolated values
      Tmp(bad_temp) = interp_values;
      %recall oringinal temperature column
    Tmp_original=data(:,3);
    % Here calculate RMS error
    RMS=sqrt(mean(Tmp_original(:).^2 - Tmp(:).^2));
    % Here calculate mean absolute error
    MAE=sum(abs(Tmp_original(:)-Tmp(:)))/n;
    % Here calculate correlation coefficient
    correl=corrcoef(Tmp_original(:),Tmp(:));
    results=[RMS';MAE';correl']';
    for ii=1:size(results,1)
        xlswrite(Statistics,results);
    end
end

Please correct it, in reward i shall accept your kind answer?

[Muhammad Usman Saleem]

How to remove this error:

Attempted to access data(:,1); index out of bounds because
size(data)=[0,0].

When i use

n=size(data,1);
or 
n=324;

Error remains same please guide me?

[dpb]

Your attempt to load a file apparently failed; you have no data in data.

Why are you using load on what is apparently a formatted file, without the '-ASCII' switch, anyway? I'd have thought you'd have gotten a "Not a .MAT file" error before reaching the subsequent line referencing data.

[Muhammad Usman Saleem]

@dpb for breaking such silence...

Yes you are absolutely right. There was a wrong file in that folder which was empty. I have delete it. Now when i try make a matrix of each results through

results=[RMS';MAE';correl']';

and even

rep=[RMS(:);MAE(:);correl(:)];

It is storing statistics of only 24 no text file(last one) skipping results of others . How can i store results of each text file in results and at end 24 * 3 matrix will be summery statistics of all files and then how can i export to results in excel? Please

[dpb]

No, it isn't "skipping" anything; your assignment is simply that--an assignment of the current values to a variable. It's doing precisely what you told it to.

Read the documentation on <elementary-matrices-and-arrays> particular the section and examples <Creating-and-concatenating-matrices>

ADDENDUM

HINT: As I described for you quite some time ago, you must either

1. preallocate room for the output array and then populate that array with an index expression to store the results in subsequent rows, or
2. initialize the array variable to the empty vector by '[]' and then dynamically concatenate each element into that array within the loop.

As is, as I've pointed out also multiple times, you're simply assigning the results to the same array variable in its entirety over and over, each pass thru the loop consequently overwriting the previous loop's result. Consequently, the symptom you see of the last pass only being the one result left standing.

Also, once you have finally worked this out, you'll want to move the instruction to write that array to an external file to after the loop; else't you'll be writing the sequential pieces over and over, too...

OR (note well) instead of creating an array, you could simply write each result as it is computed. If this is a sequential file, nothing more need than to ensure include the '\n' in the format string. If indeed must use (ugh!) Excel, then you'll have to (as again noted long ago) create the addressing expression for each write to avoid doing the same thing there you're now doing in memory. For that reason, this option is less palatable than the previous but include it for completeness.

Answer 2

[Walter Roberson]

I have glanced at your question and I see that Duane is handling it perfectly well. You do not need me for it.

[Muhammad Usman Saleem]

Thanks i have already solved it in last night. Please assist to update other.

[Walter Roberson]

I did, several hours before.

[Muhammad Usman Saleem]

let the story to be hear soon. If you help me to find correction in the other question. I hope and sure for this question you have did, several hours before.