Calculate daily standard deviation from timetable

Hello, I want to calculate the mean max and mean min standard deviation of temperature data from a daily timetable.

t=timetable(dt, Temperature);
temperaturePerDayMin = retime(t2, 'daily', 'min');
tmpPerDayMin_std= std(temperaturePerDayMin.Temperature);
temperaturePerDayMax = retime(t2, 'daily', 'max');
tmpPerDayMax_std= std(temperaturePerDayMax.Temperature); 

Right now tmpPerDayMin_std and tmpPerDayMax_std return only one value. How can I calculate the standard deviation for each day?

Answer 1

[Star Strider]

Try something like this —

DT = datetime('01-Dec-2022 00:00:00') + hours(0:720).';
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;
TT1 = timetable(DT, Temperature)

TT1 = 721×1 timetable
             DT             Temperature
    ____________________    ___________

    01-Dec-2022 00:00:00      -0.6988  
    01-Dec-2022 01:00:00       1.2667  
    01-Dec-2022 02:00:00       3.0934  
    01-Dec-2022 03:00:00       7.2162  
    01-Dec-2022 04:00:00       10.372  
    01-Dec-2022 05:00:00       12.655  
    01-Dec-2022 06:00:00       15.885  
    01-Dec-2022 07:00:00       18.277  
    01-Dec-2022 08:00:00        16.86  
    01-Dec-2022 09:00:00       16.837  
    01-Dec-2022 10:00:00       19.089  
    01-Dec-2022 11:00:00       14.336  
    01-Dec-2022 12:00:00       12.287  
    01-Dec-2022 13:00:00       9.4703  
    01-Dec-2022 14:00:00       2.5353  
    01-Dec-2022 15:00:00      -1.0999  

figure
plot(TT1.DT, TT1.Temperature)
grid

DailyMax = retime(TT1, 'daily', 'max');
DailyMin = retime(TT1, 'daily', 'min');
DailySTD = retime(TT1, 'daily', @std);
TT2 = table(DailyMax.DT, DailyMax.Temperature, DailyMin.Temperature, DailySTD.Temperature, 'VariableNames',{'DT','Max','Min','StDv'})

TT2 = 31×4 table
        DT          Max        Min       StDv 
    ___________    ______    _______    ______

    01-Dec-2022    19.089     -8.976     9.209
    02-Dec-2022    16.877    -7.9704    8.9778
    03-Dec-2022    18.891    -8.2106    8.3563
    04-Dec-2022    17.053    -7.7505    8.9039
    05-Dec-2022    19.937    -8.5247    8.9055
    06-Dec-2022    17.545    -9.3624    8.7491
    07-Dec-2022    19.835    -6.6562    8.0717
    08-Dec-2022     18.21    -8.5766    8.8617
    09-Dec-2022    17.416    -6.1877    8.1288
    10-Dec-2022    17.298    -8.2496    8.4282
    11-Dec-2022     18.21    -6.3769     8.891
    12-Dec-2022    19.579    -8.3538     9.035
    13-Dec-2022    20.088    -7.3316    8.5688
    14-Dec-2022    19.397    -8.5091    8.7861
    15-Dec-2022    17.578     -8.363    8.9894
    16-Dec-2022    17.249    -8.5435    9.1342

The standard deviation of a single value is zero by definition, so it is only possible to calculate the standard deviation of all temperatures for a single day for that day.

It is possible to calculate the standard deviation for the maximum and minimum temperatures for several days (at least more than 1) and of course for the entire month.

.

[Ancalagon8]

Works perfect! Thanks! In the initial table (TT1) for December, is it possible to calculate the number of days that the temperature was higher than a specific threshold?

[Star Strider]

Thank you!

Yes!

Example —

DT = datetime('01-Dec-2022 00:00:00') + hours(0:720).';
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;
TT1 = timetable(DT, Temperature)

TT1 = 721×1 timetable
             DT             Temperature
    ____________________    ___________

    01-Dec-2022 00:00:00     -0.91782  
    01-Dec-2022 01:00:00       3.4756  
    01-Dec-2022 02:00:00       5.1349  
    01-Dec-2022 03:00:00       5.9696  
    01-Dec-2022 04:00:00       10.697  
    01-Dec-2022 05:00:00       13.404  
    01-Dec-2022 06:00:00       14.426  
    01-Dec-2022 07:00:00       14.551  
    01-Dec-2022 08:00:00       16.658  
    01-Dec-2022 09:00:00       16.201  
    01-Dec-2022 10:00:00       14.351  
    01-Dec-2022 11:00:00       11.749  
    01-Dec-2022 12:00:00       10.131  
    01-Dec-2022 13:00:00       8.4033  
    01-Dec-2022 14:00:00      0.49336  
    01-Dec-2022 15:00:00       0.1879  

figure
plot(TT1.DT, TT1.Temperature)
grid

DailyMax = retime(TT1, 'daily', 'max');
DailyMin = retime(TT1, 'daily', 'min');
DailySTD = retime(TT1, 'daily', @std);
T2 = table(DailyMax.DT, DailyMax.Temperature, DailyMin.Temperature, DailySTD.Temperature, 'VariableNames',{'DT','Max','Min','StDv'})

T2 = 31×4 table
        DT          Max        Min       StDv 
    ___________    ______    _______    ______

    01-Dec-2022    16.658    -8.8893    8.2913
    02-Dec-2022    16.515    -6.5274    8.2742
    03-Dec-2022    18.799    -7.4772    8.7142
    04-Dec-2022    17.145    -9.1443    8.4744
    05-Dec-2022    18.821    -8.4418    8.5831
    06-Dec-2022    18.391    -5.9905    8.9501
    07-Dec-2022    15.976    -8.7264    8.6104
    08-Dec-2022    15.344    -7.7227     8.062
    09-Dec-2022    18.631     -9.265    8.6461
    10-Dec-2022    19.929    -7.4301    9.0601
    11-Dec-2022    17.767    -9.7136    8.8605
    12-Dec-2022    17.554    -7.4757    8.7277
    13-Dec-2022    18.164    -7.6267     9.016
    14-Dec-2022    20.333    -8.9258    9.2568
    15-Dec-2022    16.008    -9.7254    8.7095
    16-Dec-2022    20.091    -7.3932    9.2243

DaysMeetingThreshold = nnz(T2.Max >= 18)                    % Days Equal To Or Higher Than 18°C

DaysMeetingThreshold = 17

Adjust the conditions to select either greater than (>) or greater than or equal to (>=) and the desired temperature threshold (here 18°C). I did not include that in the timetable because it is for the entire time period and so would have to have a column length equal to the entire timetable. This could also be done for calendar months or calendar weeks or for the entire table. That would require extra code and probably an additional timetable as well.

An example of doing that using retime wouild be:

Threshold_1 = retime(TT1, 'weekly', @(x)nnz(x>=18))

Threshold_1 = 5×1 timetable
        DT         Temperature
    ___________    ___________

    27-Nov-2022         1     
    04-Dec-2022         6     
    11-Dec-2022         8     
    18-Dec-2022         4     
    25-Dec-2022         7     

TT2 = table2timetable(T2);
Threshold2 = retime(TT2, 'weekly', @(x)nnz(x>=18))

Threshold2 = 5×3 timetable
        DT         Max    Min    StDv
    ___________    ___    ___    ____

    27-Nov-2022     1      0      0  
    04-Dec-2022     4      0      0  
    11-Dec-2022     4      0      0  
    18-Dec-2022     3      0      0  
    25-Dec-2022     5      0      0  

The ‘Threshold1_1’ timetable counts the number of times each week any ‘Temperature’ equalled or exceeded 18°C.

The ‘Threshold_2’ timetable counts the number of times the maximum ‘Temperature’ equalled or exceeded 18°C.

It could be added to the existing timetable, however that would require additional code and that code would not be trivial, since the lengths of the constant values would have to match the entries of the corresponding dates in position and length.

.

[Ancalagon8]

Thank you for the detailed answer! Is it possible to put

DaysMeetingThreshold = nnz(T2.Max >= 18)                    % Days Equal To Or Higher Than 18°C

inside a loop and store the result in a variable or cell or table? And what about different steps in thresholds? (e.g. 20,25,30, 35,36,38,40,42,44).

[Star Strider]

As always, my pleasure!

I am not certain that I understand what you want to do.

You would likely need a loop for the various thresholds. After that, putting them into a table is straightforward —

DT = datetime('01-Dec-2022 00:00:00') + hours(0:720).';
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;
TT1 = timetable(DT, Temperature)

TT1 = 721×1 timetable
             DT             Temperature
    ____________________    ___________

    01-Dec-2022 00:00:00      -2.4124  
    01-Dec-2022 01:00:00       1.8933  
    01-Dec-2022 02:00:00       2.5341  
    01-Dec-2022 03:00:00       6.1807  
    01-Dec-2022 04:00:00       9.8868  
    01-Dec-2022 05:00:00       14.528  
    01-Dec-2022 06:00:00        14.26  
    01-Dec-2022 07:00:00       15.974  
    01-Dec-2022 08:00:00       14.304  
    01-Dec-2022 09:00:00       17.543  
    01-Dec-2022 10:00:00       12.134  
    01-Dec-2022 11:00:00       14.879  
    01-Dec-2022 12:00:00       11.453  
    01-Dec-2022 13:00:00       6.3793  
    01-Dec-2022 14:00:00        5.644  
    01-Dec-2022 15:00:00        2.349  

figure
plot(TT1.DT, TT1.Temperature)
grid

DailyMax = retime(TT1, 'daily', 'max');
DailyMin = retime(TT1, 'daily', 'min');
DailySTD = retime(TT1, 'daily', @std);
T2 = table(DailyMax.DT, DailyMax.Temperature, DailyMin.Temperature, DailySTD.Temperature, 'VariableNames',{'DT','Max','Min','StDv'})

T2 = 31×4 table
        DT          Max        Min       StDv 
    ___________    ______    _______    ______

    01-Dec-2022    17.543    -9.0929    8.8102
    02-Dec-2022    19.193     -6.685    8.5592
    03-Dec-2022    19.767    -11.037    9.8884
    04-Dec-2022    18.123    -8.7118    8.8951
    05-Dec-2022    17.171    -7.8953    8.8536
    06-Dec-2022    18.443    -8.2063    8.3054
    07-Dec-2022    18.364     -8.341    8.5502
    08-Dec-2022    18.076    -6.3634    8.7735
    09-Dec-2022    18.339    -8.4784    8.7009
    10-Dec-2022    17.352    -8.1611    8.4984
    11-Dec-2022    18.944    -7.9201     8.536
    12-Dec-2022     17.78    -7.6084    8.6725
    13-Dec-2022    18.489    -8.9344    9.1313
    14-Dec-2022    19.052    -7.7588    8.8545
    15-Dec-2022    17.135    -7.8149     8.897
    16-Dec-2022    19.373    -8.0265    9.1146

Thresholdv = [16; 17; 18; 19; 20];                                      % Column Vector
for k = 1:numel(Thresholdv)
    DaysMeetingThreshold(k,:) = nnz(T2.Max >= Thresholdv(k));           % Days Equal To Or Higher Than Thresholdv (°C)
end

ThresholdTable = table(Thresholdv, DaysMeetingThreshold)

ThresholdTable = 5×2 table
    Thresholdv    DaysMeetingThreshold
    __________    ____________________

        16                 30         
        17                 30         
        18                 17         
        19                  5         
        20                  0         

.

[Ancalagon8]

That was exactly what I needed. Once more thank you!

In:

DailyMax = retime(TT1, 'daily', 'max');

can I also determine the hour that the maximum value was observed?

[Star Strider]

As always, my pleasure!

EDIT — (30 Dec 2022 at 16:56)

Not easily.

DT = datetime('01-Dec-2022 00:00:00') + hours(0:720).';
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;
TT1 = timetable(DT, Temperature)

TT1 = 721×1 timetable
             DT             Temperature
    ____________________    ___________

    01-Dec-2022 00:00:00      -1.1057  
    01-Dec-2022 01:00:00       1.3984  
    01-Dec-2022 02:00:00       6.8774  
    01-Dec-2022 03:00:00       6.4973  
    01-Dec-2022 04:00:00       12.631  
    01-Dec-2022 05:00:00        14.64  
    01-Dec-2022 06:00:00       16.174  
    01-Dec-2022 07:00:00       17.047  
    01-Dec-2022 08:00:00       17.984  
    01-Dec-2022 09:00:00       13.908  
    01-Dec-2022 10:00:00       12.703  
    01-Dec-2022 11:00:00       12.548  
    01-Dec-2022 12:00:00       11.678  
    01-Dec-2022 13:00:00       8.3992  
    01-Dec-2022 14:00:00       4.2721  
    01-Dec-2022 15:00:00       1.7873  

figure
plot(TT1.DT, TT1.Temperature)
grid

DailyMax = retime(TT1, 'daily', 'max');
HourlyMax = retime(TT1, 'hourly', 'max')

HourlyMax = 721×1 timetable
             DT             Temperature
    ____________________    ___________

    01-Dec-2022 00:00:00      -1.1057  
    01-Dec-2022 01:00:00       1.3984  
    01-Dec-2022 02:00:00       6.8774  
    01-Dec-2022 03:00:00       6.4973  
    01-Dec-2022 04:00:00       12.631  
    01-Dec-2022 05:00:00        14.64  
    01-Dec-2022 06:00:00       16.174  
    01-Dec-2022 07:00:00       17.047  
    01-Dec-2022 08:00:00       17.984  
    01-Dec-2022 09:00:00       13.908  
    01-Dec-2022 10:00:00       12.703  
    01-Dec-2022 11:00:00       12.548  
    01-Dec-2022 12:00:00       11.678  
    01-Dec-2022 13:00:00       8.3992  
    01-Dec-2022 14:00:00       4.2721  
    01-Dec-2022 15:00:00       1.7873  

DailyMin = retime(TT1, 'daily', 'min');
DailySTD = retime(TT1, 'daily', @std);
T2 = table(DailyMax.DT, DailyMax.Temperature, DailyMin.Temperature, DailySTD.Temperature, 'VariableNames',{'DT','Max','Min','StDv'})

T2 = 31×4 table
        DT          Max        Min       StDv 
    ___________    ______    _______    ______

    01-Dec-2022    17.984    -9.2499    9.0444
    02-Dec-2022    18.206    -8.5237    9.1837
    03-Dec-2022    15.431    -7.5063    8.2329
    04-Dec-2022    19.696     -9.143    9.0854
    05-Dec-2022    19.471      -7.37    8.7003
    06-Dec-2022    19.677    -10.246    9.3604
    07-Dec-2022    16.639    -6.4818    8.3036
    08-Dec-2022    18.539    -6.9418    8.6188
    09-Dec-2022     17.33    -9.3591    8.8737
    10-Dec-2022    17.894    -7.7901     8.513
    11-Dec-2022     18.26    -8.2763    8.9268
    12-Dec-2022     18.52    -9.1745    9.0402
    13-Dec-2022     18.13    -8.7439    8.5561
    14-Dec-2022    19.723    -8.7907    8.8667
    15-Dec-2022    19.534     -9.033    9.2443
    16-Dec-2022    19.735    -7.8816    8.3044

Thresholdv = [16; 17; 18; 19; 20];                                      % Column Vector
for k = 1:numel(Thresholdv)
    DaysMeetingThreshold(k,:) = nnz(T2.Max >= Thresholdv(k));           % Days Equal To Or Higher Than Thresholdv (°C)
end

ThresholdTable = table(Thresholdv, DaysMeetingThreshold)

ThresholdTable = 5×2 table
    Thresholdv    DaysMeetingThreshold
    __________    ____________________

        16                 29         
        17                 26         
        18                 17         
        19                  9         
        20                  0         

[yh,mh,dh] = ymd(HourlyMax.DT);                                         % Return Hour At Which Maximum Temperature Occurred

for k = 1:numel(DailyMax)
    [yd,md,dd] = ymd(DailyMax.DT(k));
    Lv = ismember([yh,mh,dh], [yd,md,dd], 'rows');
    idx = HourlyMax.Temperature(Lv) == DailyMax.Temperature(k);
    HM = HourlyMax(Lv,:);
    DayHourMax(k,:) = timetable2table(HM(idx,:));
end

DayHourMax                                                              % Table Of Hour In Each Day At Which Maximum Temperature Occurred

DayHourMax = 31×2 table
             DT             Temperature
    ____________________    ___________

    01-Dec-2022 08:00:00      17.984   
    02-Dec-2022 07:00:00      18.206   
    03-Dec-2022 07:00:00      15.431   
    04-Dec-2022 08:00:00      19.696   
    05-Dec-2022 06:00:00      19.471   
    06-Dec-2022 08:00:00      19.677   
    07-Dec-2022 10:00:00      16.639   
    08-Dec-2022 09:00:00      18.539   
    09-Dec-2022 08:00:00       17.33   
    10-Dec-2022 08:00:00      17.894   
    11-Dec-2022 08:00:00       18.26   
    12-Dec-2022 09:00:00       18.52   
    13-Dec-2022 08:00:00       18.13   
    14-Dec-2022 08:00:00      19.723   
    15-Dec-2022 09:00:00      19.534   
    16-Dec-2022 07:00:00      19.735   

This should be easier!

.

[Ancalagon8]

Thank you for your answer, I receive the following error:

"To assign to or create a variable in a table, the number of rows must match the height of the table."

[Star Strider]

My pleasure!

My code worked when I ran it, or I would not have posted it.

I have no idea what the problem could be.

[Ancalagon8]

Having a further investigation in my data, I want to calculate the mean min and mean max Temperatures per month, as well as their mean standard deviations. So I guess that:

MonthlyMax = retime(TT1, 'monthly', 'max');
MonthlyMin = retime(TT1, 'monthly', 'min');

and then I have to calculate the MeanMonthlyMax and MeanMonthlyMin?

[Star Strider]

Probably something like this —

MonthlyMax = retime(TT1, 'monthly', @(x)mean(max(x)));

MonthlyMin = retime(TT1, 'monthly', @(x)mean(min(x)));

although I am not certain what you want.

Testing that to see what it returns —

DT = datetime('01-Dec-2022 00:00:00') + caldays(1:364).';
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;
TT1 = timetable(DT, Temperature)

TT1 = 364×1 timetable
        DT         Temperature
    ___________    ___________

    02-Dec-2022       1.8812  
    03-Dec-2022       2.0567  
    04-Dec-2022       6.6711  
    05-Dec-2022       12.135  
    06-Dec-2022       10.489  
    07-Dec-2022       13.718  
    08-Dec-2022       13.921  
    09-Dec-2022        17.49  
    10-Dec-2022       16.473  
    11-Dec-2022       16.773  
    12-Dec-2022       16.449  
    13-Dec-2022       10.577  
    14-Dec-2022       9.2101  
    15-Dec-2022       9.5114  
    16-Dec-2022       5.1743  
    17-Dec-2022      0.10695  

figure
plot(TT1.DT, TT1.Temperature)
grid
hold on
MonthlyMax = retime(TT1, 'monthly', @(x)mean(max(x)))

MonthlyMax = 12×1 timetable
        DT         Temperature
    ___________    ___________

    01-Dec-2022       17.49   
    01-Jan-2023        18.8   
    01-Feb-2023       20.47   
    01-Mar-2023       17.53   
    01-Apr-2023      18.049   
    01-May-2023      18.325   
    01-Jun-2023       19.46   
    01-Jul-2023      18.845   
    01-Aug-2023      19.449   
    01-Sep-2023      19.491   
    01-Oct-2023       20.38   
    01-Nov-2023      17.817   

MonthlyMin = retime(TT1, 'monthly', @(x)mean(min(x)))

MonthlyMin = 12×1 timetable
        DT         Temperature
    ___________    ___________

    01-Dec-2022      -8.1543  
    01-Jan-2023      -8.6214  
    01-Feb-2023      -8.5488  
    01-Mar-2023      -8.1802  
    01-Apr-2023      -8.1751  
    01-May-2023      -9.2394  
    01-Jun-2023      -9.6803  
    01-Jul-2023      -7.3772  
    01-Aug-2023       -6.802  
    01-Sep-2023       -8.406  
    01-Oct-2023      -7.7957  
    01-Nov-2023      -7.0099  

MthV = DT(1) + calmonths(0:11).';
hold on
plot(MthV, [MonthlyMax.Temperature, MonthlyMin.Temperature], '-r')
hold off

.

[Ancalagon8]

Thank you! The mean min and mean max Temperatures per month are correct using:

MonthlyMax = retime(TT1, 'monthly', @(x)mean(max(x)));
MonthlyMin = retime(TT1, 'monthly', @(x)mean(min(x)));

How about their mean standard deviations?

[Star Strider]

As always, my pleasure!

Replace mean with std:

MonthlyMaxSD = retime(TT1, 'monthly', @(x)std(max(x)));

MonthlyMinSD = retime(TT1, 'monthly', @(x)std(min(x)));

That should work without further modification.

[Ancalagon8]

Both returned a 12X1 with zeros (0)

[Star Strider]

I forgot about that. It’s taking the standard deviation of a scalar, and that’s zero by definition. The maximum and minimum of a standard deviation are going to be the same (since it’s also a scalar), and the standard deviation of the maxima or minima for the entire year is going to be scalars representing the entire year, plotting a straight line over all the months. so a different sort of metric may be necessary.

There may be other ways, for example the standard error of the mean for each month, presented as confidence intervals —

DT = datetime('01-Dec-2022 00:00:00') + caldays(1:364).';
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;
TT1 = timetable(DT, Temperature)

TT1 = 364×1 timetable
        DT         Temperature
    ___________    ___________

    02-Dec-2022      0.82051  
    03-Dec-2022       1.9947  
    04-Dec-2022       4.6657  
    05-Dec-2022       7.8428  
    06-Dec-2022       11.934  
    07-Dec-2022       14.881  
    08-Dec-2022       15.027  
    09-Dec-2022       15.659  
    10-Dec-2022       15.119  
    11-Dec-2022       16.932  
    12-Dec-2022       17.403  
    13-Dec-2022       13.246  
    14-Dec-2022       11.374  
    15-Dec-2022       7.0888  
    16-Dec-2022       4.5771  
    17-Dec-2022     -0.48706  

figure
plot(TT1.DT, TT1.Temperature)
grid
hold on
MonthlyMax = retime(TT1, 'monthly', @(x)mean(x)+std(x)/sqrt(numel(x))*1.96)

MonthlyMax = 12×1 timetable
        DT         Temperature
    ___________    ___________

    01-Dec-2022      8.1303   
    01-Jan-2023      10.545   
    01-Feb-2023      8.4384   
    01-Mar-2023      5.0567   
    01-Apr-2023      8.5555   
    01-May-2023      10.352   
    01-Jun-2023      7.3369   
    01-Jul-2023      5.7303   
    01-Aug-2023      9.5159   
    01-Sep-2023      9.2639   
    01-Oct-2023       6.069   
    01-Nov-2023      7.6833   

MonthlyMin = retime(TT1, 'monthly', @(x)mean(x)-std(x)/sqrt(numel(x))*1.96)

MonthlyMin = 12×1 timetable
        DT         Temperature
    ___________    ___________

    01-Dec-2022       2.3238  
    01-Jan-2023       4.3452  
    01-Feb-2023       2.2066  
    01-Mar-2023     -0.90149  
    01-Apr-2023       2.2013  
    01-May-2023       3.7726  
    01-Jun-2023       1.3024  
    01-Jul-2023     -0.49234  
    01-Aug-2023       3.4644  
    01-Sep-2023       3.3579  
    01-Oct-2023      0.20044  
    01-Nov-2023       1.7348  

MthV = DT(1) + days(15) + calmonths(0:11).';
hold on
plot(MthV, [MonthlyMax.Temperature, MonthlyMin.Temperature], '-r')
hold off

Other options could be calculating the monthly standard deviation of the maxima and minima for each day during the month. That would initially require one aggregation to calculate the daily maxima and minima, and a second aggretation to calculate the standard deviation of the monthly values of the daily data. That may be the only way to get a standard deviation aggregated over a month.

The aggregation depends on what you want, and what you want to do with the aggregated data after calculating it.

.

[Ancalagon8]

I understand. The standard error of the mean for each month, presented as confidence intervals is not what I need here. I am sure that the calculation of the monthly standard deviation of the maxima and minima for each day during the month would be enough. So first I calculate the daily maxima and minima like this:

DailyMin = retime(TT1, 'daily', 'min');  
DailyMax = retime(TT1, 'daily', 'max');        

How can i calculate the second aggretation (the standard deviation of the monthly values of the daily data)?

Thank you in advance.

[Star Strider]

As always, my pleasure!

I would create ‘DailyMin’ and ‘DailyMax’ as additional temporary variables, probably in a separate timetable, and then aggregate over them as 'monthly' to create the monthly standard deviations. That may be the only way to do it.

[Ancalagon8]

You mean:

MonthlyMin = retime(DailyMin, 'monthly', 'min');  
MonthlyMax = retime(DailyMax, 'monthly', 'max');  

[Star Strider]

I was thinking of something like this —

DT = datetime('01-Dec-2022 00:00:00') + hours(1:364*24).';                          % Hours
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;        % Temperatures
TT1 = timetable(DT, Temperature)

TT1 = 8736×1 timetable
             DT             Temperature
    ____________________    ___________

    01-Dec-2022 01:00:00      -1.4502  
    01-Dec-2022 02:00:00     -0.14428  
    01-Dec-2022 03:00:00        5.977  
    01-Dec-2022 04:00:00       8.8143  
    01-Dec-2022 05:00:00       9.1807  
    01-Dec-2022 06:00:00        12.38  
    01-Dec-2022 07:00:00        15.92  
    01-Dec-2022 08:00:00       16.297  
    01-Dec-2022 09:00:00       16.229  
    01-Dec-2022 10:00:00       16.797  
    01-Dec-2022 11:00:00       15.728  
    01-Dec-2022 12:00:00       12.875  
    01-Dec-2022 13:00:00        9.361  
    01-Dec-2022 14:00:00       9.4571  
    01-Dec-2022 15:00:00       5.3876  
    01-Dec-2022 16:00:00      0.11136  

DailyMax = retime(TT1, 'daily', 'max')                                              % Daily Aggregation

DailyMax = 365×1 timetable
        DT         Temperature
    ___________    ___________

    01-Dec-2022      16.797   
    02-Dec-2022        17.7   
    03-Dec-2022      17.013   
    04-Dec-2022      16.641   
    05-Dec-2022       17.55   
    06-Dec-2022      18.994   
    07-Dec-2022      20.201   
    08-Dec-2022      17.164   
    09-Dec-2022      17.171   
    10-Dec-2022      18.355   
    11-Dec-2022      16.357   
    12-Dec-2022      15.898   
    13-Dec-2022      17.103   
    14-Dec-2022      17.685   
    15-Dec-2022      20.491   
    16-Dec-2022      20.725   

DailyMin = retime(TT1, 'daily', 'min')                                              % Daily Aggregation

DailyMin = 365×1 timetable
        DT         Temperature
    ___________    ___________

    01-Dec-2022      -7.4328  
    02-Dec-2022      -5.7989  
    03-Dec-2022      -8.8891  
    04-Dec-2022      -8.6656  
    05-Dec-2022      -8.6631  
    06-Dec-2022      -8.6668  
    07-Dec-2022      -8.3697  
    08-Dec-2022      -9.3005  
    09-Dec-2022      -7.7954  
    10-Dec-2022      -8.7234  
    11-Dec-2022      -10.109  
    12-Dec-2022       -9.815  
    13-Dec-2022      -7.3838  
    14-Dec-2022        -8.13  
    15-Dec-2022      -7.6904  
    16-Dec-2022      -7.6138  

DailyTT = DailyMax;

DailyTT = addvars(DailyTT, DailyMin.Temperature)

DailyTT = 365×2 timetable
        DT         Temperature     Var2  
    ___________    ___________    _______

    01-Dec-2022      16.797       -7.4328
    02-Dec-2022        17.7       -5.7989
    03-Dec-2022      17.013       -8.8891
    04-Dec-2022      16.641       -8.6656
    05-Dec-2022       17.55       -8.6631
    06-Dec-2022      18.994       -8.6668
    07-Dec-2022      20.201       -8.3697
    08-Dec-2022      17.164       -9.3005
    09-Dec-2022      17.171       -7.7954
    10-Dec-2022      18.355       -8.7234
    11-Dec-2022      16.357       -10.109
    12-Dec-2022      15.898        -9.815
    13-Dec-2022      17.103       -7.3838
    14-Dec-2022      17.685         -8.13
    15-Dec-2022      20.491       -7.6904
    16-Dec-2022      20.725       -7.6138

MonthlyTT = retime(DailyTT, 'monthly', @(x)std(x))                                  % Monthly Aggregation Of Daily Data

MonthlyTT = 12×2 timetable
        DT         Temperature     Var2  
    ___________    ___________    _______

    01-Dec-2022       1.1873       1.0935
    01-Jan-2023       1.0858      0.82949
    01-Feb-2023       1.3895      0.80469
    01-Mar-2023       1.2204       1.1097
    01-Apr-2023      0.79565       1.0219
    01-May-2023      0.94119       1.1451
    01-Jun-2023       1.0995       1.0555
    01-Jul-2023       0.9984      0.96406
    01-Aug-2023      0.91378       1.1695
    01-Sep-2023       1.1669      0.99959
    01-Oct-2023       1.0167      0.82053
    01-Nov-2023       3.8584       1.4976

MthV = DT(1) + calmonths(0:11).';

figure
hold on
plot(MthV, MonthlyTT{:,1}, '-r') 
plot(MthV, MonthlyTT{:,2}, '-b')
hold off
legend('Max Monthly SD','Min Monthly SD', 'Location','best')

This plots the standard deviations, not the actual temperature values.

.

[Ancalagon8]

Thank you for your answer. I am wondering if the part

DailyTT = DailyMax;
DailyTT = addvars(DailyTT, DailyMin.Temperature)

instead of DailyMin.Temperature need to have DailyMax.Temperature or I am wrong?

[Star Strider]

The ‘DailyTT’ timetable is defined initially as having the daily max temperatures. The daily min temperatures were added to it as a new variable, so now ‘DailyTT’ has both as different variables. It’s just not possible to change their variable names easily to reflect that.

[Ancalagon8]

Thank you @Star Strider, it was my misunderstanding, everything is clear now. Having calculated the above, I also calculated the daily range like this:

DailyRange=DailyMax.Temperature-DailyMin.Temperature;  

and added it to DailyTT like this:

DailyTT = addvars(DailyTT, DailyMin.Temperature,DailyRange);

So now I have a 365 timetable, where my final goal is to calculate the mean,min and max range for each month.Any ideas how to proceed?

[Star Strider]

I’m not certain that I understand about ‘mean,min and max range for each month’. The mean for the month should be only one value, unless you intend the range of daily temperatures (max - min for each day) and the statistics on those values.

The coding depends on how you want to calculate the daily values and then aggregate them.

[Ancalagon8]

Indeed, the mean for each month is one value, so a 12X1. The min range for each month is the lower and upper limit of min values for each month, and the same for the max range. Any idea?

[Star Strider]

Thank you!

I just wanted to clarify my understanding of this.

EDIT — I’m kind of lost at this point. Isn’t that the result you’ree getting? That’s the result I got in the code in my earlier Comment. That dealt with the standard deviations, however calculating different parameters (min, max, and mean) should yield essentially the same sort of result. The only difference should be the last (function) argument in various aggregations:

MonthlyTT = retime(DailyTT, 'monthly', @(x)std(x)) % Monthly Aggregation Of Daily Data

to produce different results.

I’ll follow up on this tomorrow if there’s anything else to be coded.

.

[Ancalagon8]

What I mean with range is this:

DailyRange=DailyMax.Temperature-DailyMin.Temperature;  %Daily range for the whole year
DailyRange_January = DailyRange(1:31);                 %Daily range for January
DailyRange_February = DailyRange(32:59);               %Daily range for February
.
.
.
DaileRange_December = DailyRange(335:end);             %Daily range for December

and then define the mean, min max for each month

MeanDailyRange_January=mean(DailyRange_January);
MinDailyRange_January=min(DailyRange_January);
MaxDailyRange_January=max(DailyRange_January);

And the same process for all the months. So I wonder if there is a more convenient way to improve the above code and how can I add the time information of the min and max.

[Star Strider]

Try something like this —

DT = datetime('01-Dec-2022 00:00:00') + hours(1:364*24).';                          % Hours
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;        % Temperatures
TT1 = timetable(DT, Temperature)

TT1 = 8736×1 timetable
             DT             Temperature
    ____________________    ___________

    01-Dec-2022 01:00:00     -0.37909  
    01-Dec-2022 02:00:00      0.28714  
    01-Dec-2022 03:00:00       6.4977  
    01-Dec-2022 04:00:00       7.7289  
    01-Dec-2022 05:00:00       12.292  
    01-Dec-2022 06:00:00       15.373  
    01-Dec-2022 07:00:00       14.676  
    01-Dec-2022 08:00:00       15.309  
    01-Dec-2022 09:00:00       15.836  
    01-Dec-2022 10:00:00        16.44  
    01-Dec-2022 11:00:00       16.672  
    01-Dec-2022 12:00:00        14.35  
    01-Dec-2022 13:00:00       8.6757  
    01-Dec-2022 14:00:00       9.1514  
    01-Dec-2022 15:00:00       6.0855  
    01-Dec-2022 16:00:00       2.4084  

DailyMax = retime(TT1, 'daily', 'max');                                             % Daily Aggregation
DailyMin = retime(TT1, 'daily', 'min');                                             % Daily Aggregation
DailyMean = retime(TT1, 'daily', 'mean');                                           % Daily Aggregation
DailyRange = DailyMax.Temperature - DailyMin.Temperature;                           % Daily Aggregation

DailyTT = timetable(DailyMax.DT, DailyMax.Temperature, DailyMin.Temperature, DailyRange, DailyMean.Temperature, 'VariableNames',{'Max_Temperature','Min_Temperature','Range_Temperature','Mean_Temperature'})

DailyTT = 365×4 timetable
       Time        Max_Temperature    Min_Temperature    Range_Temperature    Mean_Temperature
    ___________    _______________    _______________    _________________    ________________

    01-Dec-2022        16.672             -9.4342             26.106               5.1988     
    02-Dec-2022        18.117             -8.7949             26.911               5.3819     
    03-Dec-2022        18.611             -7.9121             26.523               5.1666     
    04-Dec-2022         18.68             -8.1488             26.829               4.8373     
    05-Dec-2022         17.69             -7.1424             24.832               5.3975     
    06-Dec-2022        19.069             -8.1585             27.228               5.2481     
    07-Dec-2022        15.966             -9.9869             25.952               4.8057     
    08-Dec-2022        18.182             -9.3876              27.57               5.4564     
    09-Dec-2022        17.444             -8.3602             25.804               4.8884     
    10-Dec-2022        19.302             -7.5362             26.838                5.391     
    11-Dec-2022        16.696             -10.497             27.193                4.528     
    12-Dec-2022          18.3             -6.9914             25.291               5.1051     
    13-Dec-2022        18.151             -8.5995              26.75               5.4419     
    14-Dec-2022        17.836             -10.976             28.811               5.0321     
    15-Dec-2022        17.282             -9.1337             26.416               4.8376     
    16-Dec-2022        18.325             -8.0825             26.408               4.8119     

MonthlyTTMax = retime(DailyTT, 'monthly', 'max')

MonthlyTTMax = 12×4 timetable
       Time        Max_Temperature    Min_Temperature    Range_Temperature    Mean_Temperature
    ___________    _______________    _______________    _________________    ________________

    01-Dec-2022        20.326             -6.2942             28.852               5.4564     
    01-Jan-2023        20.882             -6.5416             30.436                5.486     
    01-Feb-2023        21.771             -6.7069             29.895                 5.42     
    01-Mar-2023        20.752             -6.3323             30.718               5.7682     
    01-Apr-2023        19.842             -6.1608              28.69                 5.81     
    01-May-2023        19.771             -6.3754             29.726               5.3571     
    01-Jun-2023        20.162             -6.2147             29.415                5.839     
    01-Jul-2023        21.085             -6.1956             30.339               5.8512     
    01-Aug-2023        20.793              -5.192             29.184               5.6606     
    01-Sep-2023        20.346             -6.2424             29.367               5.5407     
    01-Oct-2023        20.827              -5.951             29.933               5.4544     
    01-Nov-2023        20.722             -5.9304             30.332               5.5216     

MonthlyTTMin = retime(DailyTT, 'monthly', 'min')

MonthlyTTMin = 12×4 timetable
       Time        Max_Temperature    Min_Temperature    Range_Temperature    Mean_Temperature
    ___________    _______________    _______________    _________________    ________________

    01-Dec-2022         15.966            -10.976             22.838               4.4957     
    01-Jan-2023         16.409            -10.684             24.377               4.5691     
    01-Feb-2023         15.547             -10.91             24.217               4.3471     
    01-Mar-2023         15.748            -10.449             23.368               4.5982     
    01-Apr-2023         15.941            -10.591             22.533               4.2688     
    01-May-2023         16.111            -10.972             23.687               4.3221     
    01-Jun-2023         16.484            -11.179             23.418               4.3588     
    01-Jul-2023         16.605            -10.795             24.173               4.5544     
    01-Aug-2023         15.518            -10.379             22.288               4.3416     
    01-Sep-2023         15.359            -10.131             22.941               4.5646     
    01-Oct-2023         16.226            -10.036             22.756               4.4768     
    01-Nov-2023        -5.9304            -10.465                  0              -5.9304     

MonthlyTTMean = retime(DailyTT, 'monthly', 'mean')

MonthlyTTMean = 12×4 timetable
       Time        Max_Temperature    Min_Temperature    Range_Temperature    Mean_Temperature
    ___________    _______________    _______________    _________________    ________________

    01-Dec-2022        17.894             -8.3725             26.267               5.0415     
    01-Jan-2023        18.351             -8.3832             26.735               4.9523     
    01-Feb-2023        18.018             -8.3188             26.337                4.923     
    01-Mar-2023        18.023             -8.1983             26.222               5.0542     
    01-Apr-2023        17.919              -8.011              25.93               5.0405     
    01-May-2023        18.123             -8.3906             26.513                4.902     
    01-Jun-2023        17.953             -8.0177             25.971               5.0389     
    01-Jul-2023        18.193             -8.1407             26.334               5.0607     
    01-Aug-2023        18.041             -8.2389              26.28               4.9959     
    01-Sep-2023        18.051              -7.866             25.917               5.0096     
    01-Oct-2023        17.892             -7.9565             25.848               4.9462     
    01-Nov-2023         17.38             -8.1048             25.485               4.6034     

MonthlyTTSD = retime(DailyTT, 'monthly', @(x)std(x))                                  % Monthly Aggregation Of Daily Data

MonthlyTTSD = 12×4 timetable
       Time        Max_Temperature    Min_Temperature    Range_Temperature    Mean_Temperature
    ___________    _______________    _______________    _________________    ________________

    01-Dec-2022        0.89012             1.0981             1.2396              0.28508     
    01-Jan-2023         1.1506             1.0405             1.3665              0.27683     
    01-Feb-2023         1.1575             1.0735             1.6071              0.29298     
    01-Mar-2023         1.2434            0.99746             1.8445              0.28534     
    01-Apr-2023              1             1.1311             1.6295              0.34489     
    01-May-2023        0.99102             1.1128             1.4308              0.28412     
    01-Jun-2023         0.8422             1.0872             1.5087               0.3929     
    01-Jul-2023        0.87463              1.033             1.4316              0.31694     
    01-Aug-2023         1.0816             1.1013             1.7648              0.32041     
    01-Sep-2023         1.1947             1.0865             1.6608              0.27119     
    01-Oct-2023          1.043            0.97543             1.5385              0.24107     
    01-Nov-2023         4.5398             1.0787             5.0632                2.007     

MthV = DT(1) + calmonths(0:11).';

figure
hold on
plot(MthV, MonthlyTTSD{:,1}, '-r') 
plot(MthV, MonthlyTTSD{:,2}, '-b')
hold off
legend('Max Monthly SD','Min Monthly SD', 'Location','best')

The ‘MonthlyTT’ set of timetable arrays should have everything you need. If necessary, it should be straightforward to abstract the various variables from them and import them to a new summary timetable, similar to what I did to create ‘DailyTT’.

Creating numbered variables such as:

MeanDailyRange_January=mean(DailyRange_January);

is generally discouraged. You can get the various monthly values from the timetable arrays that calculated them.

I am doing my best to provide what you want, however that is not always clear to me. The result is that we need to iterate until we converge on the correct arrays.

.

[Ancalagon8]

Thank you @Star Strider. So far things are fine, but my concern is how can I add the time information of the min and max.

[Star Strider]

As always, my pleasure!

I thought we already covered that problem at some point. By now I’ve lost track of where it is (although searching for it, I believe it was here).

It would be essentially the same approach.

.