How to find Trend of a Uni variate Time series?

Hi,

I have time series data of maximum water level in cm on an hourly basis for a number of years 1961-2016. the data sets are categorized as observed set from water gauges and simulated data from some model. The observed data set has no missing value but there is a long range of data about 10 years having same negative value.(i dont know why? )should i consider it as NaN? I want to calculate the trend of these two uni variate time series? i also need to compare these two series? or one can say how well the simulated time series fit with observed time series?

i have converted my time series into timetable, then i compared them using cross correlation and Nash Sutcliffe coefficient.

tried to plot this data, as well. here is its plot. S1 is observed data plot, and S2 is Simulated data plot.

please guide me how can i find the trend of these series? observed data is attached for your convience. i am badly stuck after reading all helping texts in this regard?

[dpb]

There frankly doesn't appear to be any significant overall trend in those data...

Something is out of whack in a comparison since the mean of the second series is essentially 0 while ~500 for the measurements??

Clearly the measuring device was totally non-functional for a long period or somehow else the data file was corrupted. That section should clearly simply be ignored entirely as being meaningless.

Answer 1

[Rick Rosson]

Have you tried looking for patterns in the data by day of the week, hour of the day, month of the year, or whether the date is a holiday? Time series data often exhibits seasonality effects on different time scales depending on the nature of the observed phenomenum.

doc datetime
doc boxplot
doc scatter
doc histogram

[bushra raza]

Hi,

thank you for your wise reply.

Sir, you are very right. the seasonality may be on different time scales. but i could not get the way to group the uni variate time series. i am not very familiar to all the available options may be. can you please elaborate it more, then i may proceed in the right direction.

i did try scatter plot, which was similar to the one i attached for my question description.

@dpb, Sir, can i consider the long negative values time slot as NaN, or should i interpolate the water levels in this time slot? if i omit this time slot, then would not it be a long delay of values...

looking forward for any guidance please.

[Rick Rosson]

t = TimmendorfData.timmendorf_time;
x = TimmendorfData.timmendorf_water;

mo = month(t);
da = weekday(t);
hr = hour(t);

figure;
boxplot(x,mo);
grid on;

figure;
boxplot(x,da);
grid on;

figure;
boxplot(x,hr);
grid on;

[dpb]

I really can't comment w/o knowing much more than we do other than you have no data for that period, period. From afar, I can't see that it makes any sense to do anything other than compare the data that you do have over the time frames it exists; interpolating to make something up is simply that; there still isn't any data for that period of time available.

If you were missing a day's reading here and there, "well, maybe" you could do something, but when there are 10 years of data missing, that's kinda' a large chunk to just create out of thin air. Now, granted, whatever this is seems to be pretty consistent overall, but still...

NaN has the unfortunate habit of propogating through numerical calculations and turning everything into NaN, though, so that's probably not a viable solution other than for plotting in which it is simply ignored.

Many of the statistics calculations will ignore NaN as well; I'm not certain that the fitting toolbox has such an implementation. You can, of course, use logical addressing to select isfinite() subsets, and overall model-fitting wouldn't be terribly affected, but as others have noted, given you have what appear to be seasonal patterns, those would be drastically altered by the break if you're attempting to model the actual calendar series. If it is simply a model on the seasonal basis, independent of absolute time, then that may also work ok; you'll just have no way to do any verification that there wasn't something different going on during that period of time (albeit, given the rest, it would seem a reasonable assumption to make).

[bushra raza]

Hi,

thank you Mr.Rick and @dpb

yes sir, there is a long gap of 10 years in data set. i had put forward a request to correct this gap or provide another data set.

in the meanwhile, i am working on the suggested options by Mr.Rick.

Answer 2

[Rick Rosson]

Another option is to look at the data in the frequency domain to estimate the relative significance of seasonality at different time scales.

doc fft
doc fftshift
doc abs
doc pwelch
doc spectrogram

[bushra raza]

thanx alot ,sir.

please let me try these options, then come back here.