Briain teaser - Filtering for particular algorithm

I have a 1 second dataset of 86400 wind speed (WS) values and need assistance in filtering it. It requires a certain level of cleverness.

If the average WS exceeds:

• 25m/s in a 600s time interval
• 28m/s in a 30s time interval
• 30m/s in a 3 s time interval

If any of these parameters are met, the WS is deemed 'invalid' until the average WS remains below 22m/s in a 300 s time interval.

Here is what I have for the 600 second requirement. I do a 600 and 300 second moving average on the data contained in 'dataset'. I filter the intervals from the first appearance of an average 25m/s to the next appearance of a value below 22m/s as 'NaN'. After filtering, I will do another 600 second average, and the intervals with values flagged with a NaN will be left a NaN.

i.e.

Rolling600avg(:,1) = tsmovavg(dataset(:,2), 's', 600, 1);
Rolling300avg(:,1) = tsmovavg(dataset(:,2), 's', 300, 1);
a = find(Rolling600avg(:,2)>25)
b = find(Rolling300avg(:,2)<22)
dataset(a:b(a:find(b==1)),2)==NaN; %?? Not sure

This is going to require a clever use of 'find' and some indexing. Could someone help me out? The 28m/s and 30m/s filters will follow the same method.

Answer 1

[Walter Roberson]

T = cumsum(dataset(:,2));

Rolling600avg = (T(601:end) - T(1:end-600)) ./ 600;
Rolling300avg = (T(301:end) - T(1:end-300)) ./ 300;
Rolling3avg =   (T(4:end)   - T(1:end-3)) ./ 3;

valid = true(1, size(dataset,1));
valid(601:end) = Rolling600avg < 25;
valid(301:end) = valid(301:end) & (Rolling300avg < 38);
valid(4:end) = valid(4:end) & (Rolling3avg < 30);

dataset(~valid) = NaN;

[Braden]

Thanks Walter. There are 3 cases here:

• 25m/s in a 600s time interval, must remain below 22m/s in contiguous 300s interval to become valid again
• 28m/s in a 30s time interval, must remain below 22m/s in contiguous 300s interval to become valid again
• 30m/s in a 3s time interval, must remain below 22m/s in contiguous 300s interval to become valid again

Would this edit to your code work?

T = cumsum(dataset(:,2));
Rolling600avg = (T(601:end) - T(1:end-600)) ./ 600;
Rolling300avg = (T(301:end) - T(1:end-300)) ./ 300;
Rolling30avg = (T(31:end) - T(1:end-30)) ./ 30;
Rolling3avg =   (T(4:end)   - T(1:end-3)) ./ 3;
valid = true(1, size(dataset,1));
valid(601:end) = Rolling600avg < 25;
valid(301:end) = valid(301:end) & (Rolling300avg < 22);
valid(31:end) = valid(31:end) & (Rolling300avg < 28);
valid(4:end) = valid(4:end) & (Rolling3avg < 30);
dataset(~valid) = NaN;

[Walter Roberson]

Doesn't look quite right, but I will need to think about it more.

Isn't it the case that when a violation occurs, the "end" of the violation is not until the first instance after that window, that Rolling300avg < 22 ?

What should happen if Rolling300avg turns < 22 in the middle of a 600 violation?

[Braden]

Yes I don't think that code will work. I don't think it can build on itself like that. There has to be 3 seperate instances, all treating the end of the violation as the next time when Rolling300avg <22.

[Walter Roberson]

To confirm, "next time" refers to past the current window? Excluding any data in the current window?

[Braden]

Yes, after the WS has exceeded one of those 3 parameters, it must remain below 22m/s for a 300s time interval to be considered valid again.

[Braden]

This might work - the for loops are clunky but I think it does the trick.

[dataset = wblrnd(15,2,100000,1);
    m1 = tsmovavg(dataset, 's', 300, 1);
    m2 = tsmovavg(dataset, 's', 600, 1);

    a = find(m1<22);
    b = find(m2>25);
    m21 = m2;
    % Use a loop to isolate segments that should be NaNs;
    for ii = 1:length(b)
        firstNull = b(ii)
        lastNull  = a( find(a>firstNull,1) )-1 % THIS TRIES TO FIND A VALUE IN B GREATER THAN A(II)- IF THERE IS NO SUCH VALUE THEN NANS SHOULD FILL TO THE END OF THE VECTOR
        if isempty(lastNull),
            lastNull=length(m2);
        end
        m21(firstNull:lastNull) = NaN;
    end

    m3 = tsmovavg(dataset, 's', 30, 1);
    c = find(m3>28);
    m22 = m2;
    for ii = 1:length(c)
        firstNull = c(ii)
        lastNull  = a( find(a>firstNull,1) )-1 
        if isempty(lastNull),
            lastNull=length(m2);
        end
        m22(firstNull:lastNull) = NaN;
    end

    m4 = tsmovavg(dataset,'s',3,1);
    d = find(m4>30);
    m23 = m2;
    for ii = 1:length(d)
        firstNull = d(ii)
        lastNull  = a( find(a>firstNull,1) )-1 % 
        if isempty(lastNull);
            lastNull=length(m2);
        end
        m23(firstNull:lastNull) = NaN;
    end

    mfinal = m2(~isnan(m21)&~isnan(m22)&~isnan(m23));