Problem with Matlab sliding neighborhoods

I have an image. I want to create a sliding window of changing size and want to make some calculations depending on some criteria. For example I initialize window size=3 and then calculate the min max and median in that window for a pixel. If the criteria are not ok I want to use window size=5 etc. This depends on the pixel's neighbors and I want to do that for every pixel. So I don't want to calculate something with nfilter since the window size is going to change for every pixel.