I researched about ways to characterize how smooth boundary of an image is, please refer below my findings:
- There are literature papers on classifying the smoothness of images using wavelet image processing. In wavelet transformation to get sharp change in frequencies(which are noise in case of signal and in your case it’s the rough edges as they are high frequency change) we pass signal to convolution network with High Pass Filter and then down sample the output from the network and provide that output as input for next iteration. Even here thresholding is used to remove noise.
- You can compare your images with a reference smoothened image and depending upon the difference decide about the threshold to classify images.
As per my findings there is no direct function which can provide you a measure of smoothness for an image, you can have a look into these ways.
Hope this helps!