Optimization stopped because the objective function cannot be decreased in the current search direction.

Error message "Optimization stopped because the objective function cannot be decreased in the current search direction. Either the predicted change in the objective function, or the line search interval is less than eps."

when using fminunc to minimize a multidimension function.

The cost function is "almost smooth" with very small quantization artifact <1e-8. These small bumps are insignificant to the problem itself. Is there any method that can avoid being trapped in these "local minimums" by setting "optimization step size"? Or reduced the resolution when evaluating its gradient?

I found a few options 'DiffMaxChange' 'DiffMinChange' and 'FiniteDifferenceStepSize' potentially related, but not quite clear how they may be used.

[Walter Roberson]

When you say that the cost function is "almost smooth", do you mean cost(x) is, in places, locally constant to within +/- 1e-8 ? Or do you mean that cost(x) is linear to within +/- 1e-8 ? Or do you mean that cost(x) in theory has non-zero continuous second derivative that would be indefinitely smooth in theory but in practice there are matrices with high condition number involved and the error adds up to about +/- 1e-8 ?

[KF]

It is cost(x) is in places, locally constant to within +/- 1e8 where the derivative flips its sign. In theory cost(x) should be monotonic locally. These are real local minimums but caused by numerical/quantization artifacts.

Answer 1

[Walter Roberson]

Is there any method that can avoid being trapped in these "local minimums" by setting "optimization step size"? Or reduced the resolution when evaluating its gradient?

No. Such functions are not suitable for minimization with any minimizer that works with gradients or hessians.

You would need to use ga() or a couple of the other minimizers.

[KF]

I see. Thank you!

Answer 2

[Matt J]

I found a few options 'DiffMaxChange' 'DiffMinChange' and 'FiniteDifferenceStepSize' potentially related, but not quite clear how they may be used.

You can try setting DiffMinChange to be greater than 1e-8, but I think it would be better to remove the quantization operations from your objective function. Make sure any interpolation operations you are doing use cubic interpolation, for example, instead of nearest-neighbor.

[KF]

Thank you! That is a great suggestion. I am working on it.

Does 'DiffMinChange' controls the minmum step when updating x? Should 'FiniteDifferenceStepSize' also be set to a larger number than eps, so that it can evaluate gradient based on a longer range line search?

[Matt J]

Unfortunately, I cannot find any documentation on the relationship between 'DiffMinChange' and 'FiniteDifferenceStepSize', but I am assuming that step size is given by,

step = max(DiffMinChange, FiniteDifferenceStepSize)

That's the only thing that would make sense to me, at least.

[KF]

Thank you! I also feel the documentation for these three options is a bit ambiguous.