Lsqnonlin wrong f(x) value

I need to solve the inverse problem and I use lsqnonlin to optimize my parameters. In some cases, lsqnonlin works incorrectly. To demonstrate the problem I have two variants of simple code. First works well, second doesn't work at all.

1. In the first variant, I optimize x^2.

 options = optimoptions('lsqnonlin','Display','iter');
 param_opt = lsqnonlin(@(param)loss_func(param),0.1,-1,1,options);

 function loss = loss_func(param)
     loss = param.^2;
 end

I get this result:

                                         Norm of      First-order 
 Iteration  Func-count     f(x)          step          optimality
     0          2          0.0001                        0.0022
     1          4     7.40992e-06      0.0456004       0.000299      
     2          6     5.09591e-07      0.0248167       3.92e-05      
     3          8     3.35176e-08      0.0130147       5.02e-06

param_opt = 0.0135;

2. In the second variant, I add a small error to the loss function (as I have in my real experiment) and lsqnonlin doesn't work.

 options = optimoptions('lsqnonlin','Display','iter');
 param_opt = lsqnonlin(@(param)loss_func(param),0.1,-1,1,options);

 function loss = loss_func(param)
     loss = param.^2+(1e-6)*rand;
 end

I get this result:

                                         Norm of      First-order 
 Iteration  Func-count     f(x)          step          optimality
     0          2     0.000100005                        0.0315
     1          4     0.000100005     0.00299768         0.0315      
     2          6     0.000100005     0.00074942         0.0315      
     3          8     0.000100005    0.000187355         0.0315      
     4         10     0.000100005    4.68387e-05         0.0315      
     5         12     0.000100005    1.17097e-05         0.0315      
     6         14     0.000100005    2.92742e-06         0.0315      
     7         16     0.000100005    7.31855e-07         0.0315

param_opt = 0.1000;

What am I doing wrong? The problem that I have in my real experiment is that f(x) from lsqnonlin doesn't match with the real loss_func output.

Answer 1

[Star Strider]

It would seem that you are asking lsqnonlin to hit a moving target, since rand returns a different value each time ‘loss_func’ is called.

[Igor Belyasov]

As I understand lsqnonlin should make wrong steps and return new f(x) at each parameter value. But lsqnonlin returns same f(x) as at the first iteration. That is the problem that I have in my real experiment. f(x) from lsqnonlin doesn't match with real loss_func output.

[Matt J]

That is the problem that I have in my real experiment. f(x) from lsqnonlin doesn't match with real loss_func output.

The loss function is not supposed to be a simulation of your physical experiment. It is supposed to quantify, for a particular parameter guess, the agreement between your model (which is non-random) and a set of measurements that you have already taken. "Already taken" implies that the measurements may have come from some random process, but are not random anymore because they have already been observed: measurements that you have already recorded do not continue to fluctuate randomly. Therefore, rand() operations have no place in the loss_function.

But lsqnonlin returns same f(x) as at the first iteration.

That's because lsqnonlin is trying to estimate the derivative of your loss function (with respect to param) using finite differences

deriv = (loss_func(param + delta) - loss_func(param))/delta

But the above always has a random result because of the inclusion of the (1e-6)*rand term. Therefore, lsqnonlin is taking meaningless random steps that just happen not to hit an f(x) value lower than what it was initially.

[Star Strider]

@Matt J — Thank you!