I am optimizing a function across a range of conditions; sometimes the optimum occurs at an "x" value very close to zero (e.g. 1e-4). I am using an automatic method to generate a guess that is close, to avoid local minima. However, if this guess is very small (e.g. on the order of 1e-4), FMINSEARCH terminates immediately, even though the function is obviously not at a local minimum. As far as I can tell, that's because the inital simplex is generated by taking a step that is proportional to the initial guess, which means the initial step is on the order 1e-6. Then both the tolX and tolFun are trivially met and the algorithm terminates.
Here's a simple example: using the example given in the documentation (Rosenbrock's function), it returns the correct minimum if given initial guess around [1 1]. But if you give it x0 = [1e-4 1e-4], it will terminate immediately, returing an "optimal" function value of 0.99 (should be zero).
This seems like bad behavior to me. Having read the code for FMINSEARCH, I understand why the user should never give a small guess like this, but that's not obvious from the documentation. Even worse, I'm finding occasionally I get bad behavior when using initial guesses of exactly zero, despite the efforts of FMINSEARCH to handle this case properly (but not scaling the simplex by the initial guess). It doesn't happen in the example below, but I've encountered it in more complex functions.
Example code:
fun = @(x)100*(x(2) - x(1)^2)^2 + (1 - x(1))^2;
x = fminsearch(fun,[0 0],optimset('Display','iter')),
Iteration Func-count min f(x) Procedure
0 1 1
1 3 0.9995 initial simplex
2 4 0.9995 reflect
3 6 0.998515 expand
4 8 0.998001 expand
5 10 0.995798 expand
6 12 0.993647 expand
7 14 0.988291 expand
8 16 0.981289 expand
9 18 0.967873 expand
10 20 0.947913 expand
11 22 0.918174 expand
12 24 0.880695 expand
13 26 0.878606 reflect
14 28 0.875402 contract outside
15 30 0.875402 contract inside
16 32 0.865009 expand
17 33 0.865009 reflect
18 35 0.849041 expand
19 36 0.849041 reflect
20 38 0.814926 expand
21 39 0.814926 reflect
22 41 0.759288 expand
23 43 0.74241 expand
24 45 0.637178 expand
25 46 0.637178 reflect
26 48 0.485315 expand
27 49 0.485315 reflect
28 51 0.437854 reflect
29 53 0.314632 expand
30 55 0.314632 contract inside
31 57 0.314632 contract outside
32 59 0.225439 expand
33 60 0.225439 reflect
34 62 0.225439 contract inside
35 64 0.225439 contract inside
36 66 0.190437 expand
37 68 0.145468 expand
38 69 0.145468 reflect
39 71 0.0793485 expand
40 73 0.0793485 contract outside
41 74 0.0793485 reflect
42 76 0.0658189 contract inside
43 78 0.0632972 contract inside
44 80 0.0526552 reflect
45 81 0.0526552 reflect
46 83 0.0408746 reflect
47 85 0.0408746 contract inside
48 87 0.0321917 reflect
49 89 0.0287406 expand
50 91 0.0201617 reflect
51 93 0.0166119 reflect
52 94 0.0166119 reflect
53 96 0.00376929 expand
54 98 0.00376929 contract inside
55 100 0.00288529 contract outside
56 102 0.000102766 reflect
57 104 0.000102766 contract inside
58 105 0.000102766 reflect
59 106 0.000102766 reflect
60 108 0.000102766 contract inside
61 110 0.000102766 contract inside
62 112 2.57387e-05 contract outside
63 114 1.29086e-05 contract inside
64 116 1.29086e-05 contract inside
65 118 5.59238e-06 contract inside
66 120 3.66881e-06 contract inside
67 122 7.22445e-07 contract inside
68 124 7.22445e-07 contract outside
69 126 7.22445e-07 contract inside
70 128 1.65577e-07 reflect
71 130 8.03053e-08 contract inside
72 132 8.03053e-08 contract inside
73 134 1.70818e-08 contract inside
74 136 1.70818e-08 contract inside
75 138 1.23591e-08 contract inside
76 140 5.61681e-09 contract outside
77 142 2.02249e-09 contract inside
78 144 1.63628e-09 contract inside
79 146 3.68618e-10 contract outside
Optimization terminated:
the current x satisfies the termination criteria using OPTIONS.TolX of 1.000000e-04
and F(X) satisfies the convergence criteria using OPTIONS.TolFun of 1.000000e-04
x = fminsearch(fun,[1 1],optimset('Display','iter')),
Iteration Func-count min f(x) Procedure
0 1 0
1 3 0 initial simplex
2 5 0 contract inside
3 7 0 contract inside
4 9 0 contract inside
5 11 0 contract inside
6 13 0 contract inside
7 15 0 contract inside
8 17 0 contract inside
9 19 0 contract inside
10 21 0 contract inside
11 23 0 contract inside
12 25 0 contract inside
13 27 0 contract inside
14 29 0 contract outside
15 31 0 contract outside
16 33 0 contract outside
17 35 0 contract outside
18 37 0 contract outside
19 39 0 contract outside
20 41 0 contract outside
21 43 0 contract outside
22 45 0 contract outside
23 47 0 contract outside
24 49 0 contract outside
Optimization terminated:
the current x satisfies the termination criteria using OPTIONS.TolX of 1.000000e-04
and F(X) satisfies the convergence criteria using OPTIONS.TolFun of 1.000000e-04
x = fminsearch(fun,[1 1]*1e-4,optimset('Display','iter')),
Iteration Func-count min f(x) Procedure
0 1 0.999801
1 3 0.999791 initial simplex
Optimization terminated:
the current x satisfies the termination criteria using OPTIONS.TolX of 1.000000e-04
and F(X) satisfies the convergence criteria using OPTIONS.TolFun of 1.000000e-04
