Congratulations! You have won the award for being the 1 millionth person to ask this question! The second place award is an all expense paid trip to Hackensack New Jersey. First place is, well, who could possibly beat that second place award? ;-} Who would want to beat it?
You have a system of 4 equations in 3 unknowns. In general, there is no exact solution. But then we see the first three equations actually form a homogeneous linear sytem. This means there is no constant term. As such, the obvious solution is l2=n2=m2.
Only if that system is singular will a non-trivial solution exist.
syms l2 m2 n2
eq1=38.1966*l2==0;
eq2=100*n2-61.8034*m2==0;
eq3=100*m2-161.8034*n2==0;
eq4=l2^2+m2^2+n2^2==1;
[A,b] = equationsToMatrix(eq1,eq2,eq3,[l2,m2,n2])
As I said, ONLY if A is a singular matrix does a non-trivial solution exist.
rank(A)
And therein lies your downfall. NO solution besides the trivial one where all variables are zero can exist for the 3x3 problem. Then wanting to enforce the requirement that the sum of squares of the unknowns is 1 is impossible. Case closed. Well, almost closed, but there is a sneaky way out.
A common problem that people do not understand is they should NOT pose constants that are accurate to only a few dignificant digits. That frequently causes numerical problems, but subtle ones they don't understand because they don't understand numerical analysis.
vpa(svd(A),15)
In fact, A is ALMOST singular, which is very much like being ALMOST pregnant. The deviation from singularity is seen to be in your failure to make the coefficients of n2 and m2 in equations 2 and 3 essentially reciprocals of each other. (Then scaled by 100.) For me to have said A is singular would have required the third element of that svd call to be essentially zero compared to the others. And while it is close, as I said before about almost...
So can we find a solution to that system that is as close as possible to being a solution, given the inaccurate choice of constants in your problem? Sigh, yes. The trick is to use the third singular vector of A. That is...
[U,S,V] = svd(A);
lmn = V(:,3)
Since svd automatically scales the singular vectors to have norm 1, eq4 is trivially satisfied.
This is the closest possible solution to your problem. Any other set of values for l2, n2, and m2 will be a poorer solution. If I recall, the solution given here is also the one you were expecting, or at least as close as I can come given the actual coefficients you posed.
Finally, I might point out that had you used a better (sufficiently accurate) set of coefficients in the original problem, then the function null would have been an appropriate tool for your purposes.
