OK, so you want to solve 3 coupled equations of motion for some harmonically varying solution. First you have to understand the physics, then the mathematics and then you can solve this numerically.
Eq of motion:
Harmonic solutions ->
For a number of coupled oscillators you get
Where A has to be n-by-n, where in your case n is 3.
Here you should be able to identify your first problem, since A becomes 3-by-2!
You should be able to figure out what goes wrong in your assignment of A, it has to do with the interpretation of which terms are added(or subtracted) together to form a matrix-element. To not have that happen you should insert , (commas) between to separate the matrix-elements on a row. This is a very treacherous error we all(?) encountered when starting to use matlab.