# precision problem ? why ans for 10 ^ 5 * 0.0633 is 6.3299e+03

22 views (last 30 days)
andy on 26 Feb 2021
Edited: James Tursa on 26 Feb 2021
anyway to fix this ?
>> 10 ^ 5 * 0.0633
ans =
6.329999999999999e+03
>> 10 ^ 6 * 0.0633
ans =
6.329999999999999e+04

KALYAN ACHARJYA on 26 Feb 2021
Edited: KALYAN ACHARJYA on 26 Feb 2021
>> 10 ^ 5 * 0.0633
ans =
6.3300e+03
>> format shortG
>> 10 ^ 5 * 0.0633
ans =
6330
See the other Format options, you can set the different format the output display
Steven Lord on 26 Feb 2021
Using a different display format masks the issue.
andy, can you post for me the exact decimal value of one divided by three? The only characters you're allowed to use to write that answer are the digits 0 through 9 and a single . for a decimal point. You're not allowed to say that the answer is anything repeating; you must write all the decimal places of your answer explicitly.
Now multiply the answer you gave me by three. The result will not be equal to one. If you had been able to write an infinite number of decimal places it would have been, but you didn't. You rounded off one divided by three and that roundoff error affected the answer.

James Tursa on 26 Feb 2021
Edited: James Tursa on 26 Feb 2021
Normal floating point arithmetic effects. See this link:
The root issue is that 0.0633 cannot be represented exactly in IEEE double floating point format, and downstream calculations using this will be affected as a result. Calculations that mathematically should result in a known value often do not. You need to write floating point code that is tolerant of these effects. E.g., this is the closest number to 0.0633 that IEEE double precision can represent exactly
>> num2strexact(0.0633,'fixed')
ans =
'0.0632999999999999951594276126343174837529659271240234375'