|
|
|||||||
Am I missing something stupid here? Can someone explain to me why the following code results in an output of 1.4210854715202E-014 ? Code: $num = 729.95-629.96-99.99 ? $num get $ I would expect it to be simple 0.00. |
||||||||
|
|
|||||||
Ya think? Paste the value into Excel, then format the cell to "Number" and you get "0.00". Even more strange is that Code: $N2 = 99.99 $N2 = $N2 - 99.99 $N2 ? Glenn |
||||||||
|
|
|||||||
More info: Code: $num = 729.95 - 99.99 - 629.96 From a Libre-Office bug sheet: Quote: This is not an LO bug. I get the exact same result with a Python script, a Ruby script, and Calligra Sheets. It is a problem with internal representation of decimal numbers in any computer system. This is a well known problem in scientific computing/numerical analysis. Also, reordering the addition produced different precisions. I think the correct numerical analysis term for this is precision referring to how closely the actual value is to the true value. Glenn |
||||||||
|
|
|||||||
How would you ever know you are running into this without testing every single value passed? |
||||||||
|
|
|||||||
And unfortunately the FormatNumber() in kix doesn't even fix it. Code: FormatNumber($num, 2) results in a long string of 0's, and not just 0.00 like it should. Having to use the Round() function to get around it for now. |
||||||||
|
|
|||||||
Yes, I was going to post earlier, but can't from work. It has to do with how floating point numbers are stored in memory. It is usually just an approximation. You have to keep track of your significant digits. In your case, you probably want to multiply by 1000, convert to an integer, add 0.00 plus your variable, divide by 1000 to get a more accurate answer. Code: $num=.... $num2=$num*1000 $num3=int($num2) $num4=0.00+$num3 $num5=$num4/1000 |
||||||||
|
|
|||||||
$num = 729.95 - (629.96 + 99.99) ? $num get $ Will give you the right answer. |
||||||||
|
|
|||||||
Thanks for the tips guys. Would you say that it's a bug in FormatNumber() that it doesn't format the weird number down to 2 decimal places? Don't know why FormatNumber($num, 2) doesn't work. |
||||||||
|
|
|||||||
Yes. Bug. |
||||||||
|
|
|||||||
Well if this is a math question.... |
||||||||
|
|
|||||||
Yup |
||||||||
|
|
|||||||
Dang - I knew I should have stopped at 25... |
||||||||
|
|
|||||||
LOL Nice one Doc. |