Active Forums > QB64 Discussion

Puzzling double precision rounding inaccuracy

(1/1)

paravantis:
First post in this forum, pleased to be here.

I have been writing a simple program calculating a complete set of univariate statistics (more than Minitab, SPSS, etc.) In the process, I have come across a puzzling inaccuracy in reporting a rounded number.

I use the following function to round a number xx to dd decimal digits (solely for printing purposes):


--- Code: QB64: ---FUNCTION round# (xx AS DOUBLE, dd AS INTEGER)   round# = _ROUND(xx * 10 ^ dd) / 10 ^ ddEND FUNCTION
Although the function works well, it has given me a solitary strange result: trying to round the number


--- Code: QB64: ---91.49418496688178
to 4 decimal digits, it gives


--- Code: QB64: ---91.49420000000001
instead of the expected


--- Code: QB64: ---91.4942
When the same function is used to round the number to fewer or more decimal digits, it works fine, without any weird trailing digits.

As an alternative, I tried using


--- Code: QB64: ---PRINT USING "##.####"
which rounds and prints the number fine.

Help would be appreciated.

bplus:
You've found the solution!

Welcome to the forum!

jack:
hello paravantis
you may know or have seen discussions about the approximate nature of binary floating-point, unless the fractional part of a decimal number is a power of 2 (or a sum thereof) it will only be approximately represented in the binary form
storing a rounded number to a certain number of decimal places in a variable may not be exact, for displaying numbers I would use print using as in your last example

bplus:
And as TempodiBasic says, remember it's not a bug, it's a feature.

You learn never to completely trust the computer, good lesson!

Navigation

[0] Message Index