Text Only | Text with Attachments

QB64.org Forum

Active Forums => QB64 Discussion => Topic started by: paravantis on April 10, 2022, 03:55:59 am

Title: Puzzling double precision rounding inaccuracy
Post by: paravantis on April 10, 2022, 03:55:59 am
First post in this forum, pleased to be here.

I have been writing a simple program calculating a complete set of univariate statistics (more than Minitab, SPSS, etc.) In the process, I have come across a puzzling inaccuracy in reporting a rounded number.

I use the following function to round a number xx to dd decimal digits (solely for printing purposes):

Code: QB64: [Select]
  1. FUNCTION round# (xx AS DOUBLE, dd AS INTEGER)
  2.    round# = _ROUND(xx * 10 ^ dd) / 10 ^ dd

Although the function works well, it has given me a solitary strange result: trying to round the number

Code: QB64: [Select]
  1. 91.49418496688178

to 4 decimal digits, it gives

Code: QB64: [Select]
  1. 91.49420000000001

instead of the expected

Code: QB64: [Select]
  1. 91.4942

When the same function is used to round the number to fewer or more decimal digits, it works fine, without any weird trailing digits.

As an alternative, I tried using

Code: QB64: [Select]
  1. PRINT USING "##.####"

which rounds and prints the number fine.

Help would be appreciated.
Title: Re: Puzzling double precision rounding inaccuracy
Post by: bplus on April 10, 2022, 06:23:56 am
You've found the solution!

Welcome to the forum!
Title: Re: Puzzling double precision rounding inaccuracy
Post by: jack on April 10, 2022, 06:32:33 am
hello paravantis
you may know or have seen discussions about the approximate nature of binary floating-point, unless the fractional part of a decimal number is a power of 2 (or a sum thereof) it will only be approximately represented in the binary form
storing a rounded number to a certain number of decimal places in a variable may not be exact, for displaying numbers I would use print using as in your last example
Title: Re: Puzzling double precision rounding inaccuracy
Post by: bplus on April 10, 2022, 07:32:26 am
And as TempodiBasic says, remember it's not a bug, it's a feature.

You learn never to completely trust the computer, good lesson!
Text Only | Text with Attachments