Author Topic: Puzzling double precision rounding inaccuracy  (Read 6518 times)

0 Members and 1 Guest are viewing this topic.

Offline paravantis

  • Newbie
  • Posts: 1
    • View Profile
Puzzling double precision rounding inaccuracy
« on: April 10, 2022, 03:55:59 am »
First post in this forum, pleased to be here.

I have been writing a simple program calculating a complete set of univariate statistics (more than Minitab, SPSS, etc.) In the process, I have come across a puzzling inaccuracy in reporting a rounded number.

I use the following function to round a number xx to dd decimal digits (solely for printing purposes):

Code: QB64: [Select]
  1. FUNCTION round# (xx AS DOUBLE, dd AS INTEGER)
  2.    round# = _ROUND(xx * 10 ^ dd) / 10 ^ dd

Although the function works well, it has given me a solitary strange result: trying to round the number

Code: QB64: [Select]
  1. 91.49418496688178

to 4 decimal digits, it gives

Code: QB64: [Select]
  1. 91.49420000000001

instead of the expected

Code: QB64: [Select]
  1. 91.4942

When the same function is used to round the number to fewer or more decimal digits, it works fine, without any weird trailing digits.

As an alternative, I tried using

Code: QB64: [Select]
  1. PRINT USING "##.####"

which rounds and prints the number fine.

Help would be appreciated.

Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: Puzzling double precision rounding inaccuracy
« Reply #1 on: April 10, 2022, 06:23:56 am »
You've found the solution!

Welcome to the forum!

Offline jack

  • Seasoned Forum Regular
  • Posts: 408
    • View Profile
Re: Puzzling double precision rounding inaccuracy
« Reply #2 on: April 10, 2022, 06:32:33 am »
hello paravantis
you may know or have seen discussions about the approximate nature of binary floating-point, unless the fractional part of a decimal number is a power of 2 (or a sum thereof) it will only be approximately represented in the binary form
storing a rounded number to a certain number of decimal places in a variable may not be exact, for displaying numbers I would use print using as in your last example

Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: Puzzling double precision rounding inaccuracy
« Reply #3 on: April 10, 2022, 07:32:26 am »
And as TempodiBasic says, remember it's not a bug, it's a feature.

You learn never to completely trust the computer, good lesson!