Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - paravantis

Pages: [1]
1
QB64 Discussion / Puzzling double precision rounding inaccuracy
« on: April 10, 2022, 03:55:59 am »
First post in this forum, pleased to be here.

I have been writing a simple program calculating a complete set of univariate statistics (more than Minitab, SPSS, etc.) In the process, I have come across a puzzling inaccuracy in reporting a rounded number.

I use the following function to round a number xx to dd decimal digits (solely for printing purposes):

Code: QB64: [Select]
  1. FUNCTION round# (xx AS DOUBLE, dd AS INTEGER)
  2.    round# = _ROUND(xx * 10 ^ dd) / 10 ^ dd

Although the function works well, it has given me a solitary strange result: trying to round the number

Code: QB64: [Select]
  1. 91.49418496688178

to 4 decimal digits, it gives

Code: QB64: [Select]
  1. 91.49420000000001

instead of the expected

Code: QB64: [Select]
  1. 91.4942

When the same function is used to round the number to fewer or more decimal digits, it works fine, without any weird trailing digits.

As an alternative, I tried using

Code: QB64: [Select]
  1. PRINT USING "##.####"

which rounds and prints the number fine.

Help would be appreciated.

Pages: [1]