1
QB64 Discussion / Puzzling double precision rounding inaccuracy
« on: April 10, 2022, 03:55:59 am »
First post in this forum, pleased to be here.
I have been writing a simple program calculating a complete set of univariate statistics (more than Minitab, SPSS, etc.) In the process, I have come across a puzzling inaccuracy in reporting a rounded number.
I use the following function to round a number xx to dd decimal digits (solely for printing purposes):
Although the function works well, it has given me a solitary strange result: trying to round the number
to 4 decimal digits, it gives
instead of the expected
When the same function is used to round the number to fewer or more decimal digits, it works fine, without any weird trailing digits.
As an alternative, I tried using
which rounds and prints the number fine.
Help would be appreciated.
I have been writing a simple program calculating a complete set of univariate statistics (more than Minitab, SPSS, etc.) In the process, I have come across a puzzling inaccuracy in reporting a rounded number.
I use the following function to round a number xx to dd decimal digits (solely for printing purposes):
Code: QB64: [Select]
Although the function works well, it has given me a solitary strange result: trying to round the number
Code: QB64: [Select]
- 91.49418496688178
to 4 decimal digits, it gives
Code: QB64: [Select]
- 91.49420000000001
instead of the expected
Code: QB64: [Select]
- 91.4942
When the same function is used to round the number to fewer or more decimal digits, it works fine, without any weird trailing digits.
As an alternative, I tried using
Code: QB64: [Select]
which rounds and prints the number fine.
Help would be appreciated.