Author Topic: How do you deal with floating point rounding errors ?  (Read 12974 times)

0 Members and 1 Guest are viewing this topic.

Offline CharlieJV

  • Newbie
  • Posts: 89
    • View Profile
Re: How do you deal with floating point rounding errors ?
« Reply #30 on: April 03, 2022, 12:26:15 pm »
For anybody who wants to do some extra reading on his/her down time:  Google Search for "0.1 0.2 not equal 0.3"

It is pretty interesting stuff that I wish my digital design course (from university many moons ago) had covered.

Offline bplus

  • Global Moderator
  • Forum Resident
  • Posts: 8053
  • b = b + ...
    • View Profile
Re: How do you deal with floating point rounding errors ?
« Reply #31 on: April 03, 2022, 01:11:37 pm »
Yeah there's STx paper right there:
Quote
Computers don't usually work in base 10, they work in base 2.

So consider yourself warned. :)