Monday, August 14, 2006

Sometimes I have a hard time remembering what it was like before I learned something. The Register has a write-up about common programming mistakes that occur with fractional numbers. Apparently lots of organizations that should know better have forgotten that computers don't really count like people do.

Any time a computer uses a floating point or fractional number, there is a possibility of rounding behind the scenes. There are some cool programming libraries that deal with this (remember significant digits from science classes?). But the easiest trick, especially when working with money, is to stop thinking of $1.20 as 1.2 dollars, but rather as 120 cents. Computers handle whole numbers perfectly well.

I can't remember when I learned that, but I'm amazed to see a pension fund miscalculate things because it didn't know this little secret.

0 Comments:

Post a Comment

<< Home