Hi Edward, 32-bit float does indeed store 23 bits of significand, but there is an additional implicit bit, making the total 24 bits. When you also consider the separate sign bit, 32-bit float has as much precision as a 25-bit signed fixed-point integer. Adjusting your math without any other adjustments, that would imply a resolution of 1.2 meters and perhaps 2.5 meters unrounded. However, I don't quite follow your calculations since floating point always carries a certain number of significant digits, regardless of the exponent. Can you show why the circumference divided by a fixed precision would reflect the ability and precision of a floating point calculation? It's a snowy day here, and I don't have my floating point thinking cap... Brian On Feb 1, 2017, at 12:38 PM, Edward Cheeseman <cheesemanedward@gmail.com> wrote:
So, given the floating point library has been pulled in anyway, I think the current implementation isn’t too bad.
Floating point accuracy: - The equatorial circumference of the Earth is 40,075,017 meters. - Divided by 23bit, resolution is 4.8meters The math is probably unrounded, so probably the best you could hope for is within 10 meters.