Those of us working with relatively high-level languages spend most of our time able to blissfully ignore the mucky little details that go into making a program actually run. C# allows us to not think about allocating and freeing memory, for example, or how our strings are terminated. It’s all too easy to extend this willful blindness to other details, like how numeric types actually work. Can this variable ever have a fractional part? Yes? Double! No? Int! Can it be really big? Long! Done, sorted.

Recently I was writing some functions to do unit conversions. Specifically, I was converting between points (1/72 of an inch) and EMUs (1/914400 of an inch). Part of the point of EMUs is that you can use them for high-precision values without needing floating point arithmetic, so I was storing them in a long. The application in question deals with fractional points, so those went into a double. Now, what’s the largest positive point that we can convert into an EMU without any overflow? If we do a little math we see that there are 12,700 EMUs per point, so the obvious answer should be long.MaxValue / 12700.0, which comes out to a bit over 7.2E14. Excellent. However, if I turn that around and convert that many points back into EMUs, I get -9.2E18. Oh dear. Continue reading A loss of precision, but not a loss of magnitude →