Sqlalchemy: Latitude and Longitude Float Precision? - database

Question

I'm using Sqlalchemy to define my tables and such and here is some code I came up with:
locations = Table('locations', Base.metadata,
Column("lat", Float(Precision=64), primary_key=True),
Column("lng", Float(Precision=64), primary_key=True),
)
I read somewhere that latitude and longitude require better precision than floats, usually double precision. So I set the precision manually to 64, is this sufficient? Overkill? Would this even help for my situation?

Answer 1

Nobody else here provided concrete numbers with proof for the worst-case accuracy of a floating point lat/long. I needed to know this for something I was working on, so here is my analysis in case it helps someone else.
A single-precision floating point offers 24-bits of precision in the significand (the binary exponential notation of a number). As the whole part of the number gets larger, the number of bits after the decimal goes down. Therefore, the worst-case accuracy for a latitude or longitude is when the magnitude is as far away from 0 as is possible. Assuming you bound your latitudes to [-90, 90] and longitudes from (-180, 180], the worst-case will be at the equator for longitude 180.
In binary, 180 requires 8-bits of the 24-bits available, leaving 16 bits after the decimal point. Therefore, the distance between consecutively representable values at this longitude would be 2^-16 deg (approximately 1.526E-5). Multiplying that number (in radians) by the WGS-84 radius of the Earth at the equator (6,378,137 m) yields a worst-case precision of:
2^-16 deg * 6,378,137 m * PI rad / 180 deg = 1.6986 m (5.5728 ft).
The same analysis against lat/longs stored in radians yields the following:
2^-22 rad * 6,378,137 m = 1.5207 m (4.9891 ft)
And finally, if you normalize the latitudes to the range [-1, 1] and the longitudes to the range (-1, 1], then you can achieve the following worst-case precision:
2^-24 * PI rad * 6,378,137 m = 1.1943 m (3.9184 ft)
So storing lat/long in radians buys you around 7 inches of additional accuracy, and storing them in normalized form buys you around 1'8" of additional accuracy, both in the worst-case scenario.
If, when converting between double-precision and single-precision you rounded (instead of truncating), the single-precision value will be within half of the distance between two consecutive values computed above.

Answer 2

It depends on what you are using your data for. If you use a float it will be ok if the you only need it down to about the meter level of detail. Using the data in graphically applications will cause a jitter effect if the user zooms in to far. For more about jitter and see Precisions, Precisions. Hope this helps.

Answer 3

Update: Jeff's answer has a better analysis. However...
To improve upon Jeff's answer:
If you divide the actual angle in radians by π, thus encoding the angle in a scale going from 0 to ±1, then it should be possible to use all the digits of the significand (23 bits (24 - 1 sign bit)). The precision would then be:
2^-23 * 6,378,137 m = 0.7603 m (76 cm)
My Old answer:
A 32 bit floating point number can represent a number with about 7.2 decimal digits of precision. This is an approximation because the floating point number is actually in binary, and when converted to decimal, the number of significant digits might vary.
If we take it as 6 decimal digits of precision (to play on the safe side), and if we are storing latitude and longitude in degrees, then we get a precision of about 1/1000th of a degree which is a precision of about 111 meters in the worst case. In the best case, if we get 7 decimal digits of precision, the accuracy would be about 11.1 meters.
It is possible to get a better precision using radians as the unit. In the worst case we get a precision of 10 millionth of a radian which is about 63 meters. In the best case, it would be 1 millionth of a radian which is about 6 meters.
Needless to say, a 64bit floating point number would be extremely precise (about 6 micro meters in the worst case).

Answer 4

TL;DR: if one-meter resolution is acceptable then a single-precision float storing degrees is acceptable.
This answer is a bit late to the party but I needed a solid answer myself and so hacked out some code to quickly get it. There are of course more elegant ways to do this, but it looks to work. As noted by Jeff, the worst case scenario will be at +/- 180 degrees longitude (ie, the date line).
Per the code below, a single-precision float is accurate to 0.85 meters at the date line using single-precision floats storing degrees. Accuracy increases significantly (to w/in mm) when close to the Prime meridian.
#include <stdio.h>
// per wikipedia, earth's circumference is 40,075.017 KM
#define METERS_PER_DEG (40075017 / 360.0)
// worst case scenario is near +/-180.0 (ie, the date line)
#define LONGITUDE 180.0
int main()
{
// subtract very small but increasingly larger values from
// 180.0 and recast as float until it no longer equals 180.0
double step = 1.0e-10;
int ctr = 1;
while ((float) LONGITUDE == (float) (LONGITUDE - (double) ctr * step)) {
ctr++;
}
double delta = (double) ctr * step;
printf("Longitude %f\n", LONGITUDE);
printf("delta %f (%d steps)\n", delta, ctr);
printf("meters: %f\n", delta * METERS_PER_DEG);
return 0;
}
Output from this code is
Longitude 180.000000
delta 0.000008 (76294 steps)
meters: 0.849301