FAQ: Great Circle Calculations

Q: How can I calculate great circle distances?

A: Earth's hypothetical shape is called the geoid and is approximated by an ellipsoid or an oblate sphereoid. A simpler model is to use a sphere, which is pretty close and makes the math MUCH easier. Assuming a sphere of radius 6371.2 km, convert longitude and latitude to radians (multiply by pi/180) and then use the following formula:

theta = lon₂ - lon₁
dist = acos(sin(lat₁) × sin(lat₂) + cos(lat₁) × cos(lat₂) × cos(theta))
if (dist < 0) dist = dist + pi
dist = dist × 6371.2

The resulting distance is in kilometers.

Q: Does the Great Circle Mapper assume Earth is a sphere?

A: No, the Great Circle Mapper uses the same WGS 84 reference ellipsoid used by the Global Positioning System (GPS). The WGS 84 ellipsoid has a polar radius of 6356.752 km and an equatorial radius of 6378.137 km.

Q: How does the Great Circle Mapper calculate distances and paths?

A: The Great Circle Mapper uses formulae developed by Thaddeus Vincenty to perform geodesic calculations. (See also Vincenty's original paper.)

Q: Why does the example in Ed Williams' Aviation Formulary calculate a different distance for LAX-JFK?

A: Ed Williams' Aviation Formulary has an example which calculates this distance from LAX to JFK as 2144 nm while the Great Circle Mapper says LAX to JFK is 2151 nm. This comes from several simplifying assumptions in the example.

First, in the example the locations for LAX and JFK are rounded to the nearest arc-minute of longitude and latitude, or 1/60th of a degree. The Great Circle Mapper uses data from the FAA which is accurate to 0.000001 degree.

Second, the formulae used in the example assume that Earth is a sphere.

Back to main FAQ page