FAQ: Great Circle Calculations

Q: How can I calculate great circle distances?

A: Earth's hypothetical shape is called the geoid and is approximated by an ellipsoid or an oblate sphereoid. A simpler model is to use a sphere, which is pretty close and makes the math MUCH easier. Assuming a sphere of radius 6371.2 km, convert longitude and latitude to radians (multiply by pi/180) and then use the following formula:

theta = lon₂ - lon₁
dist = acos(sin(lat₁) × sin(lat₂) + cos(lat₁) × cos(lat₂) × cos(theta))
if (dist < 0) dist = dist + pi
dist = dist × 6371.2

The resulting distance is in kilometers.

Q: Does the Great Circle Mapper assume Earth is a sphere?

A: No, the Great Circle Mapper uses the same WGS 84 reference ellipsoid used by the Global Positioning System (GPS). The WGS 84 ellipsoid has a polar radius of 6356.752 km and an equatorial radius of 6378.137 km.

Q: How does the Great Circle Mapper calculate distances and paths?

A: The Great Circle Mapper uses formulae developed by Thaddeus Vincenty to perform geodesic calculations with modifications by Charles Karney based on work by Friedrich Bessel to improve performance and accuracy. The handling of near-antipodal cases where Vincenty's inverse method fails to converge are handled as described in the Featured Map for 4 August 2023: Near Antipodes.

(The original Great Circle Mapper used a modified version of the geod program from the PROJ.4 cartography tools. This program uses an older, less-sophisticated algorithm developed by Paul Thomas of the USGS.)

References:

• Bessel, Friedrich Wilhelm. Über die Berechnung der geographischen Längen und Breiten aus geodätischen Vermessungen. Astronomische Nachrichten, October 1825.
• Karney, Charles. Bessel and the geodesic problem. Archives of the Proj (PROJ.4) mailing list, April 2009.
• Karney, Charles F.F. Algorithms for geodesics. J Geod 87, 43-55 (2013). 26 June 2012.
• Thomas, Paul D. "Spheroidal Geodesics, Reference Systems, and Local Geometry," U.S. Naval Oceanographic Office, p. 162, Engineering Library 526.3 T36s, 1970.
• Vincenty, Thaddeus. Direct and Inverse Solutons of Geodesics on the Ellipsoid With Application of Nested Equations. Survey Review, April 1975.

Q: Why does the example in Ed Williams' Aviation Formulary calculate a different distance for LAX-JFK?

A: Ed Williams' Aviation Formulary has an example which calculates this distance from LAX to JFK as 2144 nm while the Great Circle Mapper says LAX to JFK is 2151 nm. This comes from several simplifying assumptions in the Aviation Formulary example:

1. In the example, the locations for LAX and JFK are rounded to the nearest arc-minute of longitude and latitude, or 1/60th of a degree. The Great Circle Mapper uses data from the FAA which is precise to 0.000001 degree.
2. The formulae used in the example assume that Earth is a sphere.

Back to main FAQ page