  # FAQ: Great Circle Calculations

Q: How can I calculate great circle distances?
A: Earth's hypothetical shape is called the geoid and is approximated by an ellipsoid or an oblate sphereoid. A simpler model is to use a sphere, which is pretty close and makes the math MUCH easier. Assuming a sphere of radius 6371.2 km, convert longitude and latitude to radians (multiply by pi/180) and then use the following formula:
theta = lon2 - lon1
dist
= acos(sin(lat1) × sin(lat2) + cos(lat1) × cos(lat2) × cos(theta))
if (dist < 0) dist = dist + pi
dist = dist × 6371.2
The resulting distance is in kilometers.
Q: Does the Great Circle Mapper assume Earth is a sphere?
A: No, the Great Circle Mapper uses the same WGS 84 reference ellipsoid used by the Global Positioning System (GPS). The WGS 84 ellipsoid has a polar radius of 6356.752 km and an equatorial radius of 6378.137 km.
Q: How does the Great Circle Mapper calculate distances and paths?
A: The Great Circle Mapper uses formulae developed by Thaddeus Vincenty to perform geodesic calculations with modifications by Charles Karney based on work by Friedrich Bessel to improve performance and accuracy.

(The original Great Circle Mapper used a modified version of the geod program from the PROJ.4 cartography tools. This program uses an older, less-sophisticated algorithm developed by Paul Thomas of the USGS.)

References:

Q: Why does the example in Ed Williams' Aviation Formulary calculate a different distance for LAX-JFK?
A: Ed Williams' Aviation Formulary has an example which calculates this distance from LAX to JFK as 2144 nm while the Great Circle Mapper says LAX to JFK is 2151 nm. This comes from several simplifying assumptions in the example:
1. In the example, the locations for LAX and JFK are rounded to the nearest arc-minute of longitude and latitude, or 1/60th of a degree. The Great Circle Mapper uses data from the FAA which is accurate to 0.000001 degree.
2. The formulae used in the example assume that Earth is a sphere.

 Information on this site may not be accurate or current and is not valid for flight planning or navigation. No warranty of fitness for any purpose is made or implied. Flight planning and navigation should only be done using official charts. Copyright © 1996-2019 Karl L. Swartz. All rights reserved. The Great Circle Mapper name and logo are trademarks of the Great Circle Mapper. All other trademarks mentioned herein belong to their respective owners. Please see credits for attibutions and further copyright information.   