That isn't the only way a smartphone gets your location.

Most people don't know this, but if you connect to most any Google service via a WiFi network while your phone has GPS enabled, the location from GPS is associated with the SSID somewhere in the bowels of Google. Thereafter, merely connecting to the SSID is enough to locate you to a general area.

Your location is also known, to some extent, by which cell tower you're homed in to right now.