Isn't Geo-Caching littering?

A friend of mine was explaining this to me as well over the weekend - how there is different clues and stuff dropped off in places.  Sure its nice to get fresh air, but I think its nothing more then littering.  If your putting objects in a place, especially man-made objects, then its littering.