Shouldn't have anything to do with the idler unless the rest of your idlers are missing all the rubber/poly off of them. I've seen where the heat given off from the muffler will do that over time. Are your tracks set so the inside lugs are directly below the exhaust?  Also have seen that happen if the tractor is driven half on the shoulder and half on the road.

Swap the tracks side to side now and you'll be better off in the long run. Kind of like rotating tires. Somewhere I have a picture of a home made heat shield that a person installed under the muffler to help. Can't promise I will be able to find it though.