That sounds logical, and probably holds up to some extent, but that filter media has seen a lot of air pass through and stood a lot of vibration and occasional moisture.  I don't really think leaving it in service indefinitely is a good plan.   

I generally replace them both if I am replacing...but that's partially based on the knowledge that it will be a cold day in hell before my brother replaces an innner one, so someone has to do it sometimes!  lol