Pay off all debt outside of your mortgage. If your mortgage rate is below 3.75 ish and fixed for the next 15 or more years I would not pay it off. If you can't sit down with your banker, explain your goals and get his input on how to move forward get another banker who is interested in working with you. Most regional banks have an investment service.
 If the banker is giving you input  but not agreeing with you maybe you need to consider his advice.....or maybe not.
 I think land prices will trend lower and interest rates will inch higher over the next 5 years fwiw.