I didn't read all the previous replies, so if I'm a repeat, I apologize.

The first thing that strikes me in what you've described is what appears some real difficulty in communicating.

If you and your son have a tough time simply asking/answering whether you both think that his coming back to the farm as a life choice is a good or bad idea, communicating while working together is gonna be a real up-hill pull.

Can you not just say, "well, son, what would be your thoughts on coming back to the farm and working into the operation with me? Do you have any interest in in that?" 

If this notion puts you off, or is hard for you to imagine, then I'd think the first thing you both need to do is find some help in learning to communicate well. It will help no matter what decisions you make later on.

Good luck to you both!