Caleb, clarify please.......

you mention "talks about buying into the business". 

Who is doing the talking?  Are you talking to him?  or is he talking with you?   There is a difference.  And I think it would be good if you could clarify that point. 

It very well might be that you have been talking to  the owner - your boss- and his interest level is not where your interest level is.  You need to be very honest with yourself about this.   I don't know details - I can only read what you wrote, and I am just making a comment.

BigNorsk makes a very good point.  Maybe you need to take a look at an ag job.  People with ag skills - especially with farm backgrounds - are in demand.  You may need to move - I don't know what it is like in your area for ag jobs.  But I would suspect that hours would be more ameniable to your family. It's something worth looking at.