You've got two things to consider:
1) The best time to get into a business sector is when it is down, PROVIDED you think it will soon turn up again.  How clear is your crystal ball?
2) The industry has pretty well left the small herds in the dust, to the point that many markets simply don't want producers who can't fill a trailer on their own.  That scale requires hired staff, larger barns, and debt, lots of debt, unless you have a good amount of cash on hand.

My suggestion is for you to find a bank that specializes in ag loans, find a loan officer there you feel comfortable working with, and lay out the entire situation for them to evaluate.  Be assured that you will have to do it eventually, so may as well get started now.  If nothing else, it will give you a good idea of just where you stand.

The single  most important asset on the farm is the cows.  I know, the seminar experts all say it is the manager, but without cows he doesn't have anything to manage, and cows are what produce the milk check.  One possible alternative is to buy heifer calves, raise them to sell, and at some time keep a bunch back to milk, once you figure out a way to make it work.  The main drawback to that is that when times are bad, no one wants the heifers badly enough for you to make a profit.

Take a hard look at where you want to be in forty years, because it will take at least that long for you to get free and clear if you aspire to having anything but a minimal operation with worn out equipment.

One final piece of advice; take it for what it is worth.  Promises from your family that are not in writing are worse than worthless.  They will keep you in limbo, and it is highly unlikely you will end up with anything remotely similar to what was promised.  If you go into business, you better be prepared to do it on your own.