That's a R-8 tool holder. It goes into a R-8 chuck, which was found on Bridgeport mills and some radial drills. It can be used to hold a honkin' large drill bit, or an end mill, or annular cutter, etc. 

The set screw in the side of the tool holder is to clamp down on the Weldon Flat that has been ground into the tool shank. Here's a Weldon flat:

https://abtoolsinc.com/prod/chamfer-hogs/weldon-flat-tools/

To change from a chuck to a fixed toolholder (such as your first picture), you'll unscrew the drawbar that comes out the top of the machine above the spindle. The threads on the back of the toolholders, chuck, or collets that go into the R8 spindle will be 7/16"-20 tpi.

The gouge that runs up the side of the R8 tooling is for alignment with the spindle - there's a small "nub" inside the spindle that the gouge has to line up with in order for you to seat the toolholder or chuck completely in the spindle. Many times, if the tool has been under excessive load and not drawn down tightly enough, that "nub" in the R8 spindle will shear off.

To pull the Jacobs chuck, you'd find the spindle brake on the drill press. Make sure the press is "off," and hold the brake to clamp the spindle in position. Put a wrench on the drawbar above the spindle, unscrew the drawbar a couple of turns, then tap on the top end of the drawbar. This should pop the chuck/toolholder/collet loose from the spindle, but keep it in the spindle so it doesn't just drop onto your vise or workpiece.  Finish unscrewing the drawbar from the chuck/toolholder/collet with one hand, while you control the drop of the chuck/toolholder/collet and tool with the other hand. When you're using a collet, as soon as you back off on the drawbar, the tool will tend to drop out of the collet - if you're using carbide tooling, this is when you tend to get chipped tooling from a carbide tool dropping onto the vise/table/work.