I have used a thin shim. I get them from the parts dept. Can't remember what they are for off hand, but they work good. Put them between the bearing and plastic half.
Also have used Locktite Bearing Mount. Clean the bearing and plastic half, put a couple beads around the bearing and stick it in. Let it set up for awhile.
Have also put plastic facing down and use a center punch and dent the metal in a couple spots.