I got it out.

I did the notch trick with my trusty dremel. Aside from the nick in the castor block now, it worked like a charm.
I'm not sure why I didn't think of the notch trick before...I've had to do it before also. Thanks what's-nitro for reminding me.