Hmmm... that sounds like it took some of the paint with it then. Im no body shop repair guy so im speculating at best here. I would say its safe to assume its too deep to buff out if it has that feeling to it. It prolly went down past the clear coat which is what your smooth out when buffing.
If you want some better advice, pm Travisafx. He does body work and could direct you better.
That or swing by a body shop tomorrow. Sorry I can't be of more help to you.