When a person of color says, "Hey, so-and-so said a racist thing," they're seen as making a personal attack. When a white person says, "You're not smart enough for your opinion to count," they're seen as arguing the issues.

I would say that the former is not a personal attack but the latter is. Accusing someone of racism is different from calling someone [insert expletive here] for being racist. And saying that "you're not smart enough" places similar emphasis on the you, personally, rather than your actions.