Frink just explained it to you. The fact that retard is interchangeable as an insult with stupid and idiot is what makes it offensive. You’re basically calling mentally retarded people stupid idiots.
Same reason calling someone gay as an insult is offensive. Because you’re basically saying gay people are uncool, stupid, etc.