The color of your skin matters. If everything were equal, it wouldn’t matter, but it isn’t so it doesn’t.
If the worst thing your skin color has brought you is flak when you criticize black people, consider yourself lucky. Or continue to cling to the white privilege that is so important to you, despite the fact that it doesn’t matter in the slightest.