Whether that makes them a “racist” or not is just semantics. Personally, I’d say no, but getting caught up on the word is just missing the point.
It does matter though. Whenever people hear that they could be a racist, they have an immediate reaction of revolt, because they know that being racist means you are a bad person.
Maybe if people focus on what people could or do do, then they wouldn’t have such a visceral reaction. That helps the point you’re trying to make. How you express yourself matters.