Whether that makes them a “racist” or not is just semantics. Personally, I’d say no, but getting caught up on the word is just missing the point.
It does matter though. Whenever people hear that they could be a racist, they have an immediate reaction of revolt, because they know that being racist means you are a bad person.
Maybe if people focus on what people could or do do, then they wouldn’t have such a visceral reaction. That helps the point you’re trying to make. How you express yourself matters.
The problem is people have been taught that racism = lynching black people, when it’s really more common, subversive, and seemingly innocuous than that. People need to learn that just because you are capable of badness doesn’t make you a bad person - intentions matter. Just because someone calls you racist doesn’t mean they think you’re the devil’s spawn.
Basically, people need to get over themselves. Maybe, instead of rejecting the accusation entirely, people should take a deeper look into why someone would call them something like that, instead of putting their fingers in their ears and pretending they are all knowing.