I don’t know, the majority thing sounds kind of arbitrary.
I consider myself a Star Trek fan (though not a Trekkie or whatever) and I only like TOS and the movies. I think part of it has to do with the OT being the original. Not a great comparison but you can be a Jaws fan and only like the first film.
If it comes down to others’ perception, then I think when most people think of SW they first think of the OT. The books/comics/games etc. don’t even come to mind. With the PT, I think many will assume there’s a good chance they’re not even a factor. When it comes to the new movies, consider this common exchange nowadays: “I’m a Star Wars fan,” “Oh, really? What did you think of the new one(s)?”
If you don’t personally feel comfortable calling yourself a Star Wars fan that’s totally fine but I don’t think being just an OT fan and calling yourself a Star Wars fan sends as wrong of a message as you think.
If necessary you can just call yourself an “original” Star Wars fan and people will likely understand completely what that means.