Gaffer Tape said:

But think about it this way. Sports is absolutely no different from being a fan of anything else, be it video games, anime, or Star Wars (but I'm sure lots of sports fans would take offense at being lumped in with such "nerds" but it's true). Yet, for some reason, it's much more culturally acceptable to be an obsessive fan of sports than any of those other things.

Fixed!

I think that is that part that annoys me. Honestly, If I had people constantly telling me how awesome the new dragon boxes are, and making predictions about the next dragon box (or Star Wars, or anything for that matter), and just generally going on about the thing as if it was quite literally the most important thing in the world, and act baffled when I fail to reciprocate their enthusiasm, then you can bet I'd feel the same level of annoyance for those things as I do sports.

I guess it is the magnitude of the obsession, added with the social disapproval I am met with for not sharing that obsession, that makes me dislike it so much.