It was among five Hitchcock films that were out of circulation for a long time. For over a decade it was difficult to even see it. That kind of thing adds to the mystique.
You won't get some of the jokes in "High Anxiety" if you haven't seen Vertigo. ;)