Actually, given the number of really stupid Family Guy fans, and the vast number of references to literature, classic film, history, and high art, that such ignorant fans are unlikely to have ever come into contact with, I am thinking it succeeds perfectly fine whether people get the references or not. I find Family Guy's tendency to reference works much more intelligent and worthwhile than itself as its lone redeeming factor, in my eyes. Anything that can make brainless potheads or culturally challenged high schoolers recite quotes from Casablanca or sing tunes from My Fair Lady gets at least a mild applause from me.

Again, I really think you guys are using the term "fail" quite incorrectly. The show is insanely popular and very successful.