I can't remember the exact formula, but I don't think it's only based on polls. There are also computer ratings that take into account strength of schedule and won-loss record, but the whole thing is still a joke.
I have a passing interest in college football, but college basketball is really where it's at for me. And the tournament is a large reason for that.