It seems to me that most of it stems from the belief that JK Rowling is a practicing witch or something. Where do people get their information?
That said, on the other hand I wouldn't let a very young child read Harry Potter, just like I wouldn't let a young child watch Sin City or play God of War, at least not one who was young and impressionable.
ETA: Also, I was going to mention that even though I don't think HP is evil, I'm not a fan. I don't hate it, it just doesn't interest me.