He doesn't care about the people who don't like what he's doing.
He still remembers the fans who made him what he is today, he just doesn't put much stock in the ones who turned their back on him since ESB or ROTJ. To him they are in the same pile he put the people who said the O-OT sucked.