If you want to be a casual fan, I agree with either what Frink or georgec said. Ender's Game itself is the crux of the series and you must read it. If you want a very different, less action-based, more thought-provoking follow-up, Speaker for the Dead is really an enjoyable book. After that, the books following Ender's story became pretty weird and I can definitely understand why they wouldn't appeal to everyone. If you wanted to stick with the action, you would probably like to follow up Ender's Game with Ender's Shadow, which covers many of the same events from a different character's POV (a little boy named Bean). I enjoy the book as an independent story better than as a parallel story because I feel that Shadow undercuts Game somewhat, and it makes Ender a weaker, less likable character. The books following Bean are pretty interesting in my mind and a little more fast paced, but again, I could see why the sequels are not as well received. In my opinion, they do have broader appeal than most of the Ender books.
There are a couple of new Ender books squeezed in there long after the fact, namely A War of Gifts and Ender in Exile, both of which take place in Ender's youth. To fully appreciate Ender in Exile, you must read the Bean series almost completely, as some events tie them together. This aspect of the story really didn't appeal to me, and I think that book could have wrapped itself up quite well a couple of chapters early and would have been better. But in any case, it's hard to include that story in the chronology, and would probably fit better in order of release.
I've written reviews on the whole series in this thread and I've tried to keep each one spoiler free, but if you know nothing about the series, particularly any surprises, I'd almost prefer you wait until you've read Ender's Game itself before reading any of them. After that I wouldn't mind as much. I hope you enjoy the series. I really enjoy every book to some extent, though some more than others.
