One of the comics pretty much explained this. When they find a baby, and I mean an infant, that is strong in the Force, they convince the parents that the child will probably lead a better life among the Jedi. The children are not taken by force. Anyway, once they've been taken, they're trained in the ways of the Jedi with no further contact allowed to their parents. This is to prevent attachments so they don't freak out and go all dark side like Anakin did if their parents are killed or just die.

I believe it's also discussed in a novel somewhere. Essentially, they don't usually train anybody older than a year or so. 6 months and younger are preferred.

I did read an interesting message on this over at TFN years ago before AOTC came out. Since Jedi aren't allowed to marry or have children, the "Jedi gene" should die out. If Lucas had left it as an ability that could be learned, everything would be fine. But since it seems to be genetic, Jedi would basically make themselves go extinct since there's no way to pass on that genetic code.