Well, the Globetrotters didn't show up until Castaways from Gilligan's Island, after they'd already been rescued and turned the island into a resort, so they don't really fit into the category of "assholes who discovered them and left them for dead."

EDIT: But off the top of my head, the biggest jerks I think was that Beatles wannabe band (I can't even remember their name) who were going to rescue them until the girls put on a concert, and they suddenly got afraid of the competition and took off! Man, those people suck!