Irvin Kershner turned down directing Empire Strikes Back. Lucas had to twist his arm to get him to do it; they had to have a second meeting where Lucas basically begged him to make the film. It turned out...pretty good, to put it lightly. To make things worse, the film he did right before Empire, Eyes of Laura Mars, sucked. Even Kershner apologized for it. And the rest of his films up until then were fairly obscure. I've seen "Hoodlum Priest," but have you? If the internet was around in 1978 we'd all be crucifying him.
A lot of people are looking for things to nail Abrams on but honestly none of us has any idea how well he will work out. I give him the benefit of the doubt until I see the movie, as he is not a bad director by any means.