Mavericks said:

Now what interested me much was why he wasn’t to direct Jedi too? It turned out Lucas offered this but Kersh himself got out. The reasons he listed seems contradicting to me:

It took too long and was too exhausting both physically and emotionally

He felt doing two for George would be too much and he didn’t wish to be «an employee of Lucas» (Wow, interesting, isn’t it? He considered Lucas a very creative and adored his earlier work, so why in the name of art and talent not to help him more than once? Eventually, he directed not for «thank you» but for being paid/gyven credits and I doubt that doing Empire didn’t result in his bank account increase)

He didn’t want

He didn’t believe the script

It's no surprise that Kershner wanted out as after he cut the film George disagreed with it and re-cut the film (ESB) and then a 3rd cut had to be made to put back in of some of kershner's original cuts. (Secret History of Star Wars -That is where I read this. Not sure how true this is?)

How much of Empire is purely Kershner's Vision remains very unclear.

He then got the job of Robocop 2 that again was altered away from making Robocop more human and a tortured soul. Which was Kersners Desire which to be honest was I think a good call on his part. But the Studio disagreed with it and altered / ordered the footage cut out.

Kershner was of a much older ilk of film maker and he was obviously desired in some sense but also shunned in others, and perhaps did not fit with the modernistic approach that films were taking at that time. But I think that you need not look any further than not having Final cut or limited creative control is enough to walk away and say "not doing this again"

But I doubt Empire is too far different from what was planned. Robocop 2 however I think finished him off as a big budget film director.