There is only one explanation… Lucas’s artistic decisions stopped making logical or coherent sense in the mid 80’s. Maybe he ran out of midi-chlorians.
http://cinetropolis.net/star-wars-splinter-of-the-minds-eye-story-conference/
I think he was mostly out of ideas by 1976.