I think the notion that Spielberg was going to direct Jedi is a bit overstated. I am sure Lucas considered it for a time, but clearly he didn't feel too strongly about it considering he let something as silly as a grudge against the DGA dissuade him from choosing him. I mean, it wasn't because Lucas had some sort of social crusade to root out corruption from unions, he was simply just sticking it to them because they fined him a couple thousand bucks (imagine: a multi-billionaire tycoon upset over a couple thousand bucks for genuinely breaking established union rules for the protection of its members' credibility which all directors and filmmakers also follow).
So, considering this is the freaking director of the film, the guy that will be on set doing most the work and basically controlling large parts of the film, who is supposed to lend his artistic hand in the most important role a movie production can have, this is an extremely petty, stupid, ultimately unrealistic reason to bypass someone. If you truely say, "this is the guy, the film needs and deserves his talent" you will fight for him.
Personally, I think it's all just a convenient excuse. Lucas didn't want Spielberg on it for the same reason he didn't want Spielberg to direct the prequels: Lucas wanted to control it, and he knew he couldn't control Spielberg the way he could control someone like Marquand. Indy, fine, Lucas was slightly removed from that since it was a joint project, but Star Wars by then was viewed by him as his, and he wanted to be sure that he could do what he wanted and not have the professional--and personal--tension of having to fight a respected co-worker whom he was about to work with again on two more Indy sequels (as per the contract of that other franchise). This is the exact same thing as in the prequels, which Lucas openly admitted to as CO pointed out. So when some bullshit about union grudges made things not so straightforward, it allowed him a convenient out, even though part of him did want Spielberg because Spielberg was a fucking great director. "Yeah, sorry buddy, you know I would love to have ya but you know, goddamn unions, blah blah blah. Oh well, we'll see each other on the next Indy in 1983."