In Secret History of Star Wars, Michael Kaminski hypothesizes that Lucas came up with the idea to make Vader into Luke’s father after Brackett completed the first draft (and sadly died). Kaminski speculates that Lucas’ thought process leading up to this decision was as follows: Lucas read Brackett’s draft and something bothered him about the scenes where Luke’s father appears as a force ghost. Lucas realized that Luke’s father (who was a typical wise old Jedi character) was actually a completely redundant character, because he’s basically the exact same character as Ben Kenobi (who is also a Force Ghost at this point). But once the concept of force ghosts was introduced, it would follow logically that Luke’s father should begin appearing to Luke, since Kenobi always does. So now Lucas was stuck with duplicate “wise old Jedi” characters. If Luke’s father started appearing to Luke, there really would be no reason for Kenobi to have any further role in the story. So eventually, while struggling to find a solution to this problem, Lucas stumbled upon the greatest movie twist in history.
This is fascinating, and makes a lot of sense.
This hypothesis seems believable to me. It’s purely a guess, but it makes sense. But on the other hand, there is some evidence that Lucas actually decided that Vader was Luke’s father even before Leigh Brackett wrote the first draft. Kaminski points out how ridiculous this claim seems, because it means Lucas purposely withheld vital story information from Brackett, even while paying her to write the script (presumably out of paranoia of leaking the twist). And yeah, this does seem kind of ridiculous - but maybe it’s more like Lucas was just indecisive about the idea. Apparently it might actually be true, at least according to JW Rinzler, who claims to have many of Lucas’ handwritten preliminary notes for Empire Strikes Back, seemingly pre-dating Brackett’s draft. One of these note pages has the words “Father changes into Darth Vader”. But unfortunately Rinzler doesn’t provide a photograph of this (but provides many photographs of other notes), and Rinzler also admits that none of these notes are dated so it’s difficult to construct a precise timeline of Lucas’ thought process.
Could the words ‘Father changes into Darth Vader’ be a reference to a Force vision, much like Luke’s Dagobah Cave experience? In the finished film Luke defeats Vader and his face turns into Luke’s face, but perhaps there was a version where Luke’s father appears to him in a dream or vision but then changes into Vader to represent the revenge that Luke feels that he needs to take.
I suggest this alternate explanation merely because if I suddenly had the idea to combine the two characters, I would simply write ‘Father is Darth Vader’, or ‘Father changed to Darth Vader’. It’s very strange to use the present form of the word ‘changes’ since it suggests something which happens within the film, rather than being a revelation of an unchanging fact.