Apparently Prowse wasn't even told that this was the line, that instead, the line was "Obi-Wan killed your father." So this is what he delivered.
According to what I've read, when Prowse saw the film at the premiere, he leaned over to George Lucas and said, "You know, if you had told me that was the line, I would have played that differently."