He needs to confront Vader to face his fear and resist the dark side in a final spiritual confrontation. It’s a continuation of the events of Empire Strikes Back. He failed in the cave specifically because he brought his weapons with him and he was full of fear. He failed at fighting Vader both because he was unprepared physically, but also spiritually. Now that he has his training, he’s prepared, so he needs to do it. As Yoda tells him, it’s his final trial to become a Jedi. He has to meet Vader again face-to-face, and gain the victory by resisting temptation. That includes the temptation to join Vader because he’s his father.
I did not consider this. Thank you for the interesting read!