That’s what I meant by rejection, that he rejects it at all is confusing. You cite a criticism from Yoda in Empire to explain his lack of resolve, as if the following film didn’t go to great lengths to demonstrate that Luke had grown beyond that. He became less of a doubter than the master who once called him the same, when he thought he could redeem a murderer through love for his father alone, and succeeded in doing so. For what reason would he have to doubt himself at that point? For what reason is there to parse from the remainder of that film that he’d remain a doubter? It’s less a character assassination than it is a character regression; that you have to compare TLJ Luke to ESB Luke is telling of exactly what’s wrong with the writing.
People don’t just magically lose self doubt and insecurity when they become successful though. In many successful people, their fears of their own failures only get amplified. They don’t see it as having won, or having bettered themselves. They see it as having a harder fall when they inevitably screw it all up again. Sometimes our heroes don’t think of themselves as heroes, and their struggles with insecurity and self-doubt are doing things to them we’d never suspect looking from the outside.
Granted, the characterization of Luke in the sequel trilogy is, on a behind-the-scenes level separate from the larger storytelling, prompted by the fact they needed a reason to keep him out of The Force Awakens. But when it came time to personalize that reason, to make it make sense, Rian Johnson and Mark Hamill worked together to come up with a version that not only makes sense, but is very emotionally resonant. I don’t think it’s confusing at all that a person as innately good as Luke would react with self-doubt, insecurity, and hesitancy in response to having the mantle of Legend placed upon him, and he’d definitely be mad at himself for allowing himself to believe he was one, even for a second, especially when, in that second, it led to his losing Ben.
The reason “happily ever after” works so well in fantasies is because you don’t have to go into the parts after THE END where nothing ever ends, and the people, despite the things they learned and the change they’ve affected, still wrestle with who they are, and why they do the things they do. Politicians are like this. Teachers are like this. Athletes, artists - any sort of role model you can think of, all you have to do is check out a biography from the library and read the parts that come after any other story would have stopped and “Happily ever after’d” and see similar struggles.