Perhaps because his work is finished, and Luke has to stand on his own now? Even in Heir to the Empire, Ben eventually said goodbye, and IIRC, he was only appearing in Luke's dreams at that point.

We also don't know if Obi Wan communicated with Luke between Yavin and Hoth. In Empire, Luke was injured and freezing to death when Ben first appears. He remains a voice again until Luke is about to leave Dagobah. Arguably, Luke's training makes it easier for him to see Ben at that point. Ben promptly fades away again soon after.

Jedi mucks all of this up a bit by having Ben walk around and sit on a log. But that could also be Luke's mental perception of his old mentor. It's like when you dream about someone who's died, and they're alive and well.

The SE tosses this all out the window with pasting Hayden in there of course. Luke could visualize his old man as a healthy venerable Jedi, but as a young man he never met and probably never had a photo of?

Has it ever been established non Jedi can see a Force ghost?