I like to imagine the secret of becoming one with the Force as being one of the ultimate secrets of the Jedi order. This is why it is not known to the Sith Lords. Further, I think that the Jedi Masters would only pass the secret on to the most worthy of Jedi Knights. This is not even known to most Jedi Knights and is certainly completely unknown by the apprentices and those outside of the order.
This is why Vader has no knowledge of the power that Ben is talking about in their duel in ANH.
I completely reject the idea that this secret was only discovered by Qui-Gon Jinn after 1000 generations of Jedi had gone before him. I think that this secret has long been held by the order as its most closely guarded and powerful secret. It means that Jedi live on and communicate with the Jedi still in our world, giving them a far greater connection to the force and their history than any others in the galaxy.
The onslaught of the Sith during the times of the Clone Wars decimated the Jedi. Not just Knights but also apprentices and even Masters. Most of these Jedi died without ever knowing the great secret of becoming one with the Force. The Jedi order was scattered throughout the galaxy.
I imagine Ben's last action, after handing Luke to Owen & Beru Lars and before retreating to the Jundland wastes, was to visit Master Yoda on Dagobah. Ben is now one of the last Jedi and is made a Master, is taught the secret and given the task of watching over Luke and to start training him as a Jedi as soon as possible. This plan is scuppered by Owen's opposition and then by Ben's death at Vader's hands. After this, Ben tells Luke to go to Dagobah so that Yoda can train Luke himself.
I therefore must come to the conclusion that Anakin and Vader will not appear as spirits at any point. Ben and Yoda will and it also gives the opportunity of meeting other Jedi long gone. This would probably be explored during the final episodes, as Luke grows in strength and skill.
Any thoughts?