There is a scene in ROTS with Yoda and Obi on the Tantive, which explains this very thing. Yoda, Kenobi, Qui-Gon and Anakin are the only Jedi so far in to come back in ghost form, not all of them.
Well since two of those four disappeared and the other two didn’t, I fail to see how a scene in ROTS explains anything.
Qui-Gon is the first guy to figure it out though.
He “learned the path to immortality, and returned from the netherworld of the force”. He learned to manifest sometime after death, not before. So no disappearing.
As for Vader in ROTJ they either goofed, or Anakin had to learn from the rest of them after he died, since he likely never learned it in life, unlike Yoda and Obi-Wan who learned it from Qui-Gon.