I wanted to break down the known Infinity Stones and how most of them have been on Earth forever. Figured I’d just come up with my notes here to share.

Space Stone (Tesseract): Hidden/lost on Earth until Loki steals it from SHIELD. Currently in Asgard.
Mind Stone (Septer): Given by Thanos to Loki who loses it on Earth. Currently in Vision’s forehead, on Earth.
Reality Stone (Aether): Hidden/lost on Earth until Malekith steals it from Jane Foster. Currently in Asgard.
Power Stone (Orb): Hidden/lost on Morag until Star-Lord steals it, & Ronan from him. Currently on Xandar.
Time Stone (Eye of Agamotto): Sitting in a libary on Earth until Doctor Strange steals it. Currently back in the library on Earth.
Soul Stone (???): ???

So 3 of the Stones have been on Earth for hundreds/thousands of years (ignoring the easter eggs in Odin’s Vault). One was in Thanos’ possession until he lost it on Earth. One had been on a dead planet for hundreds/thousands of years.

How long has Thanos been searching for them? The one he started with (Mind Stone) he gave to a trickster god who promptly failed to gain the Space Stone. The Reality Stone was hidden in a pocket dimension, IIRC, and we can assume the Time Stone is hidden through magic (it’s possible that Kamar-Taj is in a pocket dimension itself, though it still has wifi).

Infinity War will be interesting, as Thanos has to go through the Avengers (Mind), Asgardians (Space/Reality/Soul?), Guardians (Power), and Sorcerors (Time) to collect them all, even if he only has to go to three planets to do that.