So, pre-TPM we've got Jedi in their temples, mediating disputes between senators (for some reason). From AotC on, we've got them as warrior-general's doing impossible things. Neither option seems like they had much time with the general population. Even if your planet was touched by war, its not like you're sticking around to find out of if the rumors of Jedi powers are true.
Population-wise, I imagine its like hearing about a couple dudes who live in DC who supposedly have magic powers. Politicians come to them for advice, but no one else really sees them. Eventually they lead an army in a war in the Middle East, you hear about all the cool things they can do, but you've never met them. Then you hear they betrayed the country and had to be killed. Then you're told by those same politicians that they never really had powers, it was all a lot of simple tricks and nonsense.
I'm just spitballing here, but for most of the trillions of people's lifetimes, the Jedi are just 10,000 guys who hang out with famous people. The Jedi were getting insular, because their ability to use the Force was diminishing. Or their ability was diminishing because they were getting insular.