The Jedi could have just as well been a secret society. Not many outsiders might have known about them in the first place. The whole galaxy police with a big temple on the galactic capital isn’t anywhere in the OT.
It takes very little for a society to erase religion from the minds of the people. Sure, there will be loyal adherents, but if the government suppresses it and disenfranchises believers, then the majority of the people will come to accept the ideology and the faithful will remain alienated, even in a short time. See the USSR.