I’m currently imagining that after the Empire completely crushed the Mandalorians in response to their attempted rebellion (The Purge, as it’s apparently called), the survivors went into hiding. They’re staying hidden, despite the fall of the Empire, because they’re waiting to recuperate some strength before trying to retake their homeworld. They’re too few, too vulnerable 5yr-post-ROTJ.
That’s how it looked to me also. Stay out of the way and grow in strength before staging a return to the mainstream. Makes sense and works within the story.