Yeah, I could imagine there would be inherent fears within the Empire of the fearsome Mandalorians of old wanting to return to power. The Mandalorians have also always been independent, so the Empire trying to control them would inevitably resort to opposition. And the Mandalorians could be a huge threat if they were align with a larger cause like the Rebel Alliance.

A central focus of the Empire’s campaign against them was beskar. Not only is it a valuable resource, but taking away beskar was taking away a central piece of Mandalorian identity: their armor. And if they could wipe away their identity, they could put in place their own regime, which was something we saw in Rebels with the Saxon clan being the Empire’s lapdogs. So this is probably why some Mandalorians take working with the Empire so personally, because that stolen beskar was a part of an effort to destroy their culture.