"wielding a lightsaber would be like flying a spaceship; lots of people can do it, so we need to raise the stakes"
I like this. Have the Jedi wielding a different weapon. Maybe bring the Kyber Crystal idea back. Then it starts becoming fantasy/magical, though.
But Star Wars is a fantasy to begin with.