A Jedi should be able to control his fear, for the most part, sure. But this is probably the scariest thing Luke has ever faced, I think expecting him to act completely perfect in this situation is downright silly. Ben is not “one student.” You can’t just fucking call him that. He’s Luke’s nephew. He’s the next Skywalker, which of course means a great potential for power but also a great potential for darkness.
It’s pretty easy to see why sensing darkness in Ben would freak him out to the point of trying to ascertain for sure where his heart’s at. If Ben had done something drastic beforehand, I’m not sure Luke would’ve even handled it this way. This is just Luke trying to get ahead of a situation before it turns bad. Sneaking into his nephew’s bedroom to sense his feelings isn’t even that irrational.
Not to mention, when Luke first tells the story it’s that he went to Ben’s hut to confront him. Obviously it didn’t go down that way, but I think it’s fair to assume that this was probably Luke’s initial plan. Note that the first flashback is the only time we see Luke before he enters the hut - in Luke’s second telling he’s already there - so it’s not unreasonable to assume this part of the prior telling is true. The implication being, Luke sensed darkness in him, went to confront him but before he did he decided to get a better look, saw more than he could have imagined which caused him in a brief moment of pure instinct to ignite his lightsaber, and then… you know.
The thing is, this isn’t the act of “some novice.” This is the act of a master who everyone, including to an extent himself, thinks can’t make mistakes. And yes, he is emotionally compromised. But what has prepared him for this? Especially when it’s implied there has been balance and peace for almost 30 years, how could Luke have been so perfect as to dodge this fear? Just because Luke is a Jedi master means he’s incapable of being afraid?
There is nothing out of character about this. It is 100% in character.