Luke is very Vader like when he heads into Jabba’s but that is deliberate. You see him wearing all black. He force chokes a Gamorrean guard. He is there to get revenge on Jabba for his treatment of Han Solo and to free his friend. He at first tried to bargain with Jabba but Luke was prepared to carry out murder if necessary.
Well that’s the whole point, you’re not sure if he will turn in the finale. It’s a character moment, like in a real movie.