I think it would’ve been perfect for Vader’s one and only appearance in RO to have been him emerging from the shadows and mercilessly killing a dozen rebels.
Perfect other than being completely out of character. He never does that in the OT. He’s not a soldier, why risk his life that way? And why not do it five minutes later when they actually board the ship in ANH?
Just because he didn’t do it in the OT doesn’t make it “out of character.”
Also he clearly wasn’t risking his life. As was already mentioned, being in the dogfight at the end of ANH was far riskier.
And he was trying to stop them from escaping with the stolen data tapes. He failed. When they caught up again, his soldiers had already cleared things out so there was no need to do what he did in RO. So back to your first point, if his soldiers had failed on the Tantive, of course he would have done there what he did in RO.