I don't think Luke ever seriously expected Jabba to play ball. It was all a ploy to get Artoo inside the palace with the lightsaber in case "Plan A" with Leia failed. (Lando having already been working on the inside for an unknown period of time, was probably going to aid in the escape.) I'm still puzzling over Threepio's line about Chewbacca having never returned from Jabba's lair since Chewie and "Boushh" don't show up until later.

Marvel comics notwithstanding, it would have been a lot easier to pay Jabba off with the money Han got from the Rebels at the end of ANH.