Luke’s plan to rescue Han makes complete sense when you realize the reason his plan is so complex is because he’s going out of his way to not have to kill Jabba. Because that’s what Jedi do. Negotiate, not kill.
His first plan is to use the droids as a bargaining chip. Lando is planted there so he can get them out after Han is freed, and they lose nothing. That fails. So Leia’s sent in to free him, but she’s caught. Crap. So Luke has to go in himself. He calmly tries to negotiate with Jabba one last time, but he refuses. He knew about the Sarlacc, but not the Rancor, which is why he’s surprised by it. He planted the lightsaber in R2 in case he was gonna be thrown in the Sarlacc pit. Because otherwise they’d nab it off him when preparing to throw him in. And thus we get the sail barge scene.
Luke’s plan isn’t actually needlessly complicated, it’s just that we saw the worst possible scenario because Jabba is stubborn and insane.
I think this is the clearest layout for the “Plan A, Plan B” explanation. Rinzler’s Making of book actually goes into a bit of detail regarding what the plan was with Jabba.
“As one might expect, the rough draft tells a more convoluted story than the final film’s. In its general parts it is similar to the movie. In its details, however, the rough draft differs and in some ways excels. Luke’s plan to rescue Han, though overly complicated perhaps, is more logical than in its final form: It’s clear that his goal is to trick Jabba into the open, where it will be easier for Luke to do battle as a trained Jedi."
“The plan is to trick Jabba into taking Han Solo out of his fortress, as Luke has no hope of overpowering the crime lord and his vassals within his fortress.”
From the story meeting manuscript:
Lucas: What Luke wants to do is to get on that barge and the only way he can do it is as a prisoner. He has to become a prisoner and Chewie has to become a prisoner; they have to unfreeze Han and they all have to be at the same execution, which is what his plan is. He figures once he kills the rancor, then they have to go to the pit. He knows that’s where the execution is going to be anyway. What they do with ordinary nuisances, or solicitors, is they drop them into the rancor pit. Luke knows or doesn’t know that is what would happen, what kind of trap they have laid for him. He’s assuming that when he is discovered and when he is subdued, which he will be, that he is bound to end up with Han and Chewie in the skiff over the Sarlacc pit.
The plan is, “I am going to knock everybody overboard into the pit and we’re going take off”—but it goes a little awry because Boba Fett screws everything up and suddenly they are in trouble and they get into the fight.”
Kasdan: You can assume that Luke’s plan is multilayered and the court of last resort is they are going to take him to the Sarlacc pit and they’ll all be in place. But when he comes in and says, “I want to bargain for Han,” he is hoping that will work.
Lucas: Yes.
From the revised second draft
“Now when Solo comes out of the carbon block, Leia explains that he has hibernation sickness. When he asks who she is, Leia responds, “Someone who loves you.” After they’re captured, a new scene in a dungeon cell interior reunites Han and Chewie, with the latter explaining Luke’s plans to an incredulous Solo. When Luke arrives and has to face the rancor, Lucas fills in much of the action.
After Luke and the others are sentenced, the sub-stratagem of manipulating Jabba so that they’re taken out of the fortress has been dropped. Jabba makes the decision without Luke baiting him—but on the skiff, Luke explains to Han, “Jabba’s palace was too well guarded. I had to get you out of there. Just stay close to Chewie and Lando. We’ll take of everything.” Han: “I can hardly wait.”
Kasdan goes with the angle that Luke hoped he would be able to bargain for Jabba, but the Sarlacc was the last resort. I got the feeling Lucas was imagining that Luke knew Jabba would never give up Han, and everything was just meant to get the entire gang in the right place for the Sail Barge Assault. It makes sense if Lando came to the conclusion that bargaining with Jabba or escaping the fortress would be possible, and the only way Han would ever get out of there is if Jabba took him out himself… to be executed. I think in the end, elements of both interpretations were in the script.
And if you watch the film after Luke kills the Rancor, when 3PO tells them that they are going to be taken to the Sarlacc, you can see Luke nodding his head. It seems like this is what Luke expected. So I kind of see it as them going in knowing that getting to the Sarlacc as the main goal, and not the last resort. But like you said, if Jabba actually agreed to bargain, or if Leia managed to get Han out and Lando got the droids and Chewie, then that would’ve been a welcome surprise. I just got the feeling that Luke giving up the droids from the get-go shows that Luke has no intention of playing fair with Jabba. Which is understandable, since Jabba is a criminal and doesn’t play fair either. I think in the end, though, it was left intentionally vague because they weren’t 100% on which way to approach it.
This is more of a personal interpretation, but I like to think Luke foresaw what would happen on Tatooine. I think that would have satisfied most audiences questions, “Oh okay the Force, he saw what would happen”. Plus, it not only adds more reason for Luke to be so confident in what feels like a complex plan, but Luke luring Jabba into this trap parallels the Emperor luring the rebels into a trap that he also foresaw. The parallel plays into the inner darkness that Luke is in conflict with throughout the movie as well. I also think the Jabba sequence is more interesting if Luke’s emotions during it are a little more complicated. Yeah, he is playing the role of a Jedi wanting to avoid conflict at all costs, but I also get the feeling that a part of Luke wants Jabba to refuse so he can take him out and all of his minions. Yeah, not totally Zen of Luke, but I think it makes sense narratively for Luke to be dealing with that conflict at this point in the story. But the movie definitely plays it as a hero moment unquestioningly, which is fine.
I think it could have been an interesting element to introduce in the first act, though. Luke destroys Jabba and all of his minions, and he questions if violence was a last resort, or what he was actually hoping for. Then later, when we get to the Death Star, Luke actually wonders if Palpatine is right. Does he need the dark side to save his friends? Is that what saved them before? And even though Luke’s anger allows him to actually defeat Vader, Luke doesn’t let his emotions define him, or forever dominate his destiny. He throws his weapon away. Anyway, I think little things like this could have helped thematically tie the Jabba sequence even more with the rest of the film.
EDIT: I admittedly have not read to the ROTJ novelization, but a Tor article kind of summarizes the plot, comparing to the movie, and had this tidbit about the Sail Barge Assualt
The moral ambiguity of what Luke is doing to Jabba and his entourage is made more prevalent, as well as Luke’s uncertainty in how he’s meant to use his newfound power. In that way, Luke’s journey feels more cohesive, more transparent. He is being cocky on the sailbarge, and he is happy to destroy these people who caused pain to friends and neighbors.
“[Luke] found Jabba despicable–a leech of the galaxy, sucking the life from whatever he touched. Luke wanted to burn the villain, and so was actually rather glad Jabba had refused to bargain–for now Luke would get his wish precisely. Of course, his primary objective was to free his friends, whom he loved dearly; it was this concern that guided him now, above all else. But in the process, to free the universe from this gangster slug–this was a prospect that tinted Luke’s purpose with an ever-so-slightly dark satisfaction.”
And then, a few pages later:
“The deck gunners were lining up . . . their shots for the coup de grace, when Luke stepped in front of them, laughing like a pirate king. He lit his lightsaber before they could squeeze off a shot; a moment later they were smoking corpses.”
Obviously it isn’t canon, but it is interesting that the author had a similar interpretation as well.