It always bugged me that Luke deserted his training in ESB then turns up a year later to find Yoda on his death bed, asks about his Dad, then leaves with Yoda all but confirming that he is a Jedi Knight bar one task - facing Vader. What a crappy student!! And he has the nerve to call himself a Jedi Knight in front of Jabba. So how about this - Put Luke returning to Dagobah at the end of ESB!! When Luke say's to Lando and Chewie may the force be with you over the comlink as they fly off to find Han, you could dub in Luke's line from ROTJ when he says to R2 'We're going to the Dagobah system. I have apromise to keep... to an old friend'. only this time he will be speaking the line to Lando over the comlink (might even be able to find some dialogue of Lando saying 'what about you?' or something like that (there is actually a similar line in the sand storm scene which has Han asking Luke to join them on the falcon and Luke saying 'I have a promise I have to keep first... to an old friend'. which would be perfect if it could be ripped. We then cut to Luke on Dagobah, Yoda's confirmation of Vader as Dad, then he dies and the movie ends. Luke knows what he must do, and in the next movie, he does it.
To solve the problem of the really poor rescue attempt at the Start of Jedi, would it maybe be possible to have a title crawl that explains that Lando, Chewie, Leia and the droids all tried to rescue Han but never returned, and the movie then begins with Luke's arrival and Leia alraedy chained to Jabba, etc. Of course, you then have the question of Han being thawed... But maybe when Luke requests that Han is set free, We cut to Jabba's 'I like captain solo where he is' line and show Han frozen. Luke is then dropped into the Rancor's den and when he kills the Rancor, Jabba is so angry he says 'that's it, I am outraged' (or words to that effect, all done with subtitles, of course). He then orders his minions to thaw solo and bring him before him. Leia in Boussh disguise could double as one of Jabba's men. Boussh thaws Han and with some voice dubbing, instead of telling Han 'you are free of the carbonite, blah blah blah' says something like ' you'll wish you were still frozen when you hear what Jabba has in store for you'. Then the sail barge and all that stuff plays out as usual. the lightsabre hidden in R2...Well, that could just have been a plan B failsafe that Luke thought of or, perhaps with some incredible (and probably impossible) CG trickery, we could have R2 arrive with Luke... To be honest I don't know why he didn't just walk in there wielding a lightsabre and say 'Jabba, hand over my friends or I'm gonna start slicing and dicing'. I mean, if it's gonna get messy, why wait till you are on the end of a plank over the sarlaac to then catch your lightsabre and start fighting. It's not like he was searched for weapons when he came in (he mind tricked the pig gaurds at the door and tried to fire Jabba with a blaster.... kinda dumb really).
I realise this is all Waaaayyy out ther, but I think it could work. I just don't know how to do it... but maybe you can at least use some of my ideas to take you down other paths of thought.