If Han paid Jabba off, Jabba doesn’t have to pay the bounty hunters. The rebel reward looked pretty hefty from that scene where Han and Chewie were loading up the loot in Star Wars.
That Jabba is a drug addled, disagreeable fellow in Jedi doesn’t change what happened in ESB.
The rebel reward might have been hefty for Han. However it was less than a change for a galactic crime lord, as was the price for hiring bounty hunters. In the best case he would kill him quickly/painlessly and take the money. Of course he wasn’t going to be so generous, as indicated in ROTJ. For the rest, read Lord Haseo’s post above.