It never bothered me. Whose to say he isn't resting after delivering a fresh bounty? I always got the impression that jabba's palace was somewhat of a (harsh) haven for bounty hunters hiding out.
Also the shots of him in the palace are important for the pacing of my edit now that I've cut some things.