My understanding is that George Lucas maintains he intended for the Jabba scene to be in the movie, but they lacked the money to create the alien Jabba he wanted using stop motion or something.
It’s exceedingly likely that Jabba was originally intended to be a human; it’s been covered thoroughly here-
https://web.archive.org/web/20140928112036/http://secrethistoryofstarwars.com/jabba.html
According to Paul Hirsh (one of the editors) in his biography, Lucas came up with the alien Jabba idea in post-production, but it had to be dropped due to being impossible to achieve with contemporary special effects technology.