They are breaking Republic law when they contract with Republic nationals to smuggle spice through Republic territory (which I freely admit is a bit of a retrojection from Jabba's later relationship with Han Solo). And without recognition by the Republic, they cannot be regarded as a legitimate government by any official arm of the Republic government or its constituent elements. It is unlikely even the Tatooine populace considered Jabba their government. He seems to provide no services, no infrastructure, and does not hold a monopoly on the legitimate use of force--posse comitatus and tribal warfare seem common, police unheard of. I'd call him a warlord, but his crew doesn't seem to engage in any kind of warfare. Gangster is apt.