It was his use of a line traditionally interpreted as a curse on all Jews and rejected by his own denomination which were I believe one of the main bones of contention (though you would have to be an Aramaic speaker to notice it as even Gibson blinked twice and removed the subtitle).
That and cartooning 'bad Jews' with the usual hooked nosed bad teeth stereotype and 'good Jews' (proto Christians) as looking like they stepped off a biblically themed fashion shoot.