Marvel is after all ' the House that Tony built'.
We're three Hulks deep and nobody cares but if they replace Downey Jr. there will be rioting in the streets ;)
Johansson was rather meh in Avengers IMO [Loved her in IM2] and it kinda sounds like Hemsworth is just having a Whinge. Yeah, It's a big difference but he signed a contract.
I'm sure we'll see RDJ in Avengers 2, He loves playing Tony / IM. A day may come when they don't need to rely on RDJ to prop up the Avengers franchise but I don't think we are there yet.