Save for Iron Man, most of Phase One was mid-tier box-office. Marvel didn’t become MARVEL until The Avengers. It’s part of why that film’s success was so surprising. Up until that opening weekend nobody was really sure the gambit would work, much less work THAT well.
Nobody ever expected that much from the series, even after its cultural takeover had begun. It’s part of the reason comparing Star Wars to Marvel is such a loaded and unfair notion - Marvel’s never had decades of behind-the-scenes myth-making and cultural importance shoved upon it by fans and media helping raise the bar almost unreachably high with EVERY movie. It’s just been consistently producing and releasing good-to-great action films annually. So when its movies don’t excel at the box-office every time out (Ant-Man, Doctor Strange, Captain America, etc.) nobody really seems to mind. There’s not that much riding on them, normally, at least not in the way Star Wars is seen to always have the weight of a whole galaxy riding on it, both in the fictional universe and out in the real world.
Marvel Studios is actually helped by the fact they’ve never courted that level of scrutiny, or indulged in that kind of behind-the-scenes mythmaking. They seem to prefer an atmosphere where if you dig it, that’s great, and if you don’t, it’s not that big a deal, we’ll get you on the next one. Whereas at Lucasfilm if you dig it, you were supposed to, so it’s not really a victory, but if you don’t dig it, that’s a huge sign of something horribly awry on a frightening level that invites a million thinkpieces on what should be fixed. The distinction is essentially one where the studio never lost sight of the general audience as their PRIMARY audience, as opposed to Lucasfilm, whose target audience seemed to increasingly be people who felt the need to feed into such huge, heavy, cultural expectations when they went to the theater.
Age of Ultron was essentially Marvel’s “The Rise of Skywalker” and… it basically didn’t matter by the time the next movie came out. Marvel knew it could release Ant-Man or Doctor Strange and it didn’t matter if it only did 1/2 what Iron Man did, because nobody in Marvel’s target audience is expecting Avengers numbers every time out for every title.