Arguably, you had to have seen Star Wars to know what was going on in ESB. And the Holiday Special to know who Boba Fett was. ;)
Colson's death was the catalyst for the Avengers to start acting as a team. It wasn't just some throwaway tie in reference. I've only seen bits of Iron Man on tv and none of the other films before I saw Avengers, and had no trouble figuring out what was going on. Haven't even followed any Marvel comics in over a decade.