You have to. I majored in film production in college and when doing any kind of film editing, if you don't back up and get the scene "up to speed", you can't tell if you made the right decision. If you just watch the one shot or cut, all you're judging is the shot itself.
Think of it as a gourmet meal - there are all kinds of ingredients and spices you can use. They all have their own unique flavor on their own but they don't all work well together.