"should I be able to hear the difference from an average mp3 there?"

For a high-quality MP3, probably not. Some can.

There have been tons of tests on this, and the answer is that even with very good equipment most listeners can't tell the difference between a good mp3/AAC file and a higher-quality file in AB and ABX tests (where you listen to multiple clips blind and then have to identify which is which). This infuriates audiophiles who've spent fortunes on good equipment, but the science is there.

You want lossless sources of course so you can manipulate them without without introducing error, but for almost all listeners, compression is fine. (I know I'l have people chime in saying "I can hear it." Maybe you really can, but I'd wager if you took an ABX test under scientific conditions, you'd be surprised.)