One scene where bad matting stood out on ANH is this, one of the good guys biting the dust in the trench:
Another matting glitch I've noticed in ROTJ is when the rebel scum are captured and taken outside the shield's power building, there's a walker in the background in the trees and it's a nice shade of green.