The garbage mattes certainly weren't as visible in the cinema as on the home releases, but I watched a 77 print in the cinema just recently, and the tie fighter mattes in particular are clearly visible on screen.
I dug out a review a few months back from 1978 where the reviewer mentioned the 'green shields' around the tie fighters, so it seems some people back in the day thought it was an intentional special effect.
I have a memory of seeing them at the time, but memory is horribly unreliable, so it could be later conditioning, but they are certainly there when watching in the screening room.