I don't know anything about Vegas, but it's been my experience that it's easiest to split the 5.1 AC3 into six mono WAV files (being sure to label them L, C, R, Ls, Rs, and LFE, so you don't lose track of which channel is which), edit using those, then render the finished audio out as six mono WAV files, which you then combine into an AC3 file once again.
I can't help you much beyond that, 'cause I use a Mac and don't know squat about transcoding and video editing on a Windows platform.