ThrowgnCpr said:
If you do end up creating a mask externally, I don't think you would have too many problems with a jagged or stairstepped edge. How would you render from Vegas? You could save it as a lossless AVI, or frameserve directly to something like CCE for a good DVD render (I usually do not render to DVD straight from Vegas). Either way, I think that the result would be fine.

I would probably use avisynth/virtualdub to do the resize, then - and this presupposes that I am using Vegas to do the curved border instead of avisynth - I would take the resized .avi file and open it in Vegas to add the curved additional border. I'd render that as an avi, and use TMPGEnc to encode the final result for DVD.