VLC isn’t a format it’s just a program that opens videos for you, so when you opened it, the icon probably updated to the VLC cone.

To convert it so your program can read it, you can do the video and audio separately it doesn’t really matter, because we have the power of video editing. Simply convert your video, drag it in, looks like you’ve already done this. Then convert your audio by dropping the original .avi onto an audio program like audacity, then export each channel as 24 bit wav files, and load those into your video editor. Then just make sure video/audio are lined up with eachother.

If it’s 5.1 audio then you’ll need to look up how to set that up in your video editor, you will see 6 channels pop up in audacity, right click the first one - hit “split as mono”, and export each of the 6 as a mono track in order from top to bottom, L R C LFE sL sR.