You’re using tsmuxer to make your BD ISO, right? Use the same program to demux - check only the video stream, then select the “demux” option instead of the “Blu-Ray ISO” option before you run the operation.

When it’s done, close tsmuxer, then reopen it. Open the .264 file you created in the previous step, then open the MKV, uncheck the video stream from the MKV, and proceed as before, though you’re pretending now that the .264 file you opened first is the video track from the MKV file (which it is, really, but something about it as contained within the MKV file isn’t playing nice).