Try first demuxing the video stream into its own .264 file, then using that .264 file as your video source when muxing the ISO.
I think some people over at the project’s forums had the same issue and this was the best solution, but I could be misremembering…