I believe you need the DTS encoder from Nero package and use EAC3To. I’ve seen many people convert it to FLAC 5.1.
Don’t know what you guys are doing wrong.
Just beware that even the outstanding blu-ray player Oppo doesn’t support 7.1 FLAC. 5.1 is the limit.
Also the problem is just rebuilding the seamless branching not DTS-HD-MA itself. This discussion is pointless.
It MIGHT be possible to do it with BDEdit software.
You would need to mux the M2TS file with the new audio or subtitles using TsMuxer and output as a blu-ray.
After that you need to copy the CLPI file, which can’t be edited by any software as far as I know, and replace the original one.
Later you take the M2TS you muxed and replace the original one.
Now you need to take BDEdit and edit the playlist files, both of them and add the ID for the new track.
Also you need to edit the offset time for the playlist to match what TsMuxer spits out. Or the other way around, you take the offset timing for that playlist and insert in TsMuxer.
I did that for the Saint Seiya blu-rays which have a single playlist for all episodes and the audios and subtitles were split per episode.