1. tell us what mac os you run, e.g. 10.8.4, and which mkvtoolnix version did you use
2. when using tsmuxer, just drag your source files into, then select all wanted/deselect all unwanted tracks, select outformat (.ts, .m2ts or blu-ray) and then click "mux"
afaik aac is no valid codec for blu-ray. so you may only remux to .ts - somebody else may verify this in case i'm not correct.