Tsmuxergui is what we generally use around here. The ps3 is indeed much better, but it only works with fat32 drives, which is problematic for files this big.

I would generally recommend not extracting or remuxing directly to a USB drive if it is giving you trouble just make the file on a local drive and copy them over when you confirm that they are correct.

Does your mkv app you’re using reencode? I think the best option may be to remux to m2ts and stream to ps3. Though my ps3 is hacked to do NTFS so I need to be careful about making suggestions that will actually work for other users.