Handbrake should be able to deal with m2ts files. Your quality settings are probably too high - try to find the h.264 capabilities of your devices (max ref. frames etc.) and check these settings. Google handbrake + device name and you may find someone has figured out the best settings for your device already.
If the m2ts continues to cause trouble, you can always use mkvmerge to convert to mkv without re-encoding before opening in Handbrake.