Hi.
I'm also a Mac user and I use Ivi, available on the Mac App Store for a few bucks (about €4 if I remember correctly). It has a very good Quality/Size ratio. Only downside is you can't convert DVD ripped as a VIDEO_TS folder. So now I'm using MakeMKV to rip them, but that's another story. I can tell it works, I have the despecialized trilogy in my iTunes library.
For playing Mkvs on an Apple TV without RE-encoding, you can try Video stream if you own an iOS device (don't forget to install the Mac client).
Both solutions work great. Hope it helps.