I'm no expert, I could easily be doing something wrong, but...

For AVC:

x264 --preset=veryslow --tune=grain --bluray-compat --fps=23.976 --force-cfr --vbv-maxrate=25000 --vbv-bufsize=30000 --crf=18 --level=4.1 --keyint=24 --open-gop --slices=4 -o output.264 input.264

To be clear, I target BD25@1080p and this usually hits the target for a standard-length film (19-21GiB video--not much headroom for extras--I'd increase crf to make more room). Also, the vbv-maxrate could be unneccessarily low but I've heard about burned media sometimes needing lower peak rates--maybe this is unsubstantiated. Either way, I do know my encodes work across a wide range of media and players, so that's the test I'm most concerned with, and the results are generally pretty good-looking, so I'm happy.

For MPEG-2:

I use the HCEnc GUI, targetting DVD9@720x480 avg bitrate 6500, max bitrate 8000, pulldown yes, profile best. This usually gives you ~5.5GiB video files and lots of headroom for extra audio and subtitle streams.

All Windows for now, although I'm making plans to jump that ship.