to go from lossless AVI to x264, try meGUI. Its pretty straightforward. I recently did this for a 1080p fanedit I created.

When setting up meGUI, I think you have to place the x264.exe in the same directory (it should tell you this during install). After that, just use the avisynth script creator tool in the menu to open the AVI and load it into meGUI. You can then adjust the settings to whatever you want (it runs off of an avisynth script in the background). A pretty handy interface for using avisynth and x264.