Encoding in 64bit is unfortunately not as straight forward as replacing x264.exe with a 64bit build. There's basically two ways as far as I know:

1) Avisynth 64bit + x264 64bit

Pros: No 2GB memory limitation for Avisynth, speed

Cons: Only 64bit Avisynth filters

2) Avisynth 32bit + avs2pipe or avs2yuv + x264 64bit

Pros: 32bit Avisynth filters are useable (if needed)

As I only encoded in 32bit so far my knowledge on this is rather limited. I hope that's enough to manage the rest via Google search.