x264 is not a VFW codec - so, unlike Xvid or DivX and so on, you cannot use it from within VirtualDub. (Actually, I see there is a VFW version, but it's not officially supported).
Most people use Mencoder or MeGUI to encode.
This thread should tell you everyhting you need to know:
http://forum.doom9.org/showthread.php?t=89979