Quite a nice machine Harmy:-) I'd have chosen a SSD as system drive though. If you ever had one you'd never want to go back to HDDs.
Year, 32bit software has a RAM limitation of 2GB per process, which is quite easily maxed out when encoding HD with a few Avisynth filters or slow x264 settings (especially lookahead and merange).
As I said earlier, the whole encoding process should be about 20% faster when using 64bit for Avisynth and x264, so it's probably worth to switch. I can check whether all filters are available in 64bit if you PM me your script.