Most regular capture cards use this format internally, so there is no benefit in choosing RGB24. The results will be identical, and RGB with huffyuv is slower and does not compress as well.
However, as the PDI Delux is a bit more than your regular capture card, it might be worth a short test to see if there is a difference. Some colour correction operations, and post-processing with VirtualDub, require RGB format, so capturing in RGB would save one conversion step.
I would assume you should use NTSC-J for Japanese discs, and NTSC-M for US discs. The only difference is in the black level, which is a lower voltage for NTSC-J.
The artifacts shown in the screenshots look like rainbow effects (aka cross colour - hi-frequency luma data upsets the chroma demodulator). It's caused by imperfect separation of the luma and chroma components of a composite video signal.