They are very rare, but I have heard that some upscaling DVD players with HDMI output were able to play "higher resolution" MPEG4 ASP (i.e. Xvid) in AVI. But most player chipsets can only manage up to 720 x 480/576.
One of our TVs is able to play such files from a USB stick, but cannot decode AVC (i.e. x264) video.
What is your DVD player make/model and does it have an HDMI output?