Yeah, some sort of problem with frame height not being properly recognized. The gibberish near the bottom is the vertical blanking interval, which should be at the top of the video frame and outside the captured area, not visible.
The SNES issue is probably unrelated and due to incompatibility with its non-standard 288p signal. The N64 should have that same problem with most games but perhaps you played one of the few SD games.