I had that happen too. Actually, now that you mention it, my N64 controllers have almost completely broken down. The Blue and Red ones especially. The red one is torn in two in the plastic coating on the cord, and the blue one's joystick is iffy.
Ironcially, the one that works best is the original one that came with the system.