I also have a hard time accepting Yoda's lightsaber skills. Yes, they look cool and it's exciting, but beyond that initial wow factor three years ago, you realize how ridiculous the concept is. Yoda always seemed somehow above swordfighting. He was an inner strength kind of guy, and it didn't need to manifest itself outwardly. He had insight and could move objects with the flick of the wrist. He didn't need to be so overtly kickass. It made him seem less Yoda and more like your generic Jedi thug. And it was all done for eye candy. Palpatine got the same sucky treatment in Sith. Apparently you can't have knowledge of the Force without resorting to making a laser sword.
And it makes little sense for Yoda's ability to be active. You can argue that spending a generation on Dagobah could ruin his physical side. But he seemed active enough in Empire. Plus, you have to understand that Yoda lived to be 900 years old. Twenty years is just a drop in the bucket for him. So I was flabbergasted in TPM when Yoda looked like a young... um, man, whatever. And, yeah, the inconsisitencies between flying squirrel and doddering invalid from scene to scene doesn't even deserve any more discussion. It simply makes no sense.