Well, to be fair, he does claim illness. "Sick have I become, old and weak."
But personally, I agree. The first time I saw the Yoda duel in AOTC, I thought it was really kewl. Then the more I watched it, the more it unnerved me. It's just not in his character to spring about like a frog on methamphetamine.