True. Also remember in "The Killing Joke" he had a wife who was expecting a child before he even became the Joker, so at least in the comics, he's fairly aged.
Then again, he also said that might not have been what actually happened considering his memories fluctuated.