It would make sense either way for me, but I am glad there wasn’t a lot of emphasis on it. The Doctor was never really bothered at having become an old man after being young Matt Smith for hundreds of years.
But he had been an old man before. He is used to being a man of varying ages. Also, he is, regardless of appearance, old. The Doctor has never, ever been a woman before. Finally, I could have he was bothered at being old in “Deep Breath”.