The standart stormtrooper “were the elite soldiers of the Galactic Empire (wookiepedia)”. If you let the movie thing that they can not hit something aside, an elite soldier is the best soldier you have. It is very strange to believe that it is necessary to make a design-variant of this elite trooper.
That’s just it though, the key word you used was soldier. Soldiers fight in wars, and by the time of RO and ANH, the war had ended. Yes, skirmishes and battles still ran rampant, but the Empire wasn’t fighting planet wide battles against an opposing army anymore. Their enemies now hid in the shadows, in more volatile environments, and on less inhabited planets, etc. A faceless army is far less useful in fighting these hidden enemies. And so Stormtroopers essentially devolved into armed guards, patrol, law enforcement, etc.
I’m sure the Empire kept a large number of standard Stormtroopers on their bases (just like on the Death Star in ANH) so that they could be called upon if necessary, but large scale army vs army battles were no longer common place like they were during the Clone Wars.
Long story short, when your enemy is hiding, you need your troops to be capable of working in the same way, and thus more specialized troops were trained for more specialized missions. To me, it just seems logical.