The number of capital ships alone makes it kind of ridiculous. Admiral Piett says they have orders not to engage, but what happened after DSII exploded did they all just retreat?
Well the only explanation for their retreat (or destruction) would be the one I mentioned before. See below:
Assuming the inspiration came from WW2 (as stated by Lucas), where ships were very vulnerable to air attack, I guess the same could apply to Star Wars universe. The explanation then could be that since imperial ships initially sent all TIE fighters to engage the rebel fleet and they themselves did not attack, their TIE fighter complement was destroyed by rebel fighters and ships. Then by the time the Death Star was destroyed, the imperial ships were left extremely vulnerable without any TIE fighter protection.
For example, in the pacific theatre of WW2, aircraft carriers were quite useless without aircraft cover (and other type of ships even more so by default) if enemy had air power. In both Battle of the Coral Sea and Battle of the Santa Cruz Islands Japanese basically achieved tactical victory, but since they lost almost the entire aircraft complement in the air attacks on US fleet, they were forced to retreat. Like I said, ships were very vulnerable to air attacks without air cover.
I understand this metaphor but Admiral Ackbar says that the fleet “won’t last long against those Star Destroyers” so we are led to believe they are perfectly capable of shooting down the rebel fighters.