The number of capital ships alone makes it kind of ridiculous. Admiral Piett says they have orders not to engage, but what happened after DSII exploded did they all just retreat?
Well the only explanation for their retreat (or destruction) would be the one I mentioned before. See below:
Assuming the inspiration came from WW2 (as stated by Lucas), where ships were very vulnerable to air attack, I guess the same could apply to Star Wars universe. The explanation then could be that since imperial ships initially sent all TIE fighters to engage the rebel fleet and they themselves did not attack, their TIE fighter complement was destroyed by rebel fighters and ships. Then by the time the Death Star was destroyed, the imperial ships were left extremely vulnerable without any TIE fighter protection.
For example, in the pacific theatre of WW2, aircraft carriers were quite useless without aircraft cover (and other type of ships even more so by default) if enemy had air power. In both Battle of the Coral Sea and Battle of the Santa Cruz Islands Japanese basically achieved tactical victory, but since they lost almost the entire aircraft complement in the air attacks on US fleet, they were forced to retreat. Like I said, ships were very vulnerable to air attacks without air cover.
I understand this metaphor but Admiral Ackbar says that the fleet “won’t last long against those Star Destroyers” so we are led to believe they are perfectly capable of shooting down the rebel fighters.
Well yes, it still makes sense within the same metaphor framework. While a star destroyer is a carrier, it is also a battleship at the same time, and getting into range of battleship will get your carrier in a big trouble. As long as you can stay out of the range of their guns, you can use bombers to attack them. That is what WW2 carrier would do; stay away from the the battleships and use long-ranged bombers to attack them.
So in that sense, Ackbar’s “we won’t last long against those star destroyers” makes complete sense, since it was a reply to Lando’s suggestion of getting into their gun range. Assuming the majority of star destroyer fighter complement was destroyed by a combination of rebel fighters and guns from the rebel ships, the rebel fleet could just keep the range and use their bombers to attack star destroyers (that were without fighter cover at that point) from a distance.