People often complain about the ground battle of Endor and how rebels and ewoks beating the imperials is unrealistic. However, I would say it is much more realistic than rebel fleet beating imperial fleet in the space battle over Endor. At least in the ground battle the imperials were severely outnumbered (it appears to be only one infantry company plus 4 AT-ST involved in the that area) and were also being ambushed. On the other hand, in the space battle rebel fleet is severely outnumbered (judging from the scenes, around 30 ISDs plus Death Star vs around 20 rebel ships) and were the one being ambushed by the imperials.
The only reasonable explanation for the outcome of space battle I can see is that Death Star’s huge TIE fighter/bomber complement was not used at all in order to slow down the rebel fleet’s destruction and buy time for Emperor to gradually turn Luke to the dark side. Then the Death Star was destroyed before its fighter/bomber complement could be deployed.
This still leaves the advantage in number of ships. Assuming the inspiration came from WW2 (as stated by Lucas), where ships were very vulnerable to air attack, I guess the same could apply to Star Wars universe. The explanation then could be that since imperial ships initially sent all TIE fighters to engage the rebel fleet and they themselves did not attack, their TIE fighter complement was destroyed by rebel fighters and ships. Then by the time the Death Star was destroyed, the imperial ships were left extremely vulnerable without any TIE fighter protection.