I hate the celibacy vow, but I don’t think that’s why there is so much abuse. My assumption is that pedophiles become priests because they know that they will not only have access to children, but also that the Church will protect and enable them in their crimes. They would have access to children in schools, daycares, summer camps, or sports teams, but accusations typically are and have been taken very seriously in those settings. You still see tragic cases of abuse in schools and sports, but not thousands of students abused by thousands of teachers in single cities where the districts ensured that the teachers be allowed to continue their abuse. The protection is what draws them, I think.
Alternatively, I’ve often wondered if some people notice in themselves urgings that they wish to suppress (e.g., attraction to kids, attraction to same sex), and are therefore attracted to being a part of an institution that provides a supportive infrastructure for a celibate life. I once knew someone who was considering joining a monastery partly for this reason. Ironic if they were to then find themselves in an environment that supported acting on said urges.