As for saying "What's Ebbet's", I wouldn't dare.
Watch the Danny Kaye film Wonderman. In the deli, he starts asking for a pint of Prospect Park and the owner jokes that they are all out of Prospect Park, but he could get a quart of Ebbet's Field if he wants it.
I cannot disagree with Madison Square Garden.
...
Although, technically, you could consider the Colliseum in Rome a sports venue. That could be more famous.