You are correct that only the 1997 edition was ever subtitled or advertised as the Special Edition. In 2000, Lucasfilm rereleased the Special Edition on VHS as simply the "Star Wars Trilogy." However, the 2004 edition is colloquially (and occasionally officially) referred to as the Special Edition, too. You can speculate all you want about why they're so coy over which version is which, but I don't think you'll ever reach a firm answer.