There’s plenty of precedent in Star Wars for one character to speak English to another character who’s speaking an alien language that’s subtitles. Just look at Han/Greedo in ANH and Jabba in ROTJ.
I say replace the droid’s voice in every instance and subtitle it, even when speaking to the Jedi.