DuracellEnergizer said:

It'd be nice to think that people don't take lightsaber colours into consideration because lightsaber colours don't really mean anything or indicate the wielder's affiliation, but I know that's not likely to be the case.

Actually, I think that's sort of how it is seen by most people in the GFFA. I remember in TCW, it was assumed by a victim of Maul and his brother that they were Jedi judging merely from the fact that they wielded lightsabers. A bunch of space pirates made the same assumption.