Holy carp, I found it! This is the first computer game I ever played. Originally called Wizard 1, the name was changed to Oldorf's Revenge because of confusion with Sierra's games. At the time I was indeed convinced that Sierra's Wizard and the Princess was the sequel to Wizard 1.
While it was mainly a text-based graphical adventure, the game had an RPG element where you could change your character to a wizard, a cleric, a strongman, gladiator etc. in order to accomplish different tasks. You could only use each character 5 times, so you couldn't "cast a spell" every time or you would use up your wizard.
After numerous exhaustive searches I finally got a handful of results from some old computing magazines scanned with OCR software. The name change explains why I could never find it before. Internet grail search complete!
From Compute (Nov-Dec 1980):
It's not listed at mobygames even under Oldorf's Revenge, but this site reckons only 1000-2000 copies were sold.