Well that depends whether you always interpreted it as "pure fantasy" or not. I didn't, and it doesn't have to be. Are you really comparing Star Wars to magic? Sure, if you want it to be, I can see why bringing midichlorians into it would piss you off. Trust me, I don't think Lucas did it very well - at all - but if his intention was to make empirical something which had hitherto been undefined, then I can't say it bothers me.
"The Force is an energy field which surrounds all living things" isn't really that far from midichlorians being something genetic found in all living cells, is it. So that idea doesn't bother me, I just wish it had been handled a lot better.