C3PX said:
I just don't see how it is even possible for a writer to delude himself out of truly understanding what he himself actually wrote about.

It's easily possible. People don't always know the deeper reasons why they do things and that is all the more the case with a creative work like art that heavily involves the deeper recesses of the mind. Art is not solely of the conscious mind. People in the arts often don't know what they're doing, though they may think they know. But allegory isn't something you do unconsciously. It's something you do consciously and deliberately and I don't think Tolkien consciously and deliberately wrote his works as allegory.

I just don't feel it is always possible to know exactly what the author wrote about unless he specifically explained it himself.

And you may still not know even if they explain it, because they may not consciously understand it properly.