I would not say my comic lore is really affecting me personnally in the way it does to you, but I sure think of it a bit too much, and it is also a real problem in the creation of my comic. I'm precisely trying to deal with that for the past few weeks.
The thing is as my story is mostly a coming of age story, centered around characters rather than environment, the lore is not in the spotlight, although it is there, and somewhat important in the development and maturation of my main character.
But, because I think of it a bit too much (read: WAY too much), I end up with a super complicated lore which cannot be explained in the story without running into the 'lore dump' issue.
So I try to keep it reasonnable... but it's difficult to keep remembering what is disclosed in the story (ie. What the readers will know) and what is only in my head.
I'm currently developing a system to be able to keep a record of all the allusions to lore in my comic, to know precisely what my readers know.
Particularily, the chronology needs work. Some of my main character's actions and beliefs can only be explained by events dating back from the creation of the country he lives in, several hundred years ago. I want to make that known, but I won't speak about all what happened politically between then and now.. but I had to construct all of it to make the political system coherent. So, that's 300 years of politics, administration and geographic boundaries created, but never put in the story.
I'm thinking I could make an extra with all that. It's a bit of a shame to not present it, as it is enriching the story, but again, inside the story, it could be an heavy, unpalatable dump 