Anyone who gets spooked by a chibi and ignores the words/story you wrote wouldn't stay for long anyway. There are some things you should do for yourself and not solely thinking about readers that haven't read your story.
A good fan will stay no matter what you throw at them. Chibis are warm and welcoming. They are also an art style decision. Just like how you have your own writing style.
Would you change your whole writing style just because some people don't like it?
I wouldn't.
Think of adding chibis like an extension of your writing style. If you don't let others change your writing style you shouldn't let others scare you from sharing something you like doing.
That being said. Since your work isn't horror I doubt you will ruin the vibe with a chibi (chibis and horror don't really work well together).
Romance works nicely with chibis. Tapas is a great place for writers to express themselves however they like. You should take advantage of that.
If you are still afraid of ruining the vibe then you can probably add a chibi every several chapters instead of every chapter (you never mentioned doing chibis every chapter but I will assume it was something like that).