I've always taken a very hands-off approach with regards to this topic.
My comic consists of animals that are literally animals. It's meant for anyone who comes across it, and my audience reflects that, including a large number of my acquired readers that are or appear to be furries. If someone were to claim that my comic were a furry comic, then, well, that's just their opinion. The only person who truly knows my reader demographic is me. My comic is there for everyone to enjoy if they wish to, and I hope they do.
In the end, readers are readers, and if you can form a nice relationship with your audience that's all that really matters in building a good community. If anything, people who try to antagonize other groups are the real ones that should be excluded.
If certain readers try to push your artwork in a certain direction, you don't have to follow it. If they get mad at that, then they weren't good for your readerbase in the first place because they can't appreciate your art for what it is, so it's a good thing if they leave.
I used to be a bit cautious of the concept of furries too, then I actually talked to some, and they are really just like everyone else. Some are nice, some not so much, just like any other group or fandom.
Along with this, add everything that @nathanKmcwilliams and @Jenny-Toons said. I would never consider an audience built from misrepresentation to be a truly beneficial readerbase. Respect people for who they are, not what they represent.