Don't worry many people become introverts and feel safer talking about something they really like/something that has helped them pass hard times during their life.
Beleive it or not I used to be that way. What made me change was all my life experience.
Finding friends with the same hobby will help you change your way of thinking.
Btw, I did not mention this but I am a huge anime/manga/light novel fan. Lost count of anime watched after 1.5k so I totally get why talking about anime can get you pumped up.
Being a hardcore anime fan it used to get me mad when people had stereotypes about anime. But as I grew up I understood they had such stereotypes because they only watched the most popular anime or the most known anime. Anime has all topics and it is wholesome tbh.
But that's just like saying books are wholesome since anime has many different categories and authors just like books.
There is at least one anime out there for everyone just like books. Have seen great anime about motherhood, romance, horror, girl love, boy love, trans, crossdressing, harem, action, sci-fi, etc. But all those anime usually get overlooked by more popular anime.
To further expand on this, the usual harem trope is of people just getting harems by existing. The best harem I have watched are well written in which even you as the watcher understand why it exists and you even have a hard time deciding what is right or wrong on that situation. Ofc, such harem anime aren't popular.
Same goes with other genres like Girl Love. Some girl love anime have some biased and sexist views. While others are just master pieces in which it describes the real struggles of being a lesbian girl in Japan.
My point being that anime is unique since it has so many varieties thus why it may be so easy to feel identified. Don't feel frustrated if you can't talk to others outside the anime community. Just take one step at a time to reconnect with people.
P.S. A really great anime that really hit me was a movie about the bombing of hiroshima. It was really heartbreaking especially knowing it is based on historical facts.