I think the "too diverse" complaint stems from series where the authors literally have one of everything and they forego rounding out their characters just to have more characters to check off list items. It's like, "Oh, I need an ace person, a disabled person, a trans person, and have them across a gradient of skin tones, etc." Tokenism to its max. Take a random sample of friend groups and I don't think they'd be so evenly spread out.
And perhaps it can be unrealistic when all the diverse people in the cast get along? This connects to the last point I made about friend groups being evenly spread out. I'll use myself as an example, I have many friend groups, but I don't ever see all of them meshing well together. My high-school friends, they're all Asian and female (mostly Chinese with one Pakistani/Indian Muslim girl), they're extroverted introverts and are fairly LGBT+ friendly. They would do okay with my college group of friends (even mix of Asian, Latina, and white, with 2 guys, one of which is gay); my college buddies are a bit more extroverted but do well just hanging out at home too. Then there's my group of internet friends (all some sort of LGBT+, but all white), I'm the only Asian they interact with afaik; very introverted but we have a good time playing games together.
Compare these to my husband's college friends (all Asian except for himself, with one female), they're more on the conservative side and are more introverted. I really don't see a couple of them doing well with LGBT+ groups. Then there are my husband's Nebraska friends, a pair of twins, white, not as conservative as you would think but very extroverted, sometimes they go a bit overboard with their gamer talk. We've played Overwatch matches with them and my internet friends, did not go well xD, personalities didn't mesh at all!
While myself and my husband get along with all these groups, it's really difficult to see these groups together. Not everything can be rainbows and peace signs. So maybe at the end of the day, what's unrealistic about such diversity is that they're not diverse enough in their personalities and view-points.
Granted, if you want to show a bunch of diverse characters getting along swimmingly then more power to you. But maybe some see it as a pipe-dream, something unattainable, and therefore unrealistic? Because all cultures, no matter how hard some try to erase it, still have veins of prejudice and discrimination running deep.
EDIT: I'll also add that sometimes you get some interesting combinations of traits. For example, my friend from Louisiana is trans male, missing an arm, and his fiancé is gay, but from some of the jokes and other things they tell me I've gathered that they're more moderate in their political views and aren't all-out liberal which I find fascinating.