The problem with ^that caveat is that people can just arbitrarily decide whether or not it applies, and often do because there's no way to prove it either way.
Even if a producer/writer/artist said it wasn't for diversity alone*** and that they had a deeper reason, or a complex network of deeper reasons for making space for minority demographics in their work, the consumer could easily just claim they're lying and continue hating. ¯_(ツ)_/¯ Which is why 'DEI'/'wokeness'/'whatever the buzzword was before' is an easy dogwhistle for bigots to latch onto: you don't have to justify or defend your position, just claim that the creator you don't like is being puppeted by the latest form of deviant groupthink, and reject any evidence that says otherwise.
...Honestly, it's not even a dogwhistle anymore; generally if someone is railing about one of these buzzwords, 9 times out of 10 they're an overt racist/queerphobe and you're better off just ignoring them. =/ People who want to discuss the nuances of these topics don't use those kinds of words-- for example, they're more likely to mention 'tokenism' specifically than to just scream about 'DEI' ruining all their favorite shows.
***IMO, the idea that "diversity alone" is an invalid reason, and that people who aren't cis, straight and white need some kind of permission slip (???) to exist in media is part of the problem that drives this "culture war" discourse. T_T
Yes, there ARE issues with blindly tossing minority demographics into your stories because you think it'll make you look progressive, but the mere fact that they're there, i.e. 'diversity alone', is not one of them. And while bigots cry about having to see people who don't look or act like them and pretend it's a harbinger of the end times, legitimate discussions about thoughtful representation and intersectionality in storytelling are being drowned out.