My problem arises when the "characters as characters and not as a gender" are nearly all straight white men.
In society and also in media, the "basic" setting seems to be a straight white man.
TV shows with men are often seen as "for everyone", but with female main characters become "women's shows".
Harry Potter is seen as fiction for everyone, because it has a male protagonist, but the Hunger Games are seen as "girl books". Firefly is for "everyone", but Buffy is a "girl show". And there are many more examples.
There is this continuing idea that everyone can identify with a character that is a straight white man, this doesn't go when it's a straight white woman (let alone characters who aren't white or straight, or even conform to gender norms).
And, that's problematic.
Because everyone else has been taught from very early on to symphathise and identify with a straight white male character, not because they want it, but because that's all they'll get. But when they try to find characters like themselves doing these same cool things, suddenly they're delegated to the "women's fiction" or "black fiction" or "LGBT fiction" sections, not in the fantasy or the thriller or any of the other genres, but in special sections, because they're "different" from the "standard" type of main characters that those genres inhabit (white, straight and male).
For people who are white, straight and male (or even two out of three), the question of race, sexuality and gender never seems to be much of an "issue", they think that characters should be characters, but that's simple for them, especially when most of the fiction and media around you actually focuses on people who are similar to you.
You're not seen as "different", you're not deligated to a separate 2 or 3 shelves that holds fiction "with people like you", with very limited story options. The rest of the readers? They learn from a young age to identify with people very much not like them because they know that otherwise, they're going to be out of luck in most genres, this is not done by choice, this is done by neccessity.
So no, as someone who is basically the opposite of a straight white man, I don't agree with "characters as characters and not as a gender/race/sexuality" because those are very important parts of my identity and how I experience the world and how the world responds to me.
But, characters like me are generally pushed into their separate shelves because of for example their sexuality, you want to read a kickass lesbian female main character in an epic fantasy series? You're more likely to find her in the "LGBT books" section next to the non-fiction book about the AIDS epidemic in the 80s, than the epic fantasy section with the straight men. And that makes them impossible to find, but also reenforces this "straight white men only" problem on the regular genre shelves, because who is going to look for their next epic fantasy on the LGBT shelves? They're going to look on the epic fantasy shelves, and those books rarely ever show up there.
edited to add
We have a huge problem when it's okay for a male author to only write male characters for his books, and when he's asked about that answers that women are too hard to write. But he writes books with people who are partially lizards, who live in literal different dimensions, who can perform magic and teleport and stuff like that. Complex things that he has to create and come up with. But somehow, writing a female main character, something that exists in this world, a creature that he has lived with his whole life (he has a mother and a wife), that's too hard and complicated?
While some of our favourite male main characters are written by awesome women (Harry Potter by Rowling, Fitz by Robin Hobb and many more), and somehow they are able to write male characters? And that's considered normal?