Elizabeth Taylor was famously said to have "purple eyes" on occasion. In reality, her eyes were blue, but depending on the lighting, they could appear violet. This was seen as a beautiful and unique trait.
But she was also a very conventionally beautiful, major Hollywood star—so the rules were always a bit different for her. Celebrities often set new beauty standards that the general public follows.
Whether people would love or hate a trait like that depends on whether we're talking about the real world or a fantasy world.
In a fantasy setting, you could make something as common as brown hair into a trait people are mocked for. If history has taught us anything, it’s that those in power often try to make their “club” more exclusive as a way to maintain control.
There was once an entire pseudoscientific craze claiming that white people had superior skull shapes compared to the supposedly “ape-like” skulls of Black people—total nonsense, of course. But if people could be hostile over skull shape, then yes, purple or yellow eyes could easily become something others are hostile over too.
It just depends on how you write it.
As for the differing attitudes toward a Caucasian with purple eyes versus a Black person with yellow eyes—that also depends on whether we're in the real world or a fantasy one.
In the real world—especially in the West and Europe—racism would likely play a role. Yellow eyes might be demonized, especially given how we often portray demons with yellow or red eyes (and sometimes purple).
But in a fantasy world, it’s entirely up to you. If you don’t want racism in your world, you don’t have to include it. And they could simply be hated or loved based on their eye colour with skin colour playing no role. That’s the fun of fanatsy it can be whatever you want.