It really is. I never thought about it until I got into audio drama but when you think about it, listening on the radio or to books on tape or just people on the bus or out in public, there are those that just set you on edge. They may be perfectly lovely, wonderful people but the minute they open their mouths... ouch. And only ouch to specific people.
If you look into the history of Hollywood (and movies have been made about it) when sound came in it ruined the careers of a lot of silent star actors because of their vocal tones.
If you've ever seen the movie Jaws and the "music" they use for the coming shark... those undertones are such that they produce dread and are very specifically used for that.
So, frankly, I think that's all it is. No jealousy nothing else. Her voice doesn't pair well with the way you hear.