I think this is good! It will also allow you an even greater freedom with your fantasy races/ethnicities since you're not bound by real-world history. No one's asking you to write allegories for African enslavement or Jewish pogroms or even portray Japanese culture realistically since none of those exist in your fantasy world. Especially for high fantasy nowadays, people just question a cast where everyone is lily-white.
For example, in my historical fantasy, I initially didn't have a set race/ethnicity for my protagonist. I just knew that I wanted her to be not-white, and I made her ambiguous so that she could be any number of other things. After doing some research on immigration in the area in which I based the setting off of, I decided to lean toward Lebanese for her. Her specific background doesn't play into the story at all since she was adopted, and the simple fact is she's just brown and some people raise an eyebrow at that in an otherwise mostly white town.
Also as an additional note about handling stereotypes:
If you feel a certain character has to act in a certain way because of plot purposes, and it leans toward one of the negative stereotypes, simply balance it by having another character of the same identity act oppositely.
For example, have an asian character not be the best driver? Show another that drives professionally.
if you've got a black criminal, have a character who's also black but is a law enforcer/good guy.
I should also note to simply be wary of colorism (which CyberAxl mentioned earlier in this thread) where people who are darker-skinned (regardless of ethnicity) are portrayed/seen as bad and have negative traits. Like there's an issue if all of the characters on the darker half of a cast's spectrum are ugly, bad guys, poor, uneducated, etc. compared to the lighter ones that are beautiful, in happy relationships, successful, looked up to, etc.