I do sorta understand the worry. After all, specifically with places like Japan, you get people assuming it's an anime thing and you've never been there and it'll all be based on anime cliches, for example. It's not the only example, but it is the most common. So, you do have to be careful sometimes.
But think about how many people not from the US read novels set in the US. Or how many people were ok reading Harry Potter despite not being from the UK ect. Some stories don't even bother giving you a specific country, it's just generic. As long as you do actual research, I think it'd be fine. And really as long as you're not spending the entire first chapter just describing the place with nothing happening, it shouldn't be too much of a big deal, I imagine lots of people wouldn't pay too much attention to it.