Yes, extremely so. I'm a very fashion-conscious individual, but the reality is that adding a tasteful vest to standard office attire, as I do daily, takes me from normal to extremely well-dressed, and if I add some actually interesting boots and my long canvas duster, I get into highly eccentric territory quickly.
There are definitely a lack of options available, and even fewer that one can wear to anything except particular festivals. What's weird is that nearly every guy wants to dress like the characters in fantasy fiction, action movies, and sci-fi, and many women also want men to dress like that... but it somehow doesn't make it into the department stores' tiny men's clothes sections.
Combine those limited available options with the 250-year trend that a "manly" man doesn't care about the look of his clothes but only the function (and yet, in the magazines or films, he still is tailored in well-appointed clothing that emphasizes his figure and contrasts well with his skin and hair colors), and you get a real desert for menswear.
I don't like to compare suffering, but looking at the recent looksmaxxer trend - there's definitely some suffering there. Many women have also had fashion-based suffering over the decades, and when you are suffering, it doesn't really matter if someone else somewhere is suffering too, or has it worse, or whatever. It still hurts. It doesn't help that most things that make a man unattractive by society's standards are outside of his control - height, whether he goes bald, and so forth.
"Mogging" is a terrible idea long-term, because people tend to grow to resemble their peer group over time, in looks, finances, and opinions. Surround yourself with people who are slightly better than you in the ways you want to grow, then learn and adopt their habits into your own life bit-by-bit.
As a teen boy, I thought the most fashion I could do was pick the color of my T-shirt and jeans, and then wear a standard suit to formal functions. It took me most of college to accept that I could have a more eccentric fashion sense, and another 5 years to develop it into a relatively consistent, conscious style. And that's with me being a really eccentric fop of a person in general!