I do agree, often it can really limit the perspective taken but there can also be extremely stylish narratives crafted by way of first person that don't work in other modes. For example, literature like The Pillow Book (a collection of observations, essays, journal entries, etc.) and Notes from the Underground (a fictional journal) in which the narrator works as the author. For Notes from the Underground, the writing is structured in a way that transforms first person perspective into a distinctive character in active observation.
Mo Yan also does an incredible job in writing first person by injecting them with enough life that it doesn't limit point of view by way of speech and instead creates an experience. Short stories Iron Child and Autumn Waters are captivating right off because of how surreal yet familiar the narrative is constructed.
Life and Death are Wearing Me Out is another fictional novel that obscures narration across multiple characters while maintaining almost exclusive first person perspective. With proper stylisation to the writing, first person perspective can be a multi-dimensional exploration.
However, I can totally understand an aversion to first person due to repeated bad experiences with it. So many of those can put a bad taste in your mouth after a while.