I don't have problems with small or big casts. Although the thing about big cast is that it can be very overwhelming when too many characters are introduced all at the same time.
But as long as each character serves a purpose to the story then they're relevant.
I had casting problems in my story few weeks ago and I tried to fix it by taking out the each character (mostly one at a time) just to see if the story would change in any way in their absence. If it did, they're relevant but if it didn't, they aren't that relevant.
Although I don't think this method applies all the time. They don't always have to change the plot to be relevant. They can be used to set a tone or teach the main protagonist something (directly or indirectly) for character development.
But if they're just "hanging around" and going with the flow then they'd seem pretty bland and unnecessary. To me at least.