Oh man...I was just thinking about this last night and managed to get myself out of it; now I'm gonna get depressed all over again...
It's too late now, though, so I might as well dig in...
I think it should be said that Twitter is actually a terrible place to try and grow an art following. Many of the artists I follow have said as much, even the bigger ones...one artist in particular was so frustrated about it that I had to unfollow them: all the constant negativity, while understandable, was really painful to watch. 
Anyway, I think the main reason is because nothing 'translates'. You can go viral for one week and go back to getting 20 likes the next. You can have thousands of followers and still make posts that just a handful of people bother to like, let alone retweet...I don't know WHY it's like this, but it's not uncommon, and it really hampers any linear notions of 'growth'.
Yeah, I know that feeling...it's not realistic, but I've taken to thinking that people who say stuff like that don't actually know how it feels to be in that position. For them, a lack of growth is just a temporary bump in the road, not something you've been watching happen to you for years, largely due to factors outside of your control.
For example, Eclipse basically shattered my presence on DeviantArt...which was 7 years in the making and really the only one I had. It wasn't my fault, there's nothing I could have done to stop it, and now, a year later, it doesn't look like there's anything I can do to fix it. I'm just stuck waiting for things to change. =/
And even before that, I was already losing traction...I remember hearing once that stories and fleshed-out characters (my bread and butter) were good for OCs, and even for fanwork. 'Everyone loves a story...give them something to form an attachment to, and they'll keep coming back.'
Well, in reality (at least, my reality) they don't. Often they just inexplicably stop caring.
Everyone is excited when I announce a new project or AU but before I'm even halfway through telling the story I wanted to tell all fan engagement slows to a trickle. And it's like, 'I thought you wanted this? I thought you were interested and eager to learn more?? What did I do wrong; did I explain too much? Too little?? Am I boring you???'
Of course, I can't actually ask...and to this day I haven't found an answer.
Usually, when I start feeling like this, I go to work on my animations...my YouTube audience is really the only one where I feel like I'm making progress, and where the majority of my fans actually think about and enjoy my work, instead of just a select few.
And it's weird because, as I said once, 'I only average 2 videos a year'. ^^; I wish I could do more to reward them and thank them for their loyalty, but animation is such slow work...all I can do is try to keep them posted on what I'm working on, and corral them over to my Twitter where I usually hang out more. And I feel like it's actually working...whenever I post an animation WIP, there's usually a little club of 12-20 people who like/retweet it.
Sometimes they even comment...and if you're a small artist on Twitter, you know comments are basically gold-encrusted diamonds.
For someone like me with just ~150 followers to get more than one on ANY post...that's very rare and special~.
So yeah...I'm hanging in there with what little I've got. Sometimes it also helps to remember that, even when it seems like everyone is doing their best to ignore me...I still like my work. I like to read and re-read those heartfelt character essays that no one else seems to care about; I like to go through my comics and look at all the little skillful, beautiful things I've done.
Remembering that helps me feel like, even if I'm not getting rewarded for my efforts, they haven't been a complete waste of time.