Mine isn't how many readers I have (though nice it would be to break out into the double digit subscribers to a novel...)
But rather that with those few already reading it, my novel won't connect with them... I want what I write to take people out of their minds, picture it and get goose bumps or want to try something like playing random instruments or whatever, not just think "this is a good story" or "this plot has potential..."
I crave to try and connect with readers (people in general) through my writing, since I'm not really good at it personally face-face... (I fear some people think me annoying/childish and naĂŻve... so my work is my way to try and get a feeling of connection, but then also not just with people but me and my characters as well, I feel whole when a am able to "connect" with my characters! And thus my fear is suddenly having that connection go missing...