Oh man, it really, really used to. Don't let it bring you down because it really isn't a reflection of your work.
After I started losing subs, it made me really analyze my approach (and my feelings). After some soul searching, I changed my little "Thanks for Subbing" picture to always include a note saying "I hope you enjoy your stay here." That ways, it implies the temporary nature of subscribers (for my mental health- and I genuinely hope they dig hanging out with my characters for a while).
Good luck!!!! Go kick some butt, @BelleBriar