I'm not sure where you heard, that, but the "penalty" is that they won't show up in Trending or Popular as often, as the algorithms governing those relies on engagement with the pages. Likes, comments, shares, etc. And it's not like not showing up in Trending/Popular is the end of the world for a creator.
But for your "ghosting" to have any noticeable impact on a creator with, say, thousands of subscribers, then you'd need to have several hundred ghost-subs. Having just one or two people be absent is no big deal. Don't feel guilty about it!