For me personally, it's a I'll know it when I see it kind of deal, but that might not help people who aren't as knowledgeable in such matters.
Off the top of my head, generally, the more extreme the ideology being celebrated, the more likely it is to be propaganda. When it comes to politics, it's can become a question of how partisan is it. Also, how likely is it that the reader will feel as though they're being sold an ideology, or being preached to by either the characters or the story in general? Who made it, for what purpose, who paid for it?
For instance, I don't think anyone would call Lord of the Rings Christian propaganda, despite it being heavily laden with christian themes and symbology. But I myself only became aware of the links after reading it, despite being extraordinarily familiar with the christian mythos.
Whereas, a Chict tract is a pretty clear example of Christian propaganda.
Those examples are obvious, but you're right, it does become grey with many others.
A lot of comics these days will have characters that inexplicably go on rants about whatever social or political topic, which to my mind weaves into the realm of propaganda because it's often out of character, or at least devoid of the character's voice, and has clearly been inserted by the author for the sole purpose of using their platform to propagate their specific worldview (at the expense of the story's honesty and continuity.) To my mind, this is especially off putting when the author draws you in with a good hook, plot and characters, only to later throw that all away to sell you on the moral superiority of their worldview.