I don't think you can make a reliable distinction in the moment...the difference between the two will usually make itself known in a pattern of behavior over time.
The first time it happens, and you're not sure which is which, you should definitely question it. A simple "...What did you say?" will usually knock the person off their guard, and if they really didn't mean to offend you they'll probably apologize and explain right then. If they just laugh it off and say "just kidding, lol"...keep watching. And don't laugh with them.
Someone who doesn't want to offend you will do it less often if they're not getting positive reactions from you, while someone who does want to offend you (or doesn't care) will not change their behavior. You can also watch what the person in question says about you to others...is it "wow, they're so serious" or is it more like "bitch can't take a joke"...? Gather all the clues you can if you plan to act on your suspicions.
But if you really can't figure it out, or if you have figured it out and don't know what to do, just know that you can't go wrong with a neutral reaction. To those that mean well, it will signal that "hey, something isn't working" and to those that mean ill, it will show them that you won't be provoked (maybe that isn't true, but that's what it'll show them). ^^