It didn't look like a joke, it looked like a snide remark with the way it was phrased and taking into account that you neither are friends with the people you joked with nor made any attempt to relate to their situation. In order for a heavier joke to not fall flat, you usually need to do either one or the other.
Right now you're just a random person, who btw often gets into disagreeable positions with us, coming up to us telling us we'll die early, with a smile. It comes off passive aggressive.