tl;dr the only real problems I've ever had with vegans are the ones who attempt to continually guilt trip me to change my life style into something that I cannot really be asked to do.
May I first and foremost explain that vegan and vegetarian are not the same thing, see vegetarians don't eat meat, but they will still drink cow's milk and eat chicken's eggs for example, while vegans don't eat animal made products period which isn't really all that allergy related unless you were somehow allergic to all animal products, but somehow I think that's a pretty rare condition that most vegans don't actually have. That said, it also means that someone who's lactose intolerant doesn't = vegan automatically. Now baring this in mind, you can see how I would find it somewhat annoying when someone tells me that I should actually drink almond milk or something because apparently their life style is superior to mine by using arguments like "humans are animals too" or something like that (the words of an actual protest sign I've seen before). It's almost like those weird glutton free fanatics who aren't actually allergic to glutton trying to get me eat shit tasting bread when I don't have to.
I will say that i'm fine with vegans and vegetarians who actually don't try to force their life style onto me, and are really just asking for alternatives in restaurants and all that though. That's just their life style, and like a religion, or a way of life, I can respect it. That said, they should in turn also respect my life style choices as well.