How to avoid: simply draw the face in different ways. If you're always drawing the exact same jaw shape, eye shape, nose shape, ear type, etc. across all of your characters they're bound to look similar to one another. So play around. Wide faces, skinny faces, long, short, pointy chins, rectangular chins, round chins, pointy noses, stubby noses, etc.
Does it bother me?: Not particularly. Although it's the "lazy" way to differentiate characters, if they're identifiable enough through their clothes/hair/or other features I'm not too bothered.
Does it matter?: I'd say yes. While it's passable to get away with differentiating characters in other ways, it's a powerful tool in character design to be able to differentiate things like their face or body type as well. Don't sleep on experimentation just because same-face is the easy way out :> All of your chars don't have to be 100% unique, but it's good to focus on the important features of each to work on.