Haha, I see photos of artists all the time and go 'HEY! They look just like their characters! Amazing!' My family say the same thing about me.
@whydocowsfall said it. You see your face every day, it's your best and closest reference, so don't worry if that carries over into your art. And, like @AnnaLandin said, practise will change it.
Try, as an exercise, taking two pictures of celebrities and doing a direct comparison. What makes Bruce Willis look like Bruce Willis? What's different between Helena Bonham Carter and Maggie Smith? Examine the shape and direction of their eyebrows, or the length and size of their noses. Are their faces round or square? Are their eyes lidded or wide?
Or try this: Take a picture of someone distinctive, like Steve Buscemi. You must now draw him using ONLY 20 strokes of the pencil in 20 seconds or less. Repeat this until you can render him in a recognisable way using the minimal amount of effort. In time, you'll begin to recognise the basic shapes YOUR BRAIN picks up on when you glance from one person to another. You can tell the difference between two friends in two seconds flat, but why? Unlock the cooode!