I think that the subtlety of the animation mixes well with the kind of ghost theme you're going for (especially if your inspirations come from Manga, shifty shadows, things out of the corner of your eye that you might blink and miss, you've definitely captured that spirit with the hints of animation you've included so far).
Honestly the only way I can see it being distracting is if it somehow negatively influences a strong character moment but, with the way you've handled it so far I don't really see that as being an issue. Welcome to Tapastic, by the way!