I don't think it hurts at all. I do have a 4 panel gag comic though so I'm not sure if that has anything to do with it. I only have had 4 updates in over a year and I've gotten over 100 subscribers and a few comments and none of them are about lack of color. I don't think it feels right to just change my comic just to get views especially if I like it in black and white and so do current readers. Doing it in color just adds extra development time that I'd rather not have.
However I feel like the line art needs to make up for the lack of color to draw in people's attention. Therefore my comic is super detailed and also my comic icon is in color despite the pages being in black and white. So people will see the bright cat girl better than if it was in black and white. If you just go on the discover page the colored ones stick out more, regardless of what's inside the actual pages. When you color things I feel like you can get away with things that are easier to notice in black and white.
Story and quality of art I feel are more important. While reading your comic, while the first update is really good and I feel like the quality has gone down a lot. That could explain the lack in progress of the views.
P.S. I hope you don't mind my critique.