What about coloring makes you hate it? It's possible you haven't found an efficient way yet or a style that you actually like, or just that you haven't built the skill so it's frustrating. Usually if I'm coming up on something in my comic that I hate or feels tedious, I ask myself "How can I make it fun?" and that can get me through because I've reframed it in my mind.
Like many people said, you don't have to pigeon hole yourself into doing it a certain way, but I also wouldn't avoid it just because it's hard. Hard things get easier the more you do them, and it's worth improving other skill sets. Just don't push yourself so far that you don't feel enjoyment from it anymore.
Also, you don't lose skills. You may be rusty doing B&W but you'd pick it up quickly if you tried again! Besides, those skills should inform your coloring too, you still need values in your colors haha. It's a foundational skill--they're not separate.