The art definitely matters, but what makes great comic art and a great standalone illustration aren't always the same thing. There are plenty of comics I've been recommended because "the art is beautiful" and never got into because they're purely beautiful in a "you could print a page of this and put it on your wall" sense rather than a "you feel completely absorbed in these characters and their world" sense.
The comic artists I admire the most are people like Bryan Lee O'Malley, Hiromu Arakawa, Stuart Immonen and John Allison (among many others), who hit this balance between an attractive style and a consistent, appropriate level of polish for the update or publishing schedule or page output, but also have absolutely excellent clarity in how they get across what's happening, how the characters are feeling, how the audience should feel about it and even details like timing to make the action and jokes land perfectly. They are all fantastic artists whose work I probably wouldn't put on my wall (any more than I'd put a random frame from a movie on my wall), but I could devour their work in a comic and pore over it for hours trying to deconstruct their approach. Scott McCloud is similar. He is an absolute master at crafting pages, but he's not a particularly interesting pinup artist.
So yeah, it matters, but sometimes people get the wrong idea when you say "the art is important in comics!" and think "good comic art" is a page full of posed and pretty images that took an unreasonable amount of labour rather than efficient, expressive art and clear visual storytelling.