I think - at least initially - art does matter. The way I like to think about it, is that the art is the bait... But the story is the hook. You can have a really great hook, but without any good bait you probably won't be catching many fish (fans) so to speak. But, once they are hooked, they're unlikely to go anywhere.
But, likewise, you can have the best bait in the world - but without a good hook, you're not going to be catching much, either.
So in this way, I don't think art will ever be able to 'make' a comic - I've seen quite a few comics with fantastic art, that just can't keep me reading beyond a few pages... The story rendering it unreadable, or unappealing.
And I've also read some comics that, due to the art, I was hesitant to pick up at first - but afterwords, I was glad I did... The only time the art would 'break' the comic, for me, is if the art simply can't portray what it needs to be portraying - be it expressions, actions, character consistency, ect. If I can't follow the story properly because the art falls short, then that can break it.
Though, it's worth noting that most artists I know tend to be very picky about the art in comics - because they're artists! If the artwork looks like I, personally, could do a lot better - that does put a dent in things...
But for non-artists? Awesome art is cool and all, but it's not the end-all-be-all.
People will make snap judgements. After all, the art is the first thing people will see - it's good to make a good first impression. But beyond that first impression that gets people reading, it's not nearly as important as the story quality.