Definitely on team "If the background isn't relevant to what's happening and it won't look weird without it, leave it off." here... I think only about half of all Errant panels at most have backgrounds depicting the environment, and the rest are just colours and patterns! 
That said, I often use this deliberately to do fun stuff. When characters are thinking about a certain person, you'll see the background is in that character's associated colour, or if a character is dominating the scene, the background will match them, so a lot of pages go a bit like:
So, talking about Jules, purple background, then we have an actual background because the panel is illustrating Crow moving through the environment from Jules over to Rekki. Then two blue panels because the person being talked about is Sarin (blue), and then a background because again we're illustrating movement through the environment, in this case, Crow is walking away from Rekki.
But also, backgrounds tend to vanish in combat scenes unless the environment is very relevant to the action...
Other than being faster to draw, less detail makes a panel faster to read too, so it makes an action panel feel faster. Detailed panels tend to feel like they last longer, because the reader will spend more time looking at them, and it'll take longer to understand the image.
Sometimes, simplicity isn't just for ease of creation, but actually does help tell the story.