Just gonna drop this link here. I throw it into practically every thread about comics craft because it has become my bible over the years.
Basically, no amount of messing with the format is 'too much', it just needs to be done with intent: I've seen comics with no panel borders at all, comics with a different font for each character that's representative of their voice/personality, entire webtoon episodes that are one massive vertical scroll down a single ongoing image, the sky's the limit with this shit, really.
Chapter 2 in particular focuses on the way different levels and types of abstraction can be used to adjust a reader's focus and how it can be used to enhance a sense of place or of identification.
Chapter 4 also has some really great discussion on the use of panels as a sort of fluid language that can be shaped and adjusted to the needs of the narrative. This is mostly in regards to the representation of time in comics, but it can be used all over the place for whatever you need to do in the moment.
The follow-up book, 'making comics' also has a really great breakdown of the balancing act between clarity and intensity/emotion. Most of the time, amping up the 'intensity' of a panel will have a negative impact on its clarity and readability, but leaving that intensity entirely by the wayside can leave a comic feeling flat and boring and uninteresting to read.
The same set of actions and shot choices can be bland and uninteresting, or they can be a jumbled chaotic mess, depending on how much you decide to fiddle with things like breaking panels, unconventional panel shapes, text/art incorporation, and a million other factors, and every single story, chapter, scene, page, and panel is going to have its own needs, so it really just needs to be felt out on a case-by-case basis.
This is hardly practical, but I love this page because it shows just how far you can push formating and messing with the 'rules' if you really want to.
The long and short of it really comes down to that old idiom 'rules were made to be broken', but I'd personally add the caveat that you should probably know why the rules are there and how to work within them before you start breaking them. Otherwise you might just knock down a load-bearing wall or something.