I see this as a question of tone? Sometimes a scene is relevant to the plot but still feels like it's also a treat for the audience (and possibly the author as well) -- like, this is important but while we're here it'd be a darn shame if we didn't get to see his abs!! On the other hand, sometimes a scene shows a lot of skin but doesn't feel sexy (see: The Meek). That's a hard line to draw, but I'd still probably say it depends on the gaze of the comic.
But there's not really a hard-and-fast definition, either -- those who see "fanservice" as a pejorative might argue that it's not fanservice unless it's shoved in there for no reason. Those who see "fanservice" as something that's fun and good might consider something like that scene to be well-done fanservice.
Fanservice can be unfitting to the tone, but I don't think all fanservice is immediately unfitting in non-joke media just by virtue of being fanservice. When Breath of the Wild opens with shirtless Link, and then takes the time to point out that the first shirt you find doesn't fit (and is midriff baring), you can't convince me that's not fanservice -- but it wasn't jarring, it didn't take me out of the intrigue of the game or make me see Link as a joke, I just whispered to my friend "thank goodness for ill-fitting shirts," we both agreed that shirtless link is a gift to us all, and I kept playing.
As for my comic,
I think fanservice kinda only works if the author is also enjoying it, and I don't tend to have many physically fit characters or get excited about them showing skin. Is "turn the angriest character into a squirrel" fanservice? I mean, I was having a great time, and a bunch of readers were delighted, but it doesn't exactly fit the traditional definition of fanservice as "something sexy."