Don't treat backgrounds as a separate element from your characters--it creates a dissonance between the two and makes the work feel weird.
Think of backgrounds as environments: they're 3D spaces that your characters interact with, and they establish the character of your story's world. Both elements have to work in tandem.
So, never neglect them!
In my opinion, I don't think you even have to be a particularly good draftsmen to make good backgrounds: as long as you draw something that expresses that sense of place, it works. I'll dump some of my favourite examples below:
Source: Blame!
Source: Shimeji Simulation
Source: Shimeji Simulation
Source: Takemitsu Zamurai
Source: Takemitsu Zamurai
Also, if you ever want to do more studies of characters in relation to their environments, study your favourite films--get good screenshots and draw those. Films fundamentally have to tackle this issue, so I think it would be a good idea on your part to try it out.
Source: Manchester by the Sea
What's your main problem in this case? Is it keeping characters in proportion to the backgrounds, or the actual design of the spaces you're drawing?