That's the worst.
As I was (and am) working on my baby of a novel, I used to have a separate document where I'd dump all the scenes in my head. If I used them or not, didn't matter, at least they were written down.
So, I say, write it out.
Yeah, I get that. I think in the end, if you are creating a mood and it's not just some soft-porn, I wouldn't be too worried. If the style and use is consistent in the story, it should be fine. Pretty jarring to have a wildly vivid sex scene when the rest of the story isn't, you know?
There's a lot of substance use in my story, which is where I'm struggling. Like, the overabundance of alcohol should probably get tagged, but they're also ingesting mushrooms and stuff. I marked it for now.