I start off by brainstorming ideas and looking over everything I've written already and everything I like listening to. Finding what vibe I wanna go with, the instruments (and effects), the tempo, and the rhythms and all that. This helps me get in that space of identifying exactly what it is that works for me specifically, and from there I can kinda analyze why it works so I can try emulating the process later in the songwriting..
Then I go to the chords, because I've had a bad habit in the past of using the exact same chords over and over.. So, I'll go out of my way to put together a progression I'm not used to. From there, I'd rearrange the uncomfortable chords into something more comfortable by inverting them (C major for example. It's C - E - G but if you play the chord as E - G - C then it becomes C/E "C over E" and similarly if you play it as G - C - E, it'll become C/G) This makes the chords sound different, but contextually, it'd be the same, maybe even better if you close the range of the highest and lowest note.
Note: Don't stick with simple chords! Add more if it helps! A C major is so boring and basic on it's own, so spice up a little, add a B at the end of C - E - G and now it's a C7 chord. Add a D afterwards (C - E - G - B - D), C9. Or skip the seventh, now it's Cadd9.(C - E - G - D). There's lots of things you can do with them after learning how the notes compliment each other.
After chords, I go into melody. I usually just try doing whatever, but lately, I've been trying to simplify the process by trying to use only quarter notes. if the melody is strong enough there, I'd see how I can improve on it, by adding more notes in between, so it'd move on to eighth notes from there. Another technique to try out, would be improvising the melody through humming. (Don't remember which, but there was a video about being humanlike with improvising and leaving a better effect on people.) You'll see what ranges feel more comfortable to go to, and it gives the melody a more intentional feeling, I believe.
Note: You don't need to fill the melody with 16th notes in order to make it amazing, using pauses and gaps in the melody can also be good in some instances! You don't even need to make every note unique, as if it was randomized to fit in the scale! Using the same notes in similar sequences can make for a theme in a specific song.
From here, I go into percussion. It truly depends on the genre.. but I'm gonna try to sum it up that works with me. Use everything at your disposal, hell, even the sticks themselves can make a good noise. Play different speeds! Double time, half time, vary the sound more by doing these. I highly recommend looking up drum beats that match the sound you're looking for. I'm a bit of a fan of ghost notes and using dynamics (the general contrast in volumes).
Harmonizing.. is usually a filler trick I use. It's mostly just adding third notes onto the main melody. I need to work on this the most tbh anyways on to the next one
Here's where most of this work can all go to waste, mixing! Believe me, mixing is the thing that'll either make it sound rich and full, or extremely empty! You gotta know how to equalize everything and make sure that you're using the frequencies to their fullest extent, because even if you have instruments that span the registers, they're gonna lose what makes them stand out in their own musical space, Identify what the supporting instruments are and adjust them accordingly in volume and in their registers, and the same goes with the main ones as well.
I think that's enough of my process and any tips I can offer lol
Stuff I've written and composed before.
Disclaimer: I only play guitar and bass, so keep that in mind while you hear me put together the sounds from instruments I've never played xD
A lonely quiet piano with reverb.
A repetitive track with rain and other sounds. There's only two instruments being a synth keyboard and backing violins essentially playing the same thing.
Primarily two instruments, the shakuhachi and the koto. I add a few taiko drums later in the song.
I know you mentioned layering the song is a problem. I kinda used to have that problem too. After getting more experience though, I feel like I layering's not such an issue if all the layers actually add something new to the song. That's why symphonies and orchestra groups exist lol we're not on the same level, but you know.. layers aren't all bad lol
The last two songs I'm linking are by far the most effects-heavy layered songs with over 7 instruments each.. that I have to share