Even using reference can be tricky, yeah!
What I do to make sure I don't screw up is use guidelines. The stuff @zerofruits posted is very useful, and I'll follow up on that.
These are in Swedish, but the guidelines should be easy enough to understand anyway. On the upper left is a method I learned for how to think of the torso vs. the hips, and that is to imagine the ribcage to be a kind of oval-ish sphere, and to think of the hips as a kind of box, and then to imagine them being stacked on top of one another in a sack. There are more examples of that directly below in the lower left. When the torso-sphere tilts one way, the hip-box tilts the opposite way to keep the person from falling over, and the shape of the stomach follows that, etc.
On the lower right is a quick and dirty proportion-rule I use to double-check my figures. The distance between the tip of the nose to the dip in the collarbone is equal to the distance between the dip in the collarbone to the bottom of the sternum, which is equal to the distance between the bottom of the sternum and the bottom of the ribcage (but drawn at an angle outwards), and that distance is equal to the distance between the bottom of the ribcage and the edge of the hipbones, etc., etc. This isn't always true, but it's a good thing to keep in mind while drawing - it will prevent you from drawing freakishly elongated torsos, or hips that just don't work, etc.
Oh, and here's a bunch of facial construction scribbles I drew for a friend once:
(also, behold my neat and tidy handwriting!)