Oh man... There is always so much to it. Usually, I have a story idea banging around in my head for months to years before I even start it.
For my main fantasy comic series, I first thought of the characters, and the world sort of emerged around them. I had images of the main characters, and a few key scenes, and then I decided to really develop the world. The best tip I can give you is, know a lot about a lot of real cultures. For example, in my comic's backstory at some point, I needed to come up with a rite of passage for young boys coming of age, so I did my research about a lot of real types of rites of passages to get some ideas.
Also, realize that a lot of cultural practices tend to stem from another cultural aspect. As a fundamental example, if a word in a language has a negative connotation (as opposed to denotation), then that deeper meaning can affect how similar words are also treated. Or, if a culture is by nature patriarchal, their entire language and culture should reflect that. It should be reflected in the culture's religious practices and their laws.
It also helps to have visualizations of settings, characters, maps, etc. If you can't draw, it might be a worthwhile investment to find an artist who can help you visualize everything.
There is so much more that goes into world-building, but remember, the story should make sense as a result of the deeper world building, not the other way around.