I really admire people like Tolkien and Sherwood Smith who spend years building a deep, complex world. My writing method is exactly the opposite. I start with characters, then ask myself `where do these people want to live?' and build from there. Once in a while I'll begin with a situation, and spend weeks (or months. Or years.) trying to decide exactly who would be caught up in it.
One of my biggest jobs in a second draft (the first draft is to get the plot in order) is to add setting details; making sure I don't have doorknobs when there should be latches or buttons instead of laces, seeing to it that the weather pattern fits the season, all that good stuff.
The story setting is, to me, a bit like watching a photo develop. At first there's a blank, then the first gray smears appear through the chemicals. As you watch, the smears darken and sharpen into actual shapes until finally you're looking at a clear photograph.