There are pretty much two ways a sf/fantasy writer can go about writing and worldbuilding: come up with a story, and build a world for it to be in; or build a world, and come up with stories to put in it.
One way to think of this is the difference between Star Wars and Star Trek. The Star Wars universe is really only about telling one story: the rise and fall of the Empire, and the concurrent fall and rise of the Jedi. I would argue that part of the problem with the Star Wars prequels is all they do-- all they really can do-- is set up the board for the story the universe has been built to tell. Star Trek, on the other hand built its universe out of telling many different stories, and has the ability to telling many, many different kinds of stories. Of course, with the Star Trek universe, you've got a lot of patchwork worldbuilding, held together by spackle, baling wire and wishful thinking.
There's no right or wrong in this, mind you. Like plotting/pantsing, whichever method works best for your own particular style is the right way to go. However, both methods have their challenges.
Personally, I'm a Method Two guy: I start with crafting and detailing the world, and the process of doing that opens up the avenues for stories. The challenge behind that is narrowing things down to figure out what, actually, you need to write in that world. With a whole wide world of grand and epic scope, you want to treat the whole thing like an enormous brood of children. No one gets neglected. So a world that has the potential for many, many stories might not yield any stories, because no singular story you write can be as grand and far-reaching as the worldbuilding work you've done.
This was exactly my problem with my trunked project Crown of Druthal. It was nothing but a travelogue, going from place to place with no through-plot and little purpose. And why was that? Because I had made EVERYWHERE and therefore I had to go EVERYWHERE. It wasn't until after realizing the flaws with that mindset that I was able to narrow my focus to one city, and let the rest of the world only brush up to it, as the rest of the world would in a cosmopolitan city. I had a similar problem in the space opera setting for Banshee-- I had defined so much, I had to force myself to narrow the focus of what the story actually was going to be.
So why not do it the other way, and build the world to suit your story? Of course you can, like I said, nothing wrong with that. The challenge there comes from when you need or want to do more beyond that story. I've mentioned this before with David Eddings. Everything in The Belgariad was set up to tell that story, as the ultimate, saving-the-world, fulfill-the-prophecy, make-everything-right story. And then a sequel series was asked for. So they had to undercut The Belgariad by essentially saying, "Yeah, that was the dress rehearsal. Here's the REAL event." Of course, building the world to fit the story you've come up with doesn't have to have this problem. Sometimes doing it that way can open all sorts of avenues to new stories that you hadn't imagined to begin with.