One of my biggest problems is not the worldbuilding, but then taking that worldbuilding and weaving that into a story organically. The old draft of Crown of Druthal was filled with worldbuild infodumps. I thought I could get away with some of it by having a main character who is a historian-- it's at least a passable cheat to have him reading about this stuff. But it still feels like an infodump.
I think this ties into my big problem with prologues in general. They tend to be (though not always) a rather unartful dump of worldbuild info, sometimes giving away the whole story in the process. I recently watched the beginning of the pilot of a (failed) genre show that was astoundingly inartful: several minutes of prologue, mostly done with voiceover narration, rushing through several plot points, and then saying, "Our story begins seven years later".
For Thorn of Dentonhill, Holver Alley Crew and Maradaine Constabulary, I've been working on having the worldbuilding feel more natural, organic and integrated. Finding that right balance of showing Maradaine as a living, breathing, working city that is part of a larger culture, a larger nation, while not having that larger world outside intrude too much.
Because there always is that temptation, with all the worldbuilding I've done, to just throw a little more in.
I'm curious, for those who read sci-fi and fantasy, what marks that line between, "That's an interesting bit of knowledge about the world, making it and the characters seem fuller and richer" and "This is a pointless diversion that distracts from my enjoyment of the book"?