I was reminded by Mike Caton's post about the sometimes artificial limits we place on our worlds when we are building them. On some level, that's because without those limits, there's a whole other can of worms opened up that we, as writers, would prefer not to deal with. Take, for example, my 2373 Space Opera setting. In that setting, humans don't have artificial intelligence technology, robots or boutique cybernetics. Why? Because I didn't want to deal with that.
But how to explain WHY? In that case, I put a dark chapter in human history (unimaginatively called The Cyber Wars) in which AIs tried to rise up against humanity. Humanity prevailed, and from that point on put safeguards on their computer technology to keep it "dumb".
Magic is another thing that needs its limits. I know one person who insists that "rules of magic" need to be defined early in a story, which I think may be going a bit far, but certainly as a writer, one needs to know what magic can and can't do. In my various Maradaine stories, magic can't heal, touch the mind or affect the dead. Magic is physically draining, and takes energy, and energy means calories. Mages tend to be skinny and constantly eating.
Magic also has to have an impact on society. One thing I believe, and it's reflected in the technology level of most fantasy works being Medieval/Renaissance levels, is this: the presence of real, quantifiable magic impedes the progress of technology. Impedes, but doesn't halt. (That's why in Maradaine, technology is closer to 17th century instead of 10th.)
What ways do you all set limits in your worldbuilding?