Perils of the Writer: Composing the Symphony vs. Plot Jazz

In any long term creative endeavor, there is a limit to how much you can really plan out ahead of time.  Of course, you can always make detailed outlines, but the reaches a point of diminishing returns– where what you’re writing is more rough-draft than outline.*
And I’m a big planner, both in terms of individual novels and long-term series plans.  I’m a get-out-the-map-and-figure-out-where-I’m-going guy.  I’ve got outlines for novels I’m still going to write: specifically Books Three for both The Thorn of Dentonhill and A Murder of Mages, as well as Books Two and Three for Holver Alley Crew and Way of the Shield, books themselves that are still waiting in the wings for their time to shine.  And past that, well… let’s just say that as far as I’m concerned, none of those series are intended to end with Book Three.
But outlining is going to have its limits.  For me, a typical outline of a 100k-ish novel is somewhere between 1000 and 1500 words.  That hits the key plot points, but clearly lacks details of character interaction and transitions.  I find that there’s plenty of “Here is Point A” and “There is Point B”, but I still have plenty of chess pieces to move around to have things in place for Point B.
And that’s where the Plot Jazz comes in.
“Plot Jazz” is a term coined by some fans of Deep Space Nine and Farscape to talk about how those shows– shows which never really had a “plan” more than a few episodes ahead– had a great tendency to build off of things set up by throwaway lines or the possibilities a random idea offered up.  An excellent example is in Farscape, where John having hallucinations of Scorpius in “Crackers Don’t Matter”– an episode where everyone is going crazy for spacy-sci-fi-reasons–  triggered the idea that “Hey, maybe Scorpius put a chip in John’s head!”.  That concept ended up being a key factor that shaped the rest of the season, and the series as a whole beyond it.
Plot Jazz, where you play it where it takes you, and hope it sounds good.
Now, what Plot Jazz means to a dyed-in-the-wool outliner like me is that secondary and tertiary characters often become more than they were in the outline.  Not taking over the story, but having more consequence that the outline would have indicated.  And that’s usually because in writing the outline, I don’t necessarily know what I’ll need those characters to do in the specific.  The “what” is in the outline, but not the “why” and “how”, and there is a fair amount of discovery in working that stuff out.
There’s a fair amount of that in Thorn of Dentonhill, where while I was writing I’d realize something like, “Colin needs a friend he can talk to in this scene” or “I need a Constabulary officer to show up here”, and then the character that comes out of that expands into a life of its own.  That’s where, for example, Colin’s crew of Jutie, Hetzer and Tooser or Lt. Benvin came from in Thorn, and those roles ended up being more than could have been predicted in the outline.  Jutie and Lt. Benvin both get mention in the outline for Thorn II, but the details of the how and why with what they do ended up giving them both a crucial role in that book, including realizing I needed to give Lt. Benvin a whole group of constabulary officers under him.
So while I love my outlines and will never, ever let them go… that doesn’t mean I can’t embrace the Plot Jazz when I need it.
—
*- I do know of one writer who does write very detailed outlines for his novels, that the outline is around 40K for a 100K novel. He said the main difference between his outline and the novel is dialogue.