At Øredev 2013, I asked managers to tell me what makes their job difficult. I spoke to one manager who told me that they face immense pressure on two fronts: from the board of directors to deliver more features more often and from the programmers to give them more time to refactor. I’ve heard dozens of stories like these over the years and I’ve found a trick that helps. It doesn’t resolve the problem, but it helps take a significant step in the direction of a resolution.

Warning. You almost certainly have more to deal with in your exact situation than this advice can cover. You might need some more direct help. I can offer that to you, but for now, read on.

I call it the eternal struggle between the Business and Programmers. Specifically programmers, because I almost always hear the specific complaint coming from them. It goes like this.

Business: More features! Now!

Programmers: More refactoring! Now!

It usually involves more colorful language, more intense emotions, and repeats a few times, but it boils down to this. I notice a few things about this particular exchange.

First, the Business can easily justify their position: they need profit, and profit comes from sales, and sales come from features, and features come from, among other people, the Programmers. Without more profit, nobody keeps a job.¹ Clearly, the Business has this one right.

Next, the Programmers can easily justify their position: they need to deliver more features, and they can’t deliver features if they don’t understand the code, and almost nobody writes code clearly the first time, so the Programmers need to edit their work, and they call that editing “refactoring”. Without refactoring, features don’t flow, the Business has nothing to sell, we can’t generate profit, and nobody keeps a job. Clearly, the Programmers have this one right.

Unfortunately, the Business only sees that the Programmers used to deliver features more quickly, and now they want to slow down. This creates obvious tension. This tension slowly pervades every aspect of the relationship between the Business and the Programmers. It obscures something important. I’ll come back to that.

Somehow everyone has ignored that the Programmers are slowing down anyway, because the code gradually forces them to slow down. Interesting.

So the Business needs features to sell, but the Programmers want to slow down. The Business needs to feed a market that expects more features sooner, but the Programmers need to feed a design that demands cleanliness as a condition for adding new features as a steady pace. Thus, the eternal struggle, because it has always been thus.

Help Us Help You!

The Programmers plead: “If only you’d let us refactor! We have a sick design; it actively prevents us from adding features sensibly. It encourages us to do the wrong things. It encourages us to duplicate code, which makes mistakes seem inevitable. It has code that nobody understands any more, and so when we have to change it, we need weeks to read, to research, to gain confidence that changing it won’t anger our existing customers. If you’d let us refactor, then we could spot design improvements that would unblock new features, that could reduce the cost of new features by 20%, 30%, 40%, and more! Why don’t you let us refactor?!” They appear to have a point.

The Business pleads: “If only you’d stop with this refactoring nonsense! The economy has to grow. We have to grow. The board of directors yells at the vice presidents, who yell at the third-line managers, who yell at the second-line managers, who yell at us. More features! More features! They only want more features! Not only can they not afford for you Programmers to slow down, but they need you to speed up! I need you to speed up! What do you go faster?!” I don’t envy their situation.

The Central Conflict

If you’re a Programmer, then imagine yourself in the position of the Business. The Business relies completely on Programmers (among other people) for its survival. I imagine the Business finds this imbalance uncomfortable, even threatening. Without the Programmers, the Business has nothing to sell. Worse, at the source of this tension we find a haphazard, slow stream of half-working, not-quite-what-we-wanted features. It doesn’t matter that the Programmers can easily justify their behavior, because the Market doesn’t like the results, and as soon as the Market has an alternative, it disappears, and nobody keeps a job.

What can we do?

I have to admit that the programmer in me loves that a cyclic dependency lies at the center of this conflict.

The Phrase That Pays

Refactoring reduces the volatility in the marginal cost of features! It smooths out the design, distributing the problems more evenly. It helps the Programmers make many small easy-to-clean-when-needed messes, rather than several small-to-large we-have-to-stop-the-line-to-go-any-further messes. It also helps the Programmers identify game-changing structural improvements that they might not otherwise see—the kind that reduce the cost of future features by 20%, 30%, 40% and more. Refactoring even helps eliminate many classes of annoyingly-costly mistakes. (You might call them “bugs”.) As certainty in the cost of features goes up, the overall cost of features tends to go down. This short talk describes how refactoring and the cost of features relate to each other:

(If you prefer to read a transcript of this talk, click here.)

(If you’d like to see a sketchnote of this talk, click here.)

Seven minutes 26 seconds. 
Fred Brooks, Mythical Man-Month, No Silver
Bullet. How many of you read it? 
Not enough. 

Accidental complication and essential complication. 

Not complexity, because complexity implies emergence. And
I'm just talking about complication. No such thing as an
order of magnitude improvement in performance in part
because of accidental complication. 

Essential complication because the problem is hard. The
problem is hard so the system is complicated. You want to
audit a tax return? Have you read the tax code in Canada?
It's this thick. Essential complication.

Accidental complication... we're not so good at our jobs.

Accidental complication because we cut corners, we feel
pressure, we don't have to worry about it this time, we
don't have to re-factor so much, we have to get it out the
door. 

Accidental complication. Essential complication.

The cost of a feature is a function of the cost coming from
the essential complication because the problem is hard and
the cost of accidental complication because we suck at our
jobs. Roughly speaking we can add these things together. How
do you estimate, if you estimate? 

Don't estimate. 

But if you estimate how do you estimate? This thing is kind
of hard. This thing is kind of the same kind of hard. This
thing took two weeks. This thing, two weeks.

Here's the problem: Friday afternoon meeting, "Don't worry,
boss. Three days." Monday morning, "Oh shit. Make that three
months." Why? 

Accidental complication. 

Most people most of the time the cost of a feature is 
dominated, 
dominated, 
dominated 
by the cost from accidental complication. That means that
the cost of your feature has almost nothing to do with how
hard it is and almost everything to do with how much your
design sucks.

What can we do? Test driven development to the rescue. (Of
course he was going to say that.) 

Well, what the hell do you mean? 

Test driven development step one: think. (Everyone forgets
that step.) Step two: write a test. Step three: stand back
and ask, "How much does this test suck? Is it too long? Does
it have irrelevant details? Can I make the test smaller,
simpler?" Congratulations, you're removing accidental
complication. 

Step three: run the test and watch it fail. You'd be
surprised how often it doesn't actually fail. Why doesn't it
fail? Well, I'm running the wrong tests. No, that's not the
problem. I forgot to check something. No, I'm responsible. I
write the assertion first. That's not the problem. I wrote
too much code last time and it already passes the test. I
should stop doing that.

Accidental complication.

Step four: write just enough code to make it pass. 
Just enough code to make it pass. 
Just enough code to make it pass. 
Not the line of code you know you have to write. Just enough
code to make it pass. Return 12. If input is 7, return 12,
else return 13. Weird. Good enough. If every product in our 
shop is either 12 or 13 dollars, we're done.

Another if statement: now it's a lookup table. Let's move
the lookup table here, then here, then here. Now we have
MVC. Accidental complication. We have to figure out how to
remove it. And so after we write just enough code to make
the test pass the next thing we do is we clean the kitchen.
We refactor a bit now. We have made a little mess to add
some behavior and now we smooth that little mess out.
Because if we don't clean the kitchen then we have to clean
the garage. Cleaning the garage is a bigger job. We always
put it off. We hate it when we have to do it. The only time
we clean the garage is when there's no more room for the
car. Clean the kitchen.

Cleaning the kitchen, reducing accidental complication.

So far: think, write the test, watch it fail, write just
enough code to make the test pass...

(give your computer caffeine so that you can see the timer,
mistype your password because you're trying to go too
quickly.)

Reduce accidental complication.

So write a test, limit the amount of code you're trying to
write at once. Reduce accidental complication. 

Ask yourself, "Does the test suck?" Remove accidental
complication. 

Finally, make the test pass. Refactor, clean the kitchen.
Reduce the accidental complication that gets in because you
didn't do a good job of avoiding it in the first place.
You're getting better. 

Every few minutes. 
Every few minutes. 
Every few minutes.

At the top of the circle, avoid accidental complication. At
the bottom of the circle, squeeze out the accidental
complication that got in while you weren't looking. Why?
Because if the cost of a feature is the cost of the
essential complication plus the cost of the accidental
complication and if we compare estimates based on essential
complication - how hard is the problem if it's roughly as
hard as that thing that took two weeks it should take two
weeks.

But if the cost from accidental complication dominates the
cost of the feature, then you arrive at the fundamental
theorem of agile software development. 

Not enough of you are leaning in. 

The fundamental theorem of agile software development says
this: If you want to estimate little things you have to
refactor because refactoring is how you reduce accidental
complication and only by driving accidental complication
down as far as you possibly can will your relative estimates
have any meaning.

Here's the proof: f(essential complication, accidental
complication) = g(essential complication) + h(accidental
complication). f is supposed to be proportional to g. 

But you have h. 

h either has to be a perfect multiple of g (how many times
have you worked in a code base where the shit was uniformly
distributed throughout the code?) or h has to be - say it
with me - zero. Because if h is anything bigger than zero
and h is not a multiple of g, then f will not be
proportional to g and 
your 
relative 
estimates 
will 
be 
to-
tal 
bull-
shit.

Therefore, if you're going to estimate, you'd better
refactor which means Scrum cannot work without XP. This has
been seven minutes and 26 seconds.

Questions?

Why “The Problem” Is A Problem

Over time, the Business has coerced the Programmers into stealing future capacity by pressuring them to not cultivate a healthy design. They have probably done this for a long time. Don’t blame the Business for this: the Industry has largely taught them to do it. On the other hand, the Programmers haven’t used a particularly meaningful defence: they’ve appealed to arguments based on some abstraction notion of craftsmanship. Some Businesses want to buy craftsmanship; others need features now!²

Now, we can realize something significant: The Business and the Programmers want the same thing. They want the same thing: to deliver a more predictable stream of important features that will make the market happy—or, at least, knowing the fickle market, that will risk disappointing them the least.

This. Is. Glorious.

Mending The Rift

The Business and the Programmers need to agree on two things, each conceding a key point to the other. The Business needs to agree that the Programmers have been going more quickly than they really can, that they cannot sustain this pace, and that trying to do it leads to disaster. We are driving in a car accelerating towards a brick wall. The Programmers need to agree that the Business has a legitimate need to deliver more features, that everyone’s livelihood depends on this, and that “it’s the Business’ job” to ask for more than the Programmers can ever deliver. The market compels them to do this—if it doesn’t, then where does the revenue come from? In this equation, the Programmers have the responsibility to do whatever they can to maximize their ongoing delivery speed, and that includes regular, aggressive refactoring.

Finally, both the Business and the Programmers have to agree that they have wasted too much time arguing about this and need to move forward. They want the same thing! They can have it!

Thus, I propose a kind of detente. The Business agrees to remember that the Programmers always try their best to go at a fast-but-sustainable pace. The Programmers agree to remember that the Business also feels threatening pressure to perform, and that they don’t feel comfortable relying so much on the Programmers for their success. Each group needs the other desperately, and together they all want the same thing.

…and let’s not forget the Testers and Operations and other departments. If we listen to each other, we’ll realize that we all want the same things, and together, we can make things so much better.

Good programmers with great habits will help your project succeed, but even the greatest programmers with the greatest habits can’t guarantee the results you need. You need help with your tough questions. You need somewhere to go to ask whether you’re on the right track, making the right decision, solving the right problem. It’s hard to tell the good advisers from the bad. If you like what I have to say, then you’d probably enjoy having me on your team.

You could hire me for on-site coaching, but that’s expensive and you might not feel ready to make that kind of investment. You might not even need that much of me! Now you can get the advice you need from a trusted source at a very reasonable price. Visit The jbrains Experience and get started for less than it costs to get your team together for a 30-minute meeting.