Takeaways: Lessons learned during the session

From Deleting to Finishing

Core challenge

     Knight Capital lost more than $460 million following 45 minutes of erroneous trading on 1 August 2012. Developers had reused an old activation flag for new code. The flag had belonged to Power Peg, a routine unused since 2003 but still present and callable in production.

     A technician copied the new code to seven of eight servers. No written procedure required a second person to verify the deployment. When the eighth server activated Power Peg, the response team removed the new code from the seven correctly updated servers. That exposed the old routine on those servers too.

     Knight intended to delete Power Peg, it had always been the plan. The team had not verified that the retirement had taken effect across all eight servers, and lacked controls against unintended activation. The rollback made that distinction expensive.

Why we invented a company

     Knight is a clear warning and a poor mirror. Most of us do not run trading systems, and a loss of that size makes ordinary shortcuts look exotic. We needed a case the group could step into, so we invented Northline, a field-service company, and released its facts in three stages.

     Northline sends fourteen technicians to roughly 120 customer sites each week. Mia, the operations coordinator, had planned those visits for years in a spreadsheet of appointments, availability and customer notes. A week took her six hours.

     Northline bought a scheduling system and scoped the first iteration deliberately. The system would import bookings, assign technicians and calculate travel time. Mia would review each schedule before publication. The team would decide later what to automate next.

     Before the six-week pilot, they agreed three criteria: at least 95 percent of visits inside the promised window, zero missed contractual visits, and a schedule published every Thursday.

Stage one: the numbers

     All three criteria passed. On-time visits rose from 86 to 97 percent. Mia's planning time fell to ninety minutes. The operations manager proposed automatic publication from Monday and a read-only archive for the spreadsheet.

     I offered the group three options: automate fully, continue with Mia's review, or extend the pilot.

     The response was positive. We listed conditions to check before committing: security, documentation, employee absence, peak periods. The case gave us reassuring answers. What Mia actually did in those ninety minutes never made the list - her review had been filed under finalisation, in the next stage.

Stage two: the next iteration

     We asked Mia what the review involved.

     She compared the spreadsheet's customer notes against each proposed schedule and changed five to eight appointments a week. The system had placed one visit on Wednesday afternoon. The note said Tuesday mornings only - she moved it. Another customer expected the same technician. A third needed an early slot after an overnight shutdown. The team had migrated records and booking dates in the first iteration and had scheduled the notes for the second.

     The manager had imagined her review as an occasional error check. Erik Hollnagel's distinction between work-as-imagined and work-as-done names the gap. The manager was deciding on the imagined version and the pilot had measured the real one. The 97 percent was a valid result. It belonged to the system and Mia together. Removing her review would create a different arrangement, and that arrangement had not passed the pilot.

     Even migrating the notes would leave unaddressed work. When two notes collided, Mia decided which customer to call and what to offer. She could not fully explain how.

     You can observe the work, ask about specific changes while the context is fresh, test the written account against later weeks, and investigate the exceptions it fails to explain. Each step makes the account more useful; however, none truly completes it.

Ask the experts and you get the parts they can cautiously acknowledge.

Stage three: six notes, three meanings

     The account manager traced the arrangements behind Mia's changes. Two were contractual obligations. Three were preferences Northline honoured when it could and had never promised. One was an informal commitment made by a former account manager. The customer treated it as permanent; the current team had neither confirmed nor withdrawn it.

     The manager wanted all six loaded into the system. IT asked how: hard rules, preferences, or exceptions?

     Hard rules would enforce all six every week. Mia had been distinguishing obligations from preferences and managing one unresolved promise every week. Loading every note would finish the migration and give unsettled decisions the force of rules. The informal promise needed a conversation and a decision before anyone could encode its meaning.

     Resolving these six would still leave the source of future ones untouched. If account managers continued to be rewarded for satisfying individual customers without regard to wider consequences, new promises could keep appearing. Northline would need a way to authorise and record those commitments, or Mia's spreadsheet would acquire a successor.

Removing an arrangement without addressing the reason for it invites a person to rebuild it.

Where the old routine still lives

     An organisation keeps what it knows in more places than its documents: in people, routines, system structures and the physical layout of work. At Knight, people held an intention while the servers held executable code. At Northline, the old arrangement lived in Mia's head, in the spreadsheet folder, and in one customer's expectation. Archiving the file reached one of those places.

     Late in the session, someone described preparing meals in advance so that there was no evening decision to make. The evening looked effortless because the effort had happened earlier. In the same way, Mia's spreadsheet was still part of the arrangement that worked. It held information the system lacked. Calling her continued use an old habit would not make that information unnecessary.

     That revealed three separate actions the new system had been treating as one.

• Deleting the file removes access and leaves its purpose unfulfilled.
• Migrating the notes moves the information and leaves their status unresolved.
• Deciding which notes are obligations, which are preferences, and who may override them is a third action.

Do we automate Mia?

     The group reached a position I had not planned. The question had been which of Mia's remaining work to automate next. The answer was to draw the boundary for the current iteration.

     Booking import, technician assignment and travel calculation stayed automated. Resolving clashes between legitimate commitments, calling customers, and deciding whether a preference still stood remained with a person - with a name, authority, and a clear place in the workflow.

     Herbert Simon's account of satisficing helps explain why a team might stop searching once an option meets its acceptance criteria. That can be a sensible way to decide under limited time and information. The difficulty here was deciding what the criteria applied to.

     Northline had agreed a threshold in advance, exactly as last month's session required. But the manager was carrying the result from one arrangement into a decision about another. The number stayed the same while the thing it was being used to justify changed.

A threshold has a scope as well as a value.

     The next iteration could move that boundary again. It would need evidence about the work being moved, including what would replace Mia's contribution.

What we concluded

     A closing ritual has to do more than address the leftovers. Same way, Knight had already decided Power Peg's fate. Each leftover needs a decision, someone responsible for carrying it through, and evidence that it has taken effect.

     At Northline, that would mean checking that the restrictions actually change the proposed schedule, settling the informal promise with the customer and a colleague handling a clash when Mia is away. Archiving the spreadsheet would address none of those things. There would still be things the team had missed. Someone would need to receive the next exception, have the authority to resolve it, and decide whether it exposed a gap in the arrangement.

     The iteration could then finish. The known handovers would have been checked. The remaining human work would have an owner and support. The next surprise would have somewhere to be addressed by default.

     We arrived wanting assurance that nothing old could return. We left asking what the old arrangement had been doing, what would do that work now, and how we would know the handover had happened.

You have not finished an automation decision until you can say what stays human, and why it was there in the first place.

On Building Transferable Cases

The fictional case and the staged reveal

     I had left Knight Capital out of the session on purpose. A trading firm losing 460 million dollars in 45 minutes is too far from directly relatable experience. The fictional company was built as the opposite: small, ordinary, close to the group's own work. Nobody could look up the ending, so participants had to reason from the facts given at each stage.

     Revealing it in stages did the rest. Each stage gave one new fact, which kept people from jumping to a conclusion and made them notice what the previous stage had left out. Nursing education has used this format for years under the name unfolding case study: the learner commits to a judgment, then receives information that tests it and reflect.

Where the personal half drifted, and why

     The private-life part of the session went to habits in general: how they form, why they fail, how long they take. That is a good topic but it was not this month's topic. I wanted automation at home, the same question as at work: what did the old method handle that the replacement does not?

     The reason is in the analogy research. Gick and Holyoak showed in 1983 that people given one example almost never transfer its structure to a new problem on their own; given two examples and asked to compare them, they transfer far more often, because the comparison makes the shared structure visible. Gentner's work says the same from the other side: an analogy works on relational structure, and surface similarity is what people reach for when the structure has not been pointed out.

     That is what happened. I had one worked case which did include a habit automation. When presented, we lost the thread, and in the momentum, I switched the whole group to the topic of habits. If I had brought two or three everyday cases with the same three parts as the fictional one - an old method, a replacement, and a specific thing to look for - the group would have compared them and mapped the structure across.