Harvard Business Review Staff Photo
ORIGINALLY PUBLISHED: Harvard Business Review
Michael Schrage, Columnist
The book The Only Rule Is It Has To Work is the true story of how a couple of clever quants, Ben Lindbergh and Sam Miller, tried to bring sabermetric superiority to the Sonoma Stompers, a minor league baseball team nestled in California’s wine country. The self-described “statheads” have the chance-of-a-lifetime opportunity to test out their own Moneyball-style theories when the management team and owners of the Stompers invited them to run operations as part of a learning experience and team promotion.
Even if you loathe baseball, it’s a terrific yarn. This is what real-world data-driven organizational transformation looks and feels like. Practically speaking, the book is more insightful — and useful — than Moneyball. The latter is unrealistic for many organizations because, for one thing, there aren’t many bosses like Billy Beane, and, for another, major league teams enjoy resources that most companies don’t. Persuading teams to embrace statistics they don’t really understand makes a nifty tale of data-driven despair. Getting people to consistently and reliably act upon real data is a real leadership challenge. Superior knowledge doesn’t guarantee greater effectiveness.
Lindbergh and I recently discussed lessons learned from their experience bringing analytics to an ambivalent and under-resourced enterprise. These lessons deservedly extend beyond dugouts and diamonds into C-suites and boardrooms.
Effectively communicating and sharing analytic insights is as important as finding them. Lindbergh and Miller were consistently startled — and frustrated — by the resistance to even their most compelling insights. Yet remarkably little investment went into making their analytics easy, accessible, and engaging. No gamification, edutainment, or coaching epiphanies here. They were rigorous around every aspect of their analyses, except selling them. If they had a do-over, said Lindbergh, they’d be smarter missionaries and marketers for their data-driven recommendations.
“One of the keys is how to convey the insights to the players and the intermediaries in a way they’d appreciate them,” Lindbergh observed. “We were sort of at a disadvantage there. We had no way to do that consistently for managers and coaches. Our own background came back to bite us.”
Lindbergh and Miller behaved as if their evidence — or, more accurately, their presentation of the evidence — was obvious or self-explanatory. It wasn’t. Their data-driven reasoning was alien to (and sometimes alienated) the Stompers status quo. That ongoing dynamic propelled much of the season-long drama and conflict.
“I do think we eventually arrived at a kind of coaching epiphany with Yoshi [the final Stompers manager],” Lindbergh noted in a follow-up to me. “It just took a lot longer than we’d have liked (which was largely our fault).”
Predictive analytics create organizational winners and losers, not just insights. Competitive organizations want results. Players aren’t just supposed to play well; they’re expected to win. Predictive analytics explicitly seek to pick winners, sideline losers, and manage risk. That makes them as much a source of power as insight. Serious analysts know their numbers will influence who plays, who’s seen to have potential, and who gets cut. The players know this, too. But who really benefits when analyst and their spreadsheets gain power? Machiavelli proves a better guide than mathematics.
“The players we signed had a natural allegiance to us,” Lindbergh acknowledges. “They were only there because of our spreadsheets and our stats… Anything that gets them a job in pro baseball they were inclined to like.” These Stompers were more open to data-driven suggestion not because they necessarily bought into Moneyball metrics, Lindbergh observes, but out of gratitude and loyalty. “We figured they’d be more receptive because to what we had to offer,” he notes. “The downside was that they might not have the desire.”
The players the duo didn’t sign, by contrast, knew they literally didn’t measure up to the new sabermetric standards. Even when the Stompers were winning as a team, that stuck in their craw.
But analytics provoked the greatest conflict with managers and coaches. Smart spreadsheets created a rival power center to traditional leadership and openly subverted managerial influence. Increasingly, the numbers got the last word. “There was a lot of jockeying for authority,” Lindbergh agrees. “That was uncomfortable for everyone. But we couldn’t avoid it.”
Superior assessment doesn’t assure ongoing improvement. Lindbergh and Miller got the greatest pleasure and professional satisfaction from identifying and signing undervalued talent. Building the best possible roster within punishing constraints proved a statistical and financial puzzle they enjoyed solving. “Our spreadsheet was smarter than we were,” Lindbergh noted time and time again.
But what happened after talent came on board? Even though Lindbergh successfully scrambled to acquire state-of-the-art video equipment and build a world-class analytics infrastructure for the Stompers, the overwhelming focus is on acquiring and using talent — not cultivating it. The book tells shockingly few stories about coaches using data-driven insight to help players get better. No vignettes celebrate how a talented underachiever breaks through by getting a spreadsheet tutorial.
Analytics for acquiring the best possible Stompers talent dominated; analytics to make those acquisitions better was an afterthought. “It wasn’t that we were uninterested in improving players once they were there, or that we thought that wasn’t a worthwhile pursuit,” Lindbergh acknowledges. “We were just very limited in what we could do…because of the team’s lack of resources and everyone’s lack of time. The Stompers hardly had a coaching staff, and we couldn’t get a company to give us a coach the way we could get one to give us a computer system….So we were sort of forced to focus on what players had already demonstrated that they could do.” The result? There was no measurable culture of improvement.
When your people become datasets, you risk treating them that way. The oddest aspect of The Only Rule is its seamlessly schizophrenic shifts from seeing players as “people” with real lives and treating them as “data”’ with measurable attributes. On the one hand, Lindbergh and Miller wanted to emotionally connect with the talent their analytics had identified; on the other, they really, really liked the numbers. Their objective analytics subjectively biased their expectations. “We went in to this 100% objective,” Lindbergh says. “We thought our spreadsheets objective, impartial and unbiased….We tried to reduce things down to performance, rather than what we did — sort of like a blind hiring practice.”
But throughout the season, less quantifiable — and unquantifiable — human elements consistently forced compromise and recalculation. Interpersonal dynamics influenced performance outcomes, and “fudge factors” crept into analyses.
Lindbergh concedes more of their objectivity became subjective, and that pushed them to become more objective about their subjectivity. They became more sensitive to and sophisticated about the human dimensions they couldn’t measure. But whether that made them more effective, Lindbergh still doesn’t know.
Recognize, respect, and remember what’s not being measured. Miller and Lindbergh did a fantastic job bringing sabermetric sophistication to a minor league team. They brilliantly leveraged limited information to target undervalued talent. They pushed hard and sometimes succeeded in influencing game-time decisions. But as the season wore on, what wasn’t being measured — self-motivation, team chemistry, manager/player compliance with statistical insight — assumed greater importance.
Much like the way sabermetrics began when innovators like Bill James recognized that traditional metrics did a poor job explaining player value, The Only Rule highlights the relentless nature of real-world statistical insight. Today’s breakthrough creates tomorrow analytic opportunity. A willingness and ability to follow data-driven advice makes talent even more valuable. Measuring that matters.
In our talk, Lindbergh agreed his Stompers life would have been easier and more effective if the team could have identified coachable players amenable to sabermetric insight. “I would pay a premium for that quality if we could assess it accurately,” he agreed. “That would definitely effect how I perceive player value.”
Smart quants self-quantify. The more important analytics become, the more imperative it is to measure their impact and influence. Metalytics — analyzing the analytics — define how quants gain insight into how they create insights, as well as how effective they prove at communicating them. (Rising to this challenge is a key message in Lazlo Bock’s excellent book Work Rules.) Self-quantification offers the surest way for ambitious statheads to lead by example.
I asked Lindbergh how comprehensively Nate Silver, his uber-quant employer at the popular FiveThirtyEight website, quantifies how his quants are doing. “There may be some quantification going on,” Lindbergh laughs, “but if it is, it’s concealed from me.” Indeed.
These six takeaways highlight that high-impact predictive analytics are as much about power as knowledge. In the final analysis, turning undervalued opportunities into desirable outcomes demands the power to persuasively act on insight.
Ironically but appropriately, just as The Only Rule came out, Lindbergh posted a superb analysis on FiveThirtyEight identifying a surprisingly undervalued opportunity: Statheads are the best free agent bargains in baseball. “We’ve mined the data and charted the proliferation of these numbers-savvy front-office staffers over time,” Lindbergh and collaborator Rob Arthur write. “Yes, there are more of them now than ever, and yes, they’ve had a demonstrable effect on their teams’ fortunes.”