Almost ten years ago, I read a blog post about the RAT: the Riskiest Assumption Test. My hazy recollection is that this was the origin of the RAT; certainly the post spread like wildfire, and I'd not come across it as a way of thinking before.
When the post came out, I was working at Pivotal Labs, a best-in-class XP and lean product shop. Clients would come to us and pair with our teams of product managers, developers and designers. Pairing in that context meant a consultant sitting next to their client counterpart for 8 hours a day (except for the ping pong breaks), writing code or stories together and imparting 'the Pivotal way' to the client. I was working on Chop Chop, a rapid grocery delivery app Sainsbury's was launching as a response to changing shopper habits. We'd launched into two Sainsbury's stores and geographies and we were trying to figure out if we could get to product-market fit.
It had long since felt like MVP had lost its way in the five years since The Lean Startup had come out. I can't find it in myself to go and remember all of the many different arguments around what is an MVP, definitions of what it means, how to scope and test it, but the phrase jumped the shark, and the RAT turned up into a wider product environment that was desperate for a new way of talking about validating propositions, starting small, and testing as you go.
I can't remember if it was before or after the RAT post that me and Papa worked with Mitch to figure out the unit economics of Chop Chop. We'd decided, along with the client, that if we got to 50 orders per store, per day, that counted as PMF and it was justification for scaling further. And we worked out the formula for all the levers we could possibly pull or attempt to influence - money in was defined by number of orders, number of stores, average basket size, delivery fee. Money out was how many staff, time to pick the items, how many phones, how long to deliver. We prioritised our assumptions and opportunity areas by risk then chipped away bit by bit trying to experiment against some of those levers: if we reduce the delivery fee, do we get more orders? If we introduce a machine learning algorithm on search, do we increase basket size? And so on, and so on, until we got confident enough to scale out further. Those opportunity areas and risky assumptions got shaped by the usage data we were already seeing in our two stores: I remember there was a strong trend for 17 being the magic number for actually checking out. And that we had a large portion of users who opened the app something like 7 times before checking out - so the opportunity became, how do we get people to check out sooner?
I didn't enjoy my time at Pivotal particularly, but I remember that team and that project as being one of the best I've worked on. We embraced the RAT and ran at it wholeheartedly, and eventually built a whole business around it.
Earlier this year, Chop Chop got folded into the main Sainsbury's app. And last month, the Treasury published their Test and Learn annex to the Magenta Book, the government's evaluation handbook. The word assumption shows up on page 3 in the glossary: "Criteria that need to be met for a policy or intervention is to have the intended impact. These may or may not hold true in practice. Test and Learn activity prioritises assumptions that are most uncertain or risky, as these pose the greatest threat to achieving outcomes." It's mentioned 52 times across the annex.
Obviously I want this to be a neat story about how long it can take ideas to gain legitimacy at the centre of government, how ideas have a long tail, how we're playing the long game around public design and user centred service delivery. In reality, apparently the Test & Learn team were just sitting near the Evaluation lot at HM Treasury, and the approach somehow resonated enough to warrant an annex. So really I guess it's a story of how you'll never quite know how it comes good - if published guidance is the definition of coming good. We've got the Treasury, lads! But it's not done, is it. So we keep chasing Gatsby's green light (or the Treasury's Green Book), boats against the current.