Testing & Effective Learning

Testing & Effective Learning

“You must ask the kind of questions to which you don’t currently know the answer, but if you did, you’d change the way you operate. If you already know the answer, or if you are testing an insignificant detail that doesn’t matter, you’ll just be wasting time and money.” Peter Diamandis, Founder of The XPRIZE and Singularity University

To zero in on the testing you need, you have to balance two perspectives. On the one hand, you want to be aware of and prepare for all key risks/thresholds as you move from where you are now to what you want your social venture to be when it is in its ‘steady state’ (i.e. your model has proven to be a success and growth is predictable and steady. A typical barometer in this regard is what your venture will look like five years after it starts). This will also help make sure you are building towards your goals rather than building a bridge to nowhere (this is a significant risk for start-ups, as discussed below). On the other hand, a lot of your current assumptions will be wrong, so you don’t want to go into too much detail too far down the road.

To get the best of both worlds, we suggest a ‘Zoom Out and Zoom In’ approach.

*Please note that this blog on testing and learning has been designed to help you get into the nitty gritty aspects of creating a specific test for your venture. Should you be looking for an introduction to the concepts and some high level tips, check out our blog on Effective Learning

Zoom Out: build a high-level, ‘logical trail of risk reduction’ that will not only take you from where you are now to where you want your venture to be its steady state, it does so while minimizing risk

Reminder: In Steps 1 to 3 described below, you shouldn’t get into granular detail, as chances are many of your assumptions will be wrong.

Step 1: Work Backwards from Your What Your Venture Will Look Like in its Steady State

-Think of your steady state goals;

-At a high level, identify the key risks and assumptions relating to your venture (the “Operating Costs and Core Numbers” worksheet may be a good starting point in this regard);

-Think about what makes each risk risky (what do you fear, how can being wrong hurt you?); and

-At a high level, think about how you could eliminate or reduce each risk.

For example, let’s say your steady state goal is to buy a van so you can make 1,000 deliveries: To lower that risk, what if you rented a van before you bought one (renting is cheaper/ safer than buying)?  To lower that risk even further, what if you could borrow a van from a friend to do 500 deliveries before you rent one (borrowing is cheaper/ safer than renting)? To lower that risk even further, what if you felt you could do 200 deliveries in the car you have before you borrow a van (using what you have can be cheaper/ safer than borrowing, especially if you haven’t built up the orders to justify a van)?

Now you can work your way up to 1,000 deliveries in lower risk stages, assuming you satisfy the requirements to move on to the next stage. If you don’t satisfy the requirements to move on to the next stage, you can stop with low risk and limited exposure to your venture and reassess what’s going wrong. For example, if you are consistently doing 200 deliveries in your own car, you can move with confidence to the next stage of borrowing a van. If you aren’t consistently doing 200 deliveries, try to figure out why you aren’t.

Ideally, you can create a chain working backwards from your steady state with smaller and smaller, and therefore less risky, experiments until you get to what will be your initial pilot. It’s like a matryoshka/ Russian nesting doll of risk.

Step 2: Work Forward from Your Pilot

If you get stuck working backwards from you steady state, and only if you get stuck, think through what you would like to do for your pilot and how far, step by step, further testing can get you to your steady state. Again, try to make it as low risk as possible so you minimize your exposure while justifying moving on to the next step.

Why is it important to start with your steady state? If you go straight to your pilot without thinking through how it can be part of a chain that leads to your steady state, you run the significant risk of your pilot leading nowhere and spinning your wheels. As well, the steady state can be a powerful tool in helping you develop even your early pilots.

Bear in mind, you are still staying high level at this point – you don’t need a detailed series of granular tests, you just want a chain of steps that builds towards your steady state in a logical way, and fleshes out and then reduces the nature of each risk.

Step 3: Mind the Gap

Now that you have worked back from your steady state in Step 1 and looked forward in Step 2 regarding your pilot, it is important to see if there is a gap between the two (any missing steps or pieces of information) and how you can close that gap. A gap exists whenever you have no idea how to safely get from one point to the next, and failing to get to that point could severely hurt your venture. Then flesh out what you need to do to figure out how to close that gap, in a way that is de-risked. You may not (in fact, you won’t) have the answers, but you need a plan of how to get those answers. That is why Einstein said “If I had an hour to solve a problem I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions.” If you can’t even come up with a low (ideally, “no”) risk plan to get the key answers, that’s a sure sign you are in trouble.

Once you have filled in all the gaps and have your ‘logical trail of risk reduction’ from where you are now to where you want your venture to be in its steady state, flag the ‘death-star’ risks – those risks that, if things don’t go the way you want them to, could destroy your venture.

Zoom In: look at your immediate next step, and dig deep on how to test it in a way that would warrant (or guide) moving forward 

Based on your zooming out, pick your next step and devise a detailed test that will eliminate the risk of that situation:

a) Try to matryoshka/ Russian nesting doll that specific risk even further: Now is the time to get granular. Try to break down that risk even further into a sequence of decreasing risks that ends with the smallest risk possible;

b) Identifying exactly how to remove the uncertainty about that smallest risk with the least expensive, easiest and fastest test to remove possible;

c) Record what you anticipate the results of that test will be and how that would justify moving on to the next step; and

d) See if it actually produces the results you need to move on to your next test or step.

For example, Ilana, the founder of Zambia Feeds, wanted to deal with the problem of malnutrition and unemployment in the Copperbelt region of northwestern Zambia (MacMillan, 6). Zambia Feeds planned to sell chicken feed to small plot owners who would raise chicks, use them to feed their family, and earn income by selling the surplus chickens. After Ilana confirmed her estimates about market price and feed costs, rather than proceeding to rent a huge warehouse with mixing machinery, her next step was to hire six part-time workers to mix the feed by hand on a concrete floor inside a shed she persuaded someone to loan her. The mixed and bagged product was delivered to customers in the test region who, in turn, used the feed to raise and sell chickens. Ilana noted the expected costs and labour time for mixing the seed, the expected time and results for raising the chicks, and the anticipated market price for the chickens. Then, she compared those against the actual results and decided when and how to move on to the next step in her venture.

Still Struggling With the “Zoom Out/Zoom In” Framework? 

Think of yourself as a mountain climber. You don’t want to run out of rope, or oxygen, or fall to your death on the way to the top. First, you do a zoom out, a high level mapping of the path to get to the top. You keep an eye out for major challenges you will have to get through – say a huge rock formation that you will have to go around (requiring more rope), or a portion where you would have to do some straight up climbing (requiring carabiners; the small, metal clips that allow climbers to attach themselves to the mountain wall). You wouldn’t get too granular, that would be impossible as you would need to know the exact conditions and that is hard to predict at the bottom of the mountain. But you want to make sure there are no huge mystery gaps – if it is a complete mystery how you will get from summit D to summit E, that’s a prescription for disaster. Then, when you actually start up the mountain and reach a given point, say the huge rock formation you knew you would have to get around, you zoom in, examining it in great detail. Does that stretch still look safe to go around? Next, you test the rocks and ice right next to you to see if they are solid and it is safe to proceed. You look for a specific, detailed, safe path, and test each step before you proceed. And yes, the combination of a great mentor; advisory teams; and a friendly, wise skeptic is like having a super Sherpa.

Even with this metaphor, you may struggle to decide which issue should fit in the ‘zoom out’ versus the ‘zoom in’ when it comes to your venture. Don’t worry about that. It is more important to make sure that, through both exercises, you have a basic map of where you want to go, identified all the risks, and, when the time is right, further break them down into a series of increasingly granular de-risking tests.

Key Tips:

  • Always Write Down What You Think Will Happen: make sure you note your understanding and estimations of what will happen, and the views of your key advisors, before you test: “… the failure of the ‘launch it and see what happens ‘ approach should now be evident: you will always succeed – in seeing what happens. Except in rare cases, the early results will be ambiguous, and you won’t know whether to pivot or persevere, whether to change direction or stay the course” (Ries 161).
  • Common Assumptions: don’t look at it until you have tried to identify all your key assumptions (otherwise you will be overwhelmed), but here is a list of common assumptions for you to review. Feel free to read it and compare it against the worksheet you have filled out and see if it helps you catch any key assumptions you may have missed.
  • Avoid ‘Russian’ Ahead: good testing and monitoring won’t just inform your work, it will drive behaviour. A cautionary tale in this regard relates to the Soviet Union during the Cold War. The leadership decided to track a nail factory by focusing on productivity, defining productivity by tonnage of nails produced. Factory leaders responded by switching production to huge railway spikes that weren’t really needed. When the party bosses figured out what was going on, they switched the measure to number of units. The factory leaders switched to tiny nails that were mostly useless (Grenny 23). While the Soviet Union example may be extreme, the law of unintended consequences is everywhere. To stay on course, keep in mind, the most important thing to gauge is “are you improving your targets’ lives?” (Christensen 209), and testing is all about learning how to make your offerings more effective at improving those lives (Moore 222).
  • A Chain is Only as Strong as its Weakest First Link: don’t forget, while the most expensive items in your plan may jump out at you as the key areas of risk, your chain of risk should also work its way back to the most basic of assumptions/risks: does the customer actually want your product and are they willing to pay what you want them to?
Interested in learning more? Check out our A.S.E.S.S. Worksheet #5: Testing & Learning for an opportunity to dive deeper into this concept as it relates to your social enterprise and for an example of how another organization has tackled this issue. If you have any questions about the content of this blog or the worksheet mentioned, please contact us at info@tricofoundation.ca.
This blog is part of the Trico Foundation’s effort to capture what it learns as it works to close gaps in society by building capacity and innovation in social entrepreneurship. The flagship effort in this regard is our A.S.E.S.S. TLC (Tools, Links and Coaching) page where you can find most of our thinking on these issues and worksheets designed to help advance your social enterprise idea. Stay tuned for further blogs on the entrepreneurial mindset (including crossing the chasm, effective learning, and overcoming cognitive bias) and the mechanics of successful social enterprises.

We are excited by the progress we are making, but know we still have much to learn. We would value your perspective and feedback. Please join the conversation on twitter


Resources Utilized

Aulet, Bill. Disciplined Entrepreneurship. John Wiley & Sons, 2013.

Christensen, Clayton M., Taddy Hall, Karen Dillon, and David. S. Duncan. Competing Against Luck: The Story of Innovation and Customer Choice. HarperCollins, 2016.

Diamandis, Peter. “SUCCESS = EXPERIMENTATION” accessed from http://www.diamandis.com/blog/successexperimentation

Grenny, Joseph, Kerry Patterson, David Maxfield, Ron McMillan, & Al Switzler. Influencer: The New Science of Leading Change, Vital Smarts, 2013.

Gunther McGrath, Rita, and Ian C. MacMillan, Discovery-Driven Growth: A Breakthrough Process to Reduce Risk and Seize Opportunity. Harvard Business Review Press, 2009.

MacMillan, Ian and James Thompson. The Social Entrepreneur’s Playbook. Wharton Digital Press, 2013.

Moore, Geoffrey A. Crossing the Chasm: Marketing and Selling Disruptive Products to Mainstream Customers. HarperCollins Publishers, 2014.

Reis, Eric. The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation To Create Radically Successful Businesses. The Crown Publishing Group. 2011

Submit a Comment