3 reasons why DOE rollouts fail (and what to do about it)

If you’d like to keep learning about DOE after you're done with this article, make sure to check out our other DOE blogs, download our DOE for biologists ebook, or watch our DOE Masterclass webinar series.

Nobody likes to admit it, but Design of Experiments (DOE) rollouts fail all the time. It’s a huge problem, and our industry needs to talk about it. Why? Because even with all the enthusiasm in the world, if you can’t make it work—with your people, with your lab, and with the goals you have in mind—you’ll never realize a shred of benefit.

The truth is that most do okay, and some do extremely well, but then there are the few who drop the ball so badly that “abject failure” would be an understatement. Consider the cost of a failed rollout: think of all the days, weeks, months, dollars, and missed opportunities thrown down the drain. It’s an expensive mistake that few can afford to make.

So what is it that makes the difference? Why do some organizations succeed while others fail? What is it that they’re doing differently? I’ve been lucky enough to have a front-row seat on countless DOE rollouts and, in my experience, the main reasons behind failure are not what you think.

Your chances of success are more about the cultural and operational change you’re able to make, than the science you’re trying to do. There are three big ways in which change doesn’t land; let’s take a look at the three biggest reasons why.

Reason 1: you don’t know why you’re doing it in the first place

If you ask someone “Would you like to optimize that?” most people will say “oh…sure!” even if they have no idea what they’re even going to optimize. After all, we all love optimizing (who doesn’t?) and just the idea of optimizing something, anything, can be tempting enough to make us follow the advice that usually comes along with it, no matter what it is we’re even optimizing.

The same logic applies to DOE. Do you want to use DOE? While the answer may be yes, there could be many reasons why you might say yes. Perhaps you’re looking to screen factors, or explore and map an experimental space, or assess the robustness of a system. Knowing your “why” up front helps set the strategy that guides you. This, in turn, will help you answer the many questions that will come up along the way and, just as importantly, helps get people on board with your mission.

When we don’t know why we’re running a DOE we get lost, waste time, and throw money down the drain. Take assay development for example. What is most important to you? Is it cost? Robustness? Signal? Do you have 100 buffer components to screen, or do you already have a functional, but unreliable and noisy assay? This context provides the direction your DoE campaign should take, so it’s critical to know what you’re going for. An assay optimized to maximize signal may look very different to an assay where you want to minimize cost.

Similarly, when it comes to rolling out DOE in your organization, the pavlovian enthusiasm after hearing “optimize” can quickly fade when the reality of learning new techniques, changes in working practices, and new scientific challenges all set in. 

This is why a clear and regularly repeated mission for your DOE rollout is so important. An inspirational message, repeated often, has a much better chance of sinking in to the understanding of your team and helping them stay the course. You’ll have a much easier time navigating the challenges of a DOE rollout if people are bought into “achieving science with more confidence,” than those on a mission of “thoroughly and efficiently finding scientific results.”

I know which one I’d get up for in the morning.

How to avoid this mistake:

1. 1. Clearly define the main purpose behind using DOE in the first place
2. Craft an inspiring narrative so your people are inspired, not bored
3. Regularly communicate your purpose and narrative during the rollout

Reason 2: you bite off more than you can chew

The ocean can’t be boiled, but that doesn’t stop people from trying. Nowhere is this truer than in a DOE rollout going off the rails. Let’s continue with our assay development example.

When you are running a DOE on an assay you are limited by time, consumables, labware capacity, the amount of data you can actually make sense of, and your own patience. This is why it’s rarely a good idea to look at every possible factor, at every single level, in one single massive DOE. It’s more about striking a balance between pragmatism and exploration.

A “kitchen sink” approach wastes a lot of time and resources. Instead, an iterative approach, one where you start small and build from there, is a better way to go. You can always refine and narrow in on your optimization space, but you can’t reuse that expensive reagent. DOE is a flexible and powerful tool, but this flexibility can make it overwhelming—precisely the feeling that we might fight back against by being overly comprehensive.

To cut through some of this complexity when working with customers on their DOE rollout, I like to use the SAMR framework to help them build this picture. SAMR, a framework originally used for classroom digitization, gives a high-level overview of incremental stages for successful technology adoption:

• Substitution: the new tech substitutes the old, but with no substantive change to the working process
• Augmentation: new tech goes further and provides a functional improvement
• Modification: allows you to change or redesign your process entirely
• Redefinition: new possibilities arise, and work looks different from the way it used to be

The SAMR model: a gradual shift from no tech, to complete redefinition

This approach recognizes the reality of how “big bang” projects tend to fail. If you dive straight in at the deep end, not many people are going to know how to swim. Each stage should have a clear and different goal, building up to a final vision of how things will look once “Redefinition” is reached.

How to avoid this mistake:

• Use the SAMR framework to guide your thinking and planning from the start
• Define clear objectives for each stage of the SAMR journey
• Start small for your first iteration
• Go slowly, only moving forwards when you and your team is ready

Reason 3: you forget what, and who, is most important

I’ve worked with some of the most brilliant minds in biology, biomedical engineering, mathematics, and aerospace engineering in my career…and I still believe people can’t focus on more than 3 things at a time. When you get underway with a new project—especially one as potentially complex as a DOE rollout—it can be easy to lose focus.

Losing focus leads to bad decisions, and it happens in two ways: forgetting the goals, and forgetting who will ultimately benefit (or not) from the work: your people.

When it comes to goals, if they’re clearly established from the beginning, use them to guide your decisions as you move forward. If you’re taking things slowly using the SAMR model, make sure you’re clear on where you are in that process. But whatever the framework you use, remember that the chances are you’ll be starting with perhaps only one team in the beginning. While there must be a larger vision, that one team needs its own explicit and well-defined goal for each stage of the project. It needs to build toward that bigger vision, but the bigger vision won’t be the same thing.

When it comes to people, it’s important to think in terms of trade-offs in the work that your DOE rollout may drive. Will your choice of variables yield interesting results, but take months to work through? Can you design an ambitious project, but only at the expense of every available hour that your team has to spare? Can you put together a robust DOE, but find you have to pipette everything by hand for hours on end? Like it or not, you’re going to need lunch at some point.

The number one mistake I see customers make during a DOE rollout is to forget the people who are involved and forget the purpose of the project in the first place. In order to make the best use of a model like SAMR, you have to consider who will use DOE regularly and what the priority should be. You may want your whole company to eventually use DOE, but that isn’t where you should start. 

Whether it’s keeping on track with your goals or making decisions in the best interests of your people, prioritize brutally. Everything that isn’t on target is a distraction. Even better, have your team help you in the prioritizing process; odds are they understand the work best of all and will be more invested afterward.

How to avoid this mistake:

• Clearly define the purposes and goals at each stage of your project, and review them regularly
• Determine the human impact of the trade-offs you make—good, bad, or negligible
• Exercise brutal prioritization when you’re making decisions

How to make sure your DOE rollout stands the best chance of success

DOE is an industry-changing technique. We love to see companies make it work, and we’ve had the privilege of helping many teams on their journeys. Our customers have blazed a trail and shown us how to make this methodology stick, while those who have more gusto than organization and change management have gracefully shown their scars and what not to do. Because of their experience, I can confidently say how to avoid their mistakes and get DOE to not only stick but transform your organization. 

To transform your organization with DOE, the three things you need to remember are: 

1. Focus on your why: be able to articulate why you are bringing DOE in, and don’t be afraid to over-communicate with your team and peers about that why on a regular basis
2. Don’t bite off more than you can chew: change management is hard, especially with skeptical scientists. Start small and build up using the SAMR framework as a guide,  advancing incrementally and showing off success at every stage. All you need to do is improve 1% at a time.
3. Make the most of your people and their focus: maintain focus at each stage of your plan and don’t set goals that are too large. You can always push past your goal with shown success, but it’s really difficult to reign scattered focus back in.

DOE is powerful in the scope of a single experiment, and if you can scale this across an organization, you will have a serious competitive advantage. Just ask some of our customers. It won’t always be easy, no large-scale change is. But, I can promise you this: it will be worth it.

Interested in learning more about DOE? Make sure to check out our other DOE blogs, download our DOE for biologists ebook, or watch our DOE Masterclass webinar series.