How Imperfect Foods built a culture of experimentation and increased customer retention

As Ben Clarke at Fast Company said, “These days, the true test of how innovative a company can be is how well it experiments.” By that standard (and many others), Imperfect Foods is one of the most innovative companies out there.

Founded in 2015 with a mission to reduce food waste and build a better food system for everyone, they offer imperfect (yet delicious) produce, affordable pantry items, and quality meat and dairy on a weekly subscription basis via their website. With over 200,000 subscribers, serving 25 cities, they’ve saved 80M pounds of imperfect food from being thrown away.

Patti Chan, who leads the digital product department, is an avid supporter of experimentation and experiment-driven product development. However, with a small team, she was challenged with experimenting often while still keeping up with the day-to-day demands of the business. 

Companies like Netflix have teams of over 300 people running experiments, while her team consisted of six engineers, two QA engineers, a designer, and a PM that were responsible for four products for Imperfect Foods.

Patti needed a scalable way to run experiments and measure results without stretching her team too thin. That’s why her team implemented a data infrastructure that made their testing dreams a reality.

Here’s the experimentation infrastructure Imperfect Foods use:

• Collect user event data from the Imperfect Food website via Segment.

• Send user event data to Split.io and AB Tasty for experimentation.

• Send user event data and test results to Snowflake for data warehousing.

• Run queries and build reports in Mode Analytics and Amplitude.

Growing a culture of experimentation

As a small team with multiple responsibilities, it’s often difficult to make time for experimentation. Experiments require a lot of planning and the right technology to make sure you’re getting effective results from the experiments you do choose to run. But not experimenting at all can lead to countless missed opportunities.

Patti knew having the right experimentation framework and technology in place would be the best way to build a culture of experimentation, without overburdening her team. 

Here’s the process they landed on:

1. Define your problem and hypothesis clearly. Know the question you want to answer and set up your test around that question.

2. Pick a reliable leading measure to move. Don’t choose lagging measures because it will take too long to see results and gauge impact in the short term.

3. Do things that are not scalable to start. You’ll get to your findings faster and can worry about scaling things like automation and admin tools later, when you know there’s value.

4. Don’t stress over smaller sample sizes. You should aim instead for a large sample size.

5. Choose bold ideas. You won’t see big gains without breaking new ground.

6. Share your learnings broadly. This helps all departments benefit from the findings of each experiment and creates a culture that celebrates and prioritizes experimentation.

Of course, the right process will naturally fall short without the right technology to support it. So, Patti and the Imperfect Foods team implemented a stack that helped them offload countless hours of work. They rely on Segment as their customer data infrastructure to collect and deliver the customer data needed to run tests and evaluate results. In addition, they use Split.io and AB Tasty to reduce the amount of work required to change their UX and route traffic to the right experiments.

“Segment is the glue that holds our experimentation infrastructure together.” – Patti Chan, VP of Product @ Imperfect Foods

With the right experimentation process and data infrastructure in place, Patti and the team can run a test per week with the equivalent of less than five dedicated team members.

Enhancing the customer experience for big retention gains

While not all tests generate positive results — in fact, according to Patti’s estimates, as many as 50% fail — when you do get a winner it can have a big impact on the business.

Patti’s team came up with an idea when they were brainstorming ways to improve the customer experience at Imperfect Foods. They had a hypothesis that allowing customers to select foods they didn’t want in their monthly box would improve customer satisfaction and loyalty. Despite it being complicated logistically — this feature could lead to thousands of custom boxes and substitutions — they wanted to test it to see if the idea was viable.

They built the functionality in about three weeks, tested it, selected their target market (Los Angeles), launched the feature to 50% of their LA subscribers, and started waiting anxiously for the results. The team was stunned when the results came in. Despite low adoption in the first iteration, customers that used the new feature were 21% more likely to be retained than users who did not have access. 

It was clear that this was something that should be invested in further and rolled out to the rest of Imperfect Food’s customer base. 

Applying experimentation to internal processes, too

Experimentation is often incorrectly associated with only changing customer facing UI. However, experiments are also an opportunity to improve operational efficiency. One example of this is how Patti and her team worked with the customer care team at Imperfect Foods. 

Before July, the customer care team didn’t have quick access to critical information that could help them diagnose customer problems, like the delivery status of an order. Patti and the team hypothesized that getting this information to their care team members in real-time with fewer clicks during their calls would help drive customer satisfaction and reduce the time to resolution for those support calls.

Patti and team built a feature for their customer care associates that exposed delivery details like if an order was out for delivery, if the delivery was marked as completed, if there was photo confirmation, and so on. Similar to their external experiments, the team rolled it out to a select few support team members and compared their results with the rest. Yet again, her team struck gold! They managed to reduce support call times for the category by 10%.

Harvesting the right ideas for your business

For Patti and the team at Imperfect Foods, experimentation allowed them to explore more ideas and ultimately build a better product for their users. All-in-all, the results her team shared speak for themselves:

• Comprehensive experimentation framework and tech stack implemented

• 22 experiments run in 6 months

• 21% increase in retention (for Los Angeles test group)

• 10% reduction in time to resolve a customer status inquiry

One great idea has the potential to change your entire business. To get to that great idea, you’re going to need to plow through quite a few duds. It’s not easy, and not every experiment comes along with double digits metrics gains, but building a culture of experimentation within your organization will always prove fruitful.

The experimentation process can be disappointing and humbling at times. Do it anyway! The confidence we get from knowing that our solution not only fits a spec but solves for a real customer need is invaluable. – Patti Chan, VP of Product @ Imperfect Foods