ADDING EXPERIMENTS
TO PRODUCT DEVELOPMENT.

Developing products and services users love is easier said than done.
Much more than brainstorming and making ‘best guess’ decisions. The development process involves extensive research and creating numerous hypotheses.

Adding Experiments to Product Development greatly improves success by validating ‘gut feel’ before investing extensive time and dollars..

 

 

This approach can sound convoluted to those yet to adopt experiments, but it should become a core part of your development process. Although going with ‘gut feel’ is tempting and often easier, rarely is it the safest option.

Experiments are not only quick and inexpensive, they indicate your likelihood of success. Therefore, help teams make insight based decisions instead of acting on assumptions.

Adding experiments to product development, is not as easy as it sounds. So we’re here to offer you some guidance. Steps you can adopt to get the most out of your product/service design experiments.

Step 1: Research, Research, Research

First, it’s important to understand what is happening now. Understand what the existing market offers and for who, and why you’re creating the product. For that reason, research is paramount to identifying user problems or jobs to be done.

To find out what opportunities to develop, you need to gather and analyse relevant information.

You can do this in a variety of ways, but the most common techniques include:

  • Surveys
  • Interviews
  • Usage tracking
  • Direct observation
  • Jobs to be Done
When compiling data with these techniques, keep in mind your target audience. That means keep in mind the following categories:

  • Demographics (age, location, gender, etc.)
  • User Personas (characteristics — interests, behaviours, habits)
  • User Needs — Functional (tasks), Emotional (feelings), Social (status)
Finally, the research stage ends with identifying criteria which highlights problems and opportunities. By doing so, you establish a reference point for change. Constraints to work within. Criteria for success.

 

Step 2: Formulating Hypotheses

After you’ve gathered insights. Create numerous hypotheses for the items you wish to test. Some will be rather simple, such as “If we do this, we expect ‘that’ will happen”, but others will be more complex.

As a rule, you can consider your hypotheses valid if they correspond to the real world and have enough evidence to support them.

Before you move on to testing, you need to answer the following:

  • Can your hypotheses be tested?
  • Can your hypotheses be proven wrong?
Whether a hypotheses is testable or can be proven false depends on the methods you intend to use. Prepare experiments to ensure they return actionable data. Also, create feedback paths to capture expected and unexpected insights from participants.

Once hypotheses have been validated. Work with the team to select the highest priority.  Be clear why you’ve chosen one over another.

 

Step 3: Designing Experiments

Now you’ve chosen a set of validated hypotheses, you should design 3–5 experiments that aim to test the following:

  • Highest possibility of success
  • Largest opportunity
  • Ability to solve the biggest problems
The next step is to run the experiment with people in your target audience.  Remember, to gather as much data as possible, or at a minimum enough to confirm your hypothesis. In order for the experiments to be successful, you need to provide the participants with:

  • The purpose of the experiment and what will happen with the results
  • Provide the user an authentic experience
There are multiple ways to build your experiment inexpensively. Some popular ways include:

  • Clickable prototype mock-up
  • Role play the User Journey
  • Wizard of Oz
  • Concierge service
  • A/B testing
  • Landing page sign-ups

Step 4: Running Experiments

Of course there are many ways to implement experiments to product development. Common techniques include:

  • A/B testing
  • Geolocation based
  • Focus groups
  • 3rd party service
  • Off-brand public prototype

An “off-brand prototype” experiment is particularly useful for larger companies. The experiment runs without brand bias. Avoids damaging the brand if experiments fail ‘awkwardly’. Will also show customer/brand sentiment, if the branded release has contradictory results.

Specify the techniques you’ll use to capture insights. Technique may include note taking, video/audio recording, usage tracking and analytics.  Focus on gathering as many insights as possible.  Collate the evidence to assess results and reflect on actual vs predicted outcomes.

Consider the following questions…

 

Gathering Insights

It’s not just about users doing the experiment. There are more valuable insights to focus on, such as:

  • How did they approach the experiment?
  • What steps were taken?
  • How many re-tries were necessary?
  • Did they require support?
  • How did the user feel emotionally?
  • Did they display confidence when in use?
  • Were the interactions predictable for the user?
  • Did they follow up to find out more?
  • Had they tried to find this solution before?

Assessing Results

When the experiment has concluded, it is time to analyse the results. Consider:

 

  • Does data validate or invalidate the hypotheses?
  • What worked / What wowed?
  • Areas to improve or remove?
  • What could be integrated?
  • How did assumptions measure up?
  • Were there unexpected outcomes?

Take Action!

If you’re going to benefit from taking the time to run the experiments, this last step is critical. The results of the experiments need to be actioned. Assess the results and prioritise the impact. Decide on which experiments to follow through with. Things to consider when evaluating results are:

  • Which delivered the biggest impact?
  • Which results align with set goals?
  • Did any experiment truly delight the users?
  • Which one delivered mutual benefits?
  • Do any results complement your existing roadmaps?
  • What is the degree of implementation?
  • Which adhere to your Product Principles

Aim to find the best fit between user needs, your objectives and results of the experiment. Experiments aligning all three should proceed to the next iteration of the solution.

Final Thoughts

To go one step further in the process, we recommend creating an experience map as users proceed. Ask participants update their experience during each stage. This way, you capture thoughts and emotions as they happen. Providing a more intuitive view. Below is an example of a simple experience map.
Source: https://experience.sap.com/skillup/experience-mapping/

As we said at the beginning of this article, our aim is to offer guidance. This is not a strict set of rules to abide by.  Quite the contrary, we’re always interested in hearing your insights.  Feel free to share your thoughts. We’d love to add the techniques you’ve used!