Many digital-native firms have enjoyed tremendous growth. For instance, it took Google and Amazon only five years to reach $1 billion of revenue; Facebook, six years and Spotify, eight years. What these firms have in common is that they run thousands of experiments each year and have systematically approached business experiments as a key paradigm for growing and scaling their businesses.

Business experiments are far beyond just tests; they require firms to build holistic activity systems that involve best practices before, during, and after conducting a business experiment. These systematic practices help firms benefit from business experiments, and this article highlights the characteristics of these practices

1. Before the Experiments

Clarify key objectives and North Star Metric: Before running each experiment, it is important to identify the primary objective of the experiment and identify a corresponding indicator, or a North Star Metric (NSM), to measure the achievement of the objective. Such NSMs can be financial indicators (e.g., revenue growth) or customer behavior measures (e.g., the number of sign-ups to a firm’s website, retention rate, time spent using a product). Before each experiment, it is important to clarify one, rather than multiple, NSM. This is because achieving intended experimental objectives typically requires many iterations. Without a clear focus on an NSM, firms can only respond to unpredictable experimental results rather than follow a clear goal to search for growth or improvement opportunities. Furthermore, firms should also focus on leading, rather than lagging, indicators as NSMs. This allows firms to take preventative actions. For example, if a firm intends to use experiments to reduce customer churn, the firm can focus on a leading indicator of customer churns, such as customer engagement or customer experience with its product as an NSM. If the firm is able to improve the leading indicator NSM via experiments, it can address the customer churn problem before it happens.

In the same vein, business experiments should not overemphasize financial indicators as NSMs; rather, firms should focus on customer-behavior measures as NSMs, which are commonly leading indicators of final measures. This can also be illustrated by a clock analogy: changes to the minute hand’s movement will not generate noticeable changes in the hour hand immediately; however, in the long run, the hour hand will move accordingly. Similarly, in business experiments, customer behavior indicators correspond to a clock’s minute hand, while financial indicators are the hour hand. If an experiment is successful, it is easier to observe the effect on customer behavior than on financial performance, but if customer behaviors continue to improve, they will eventually improve the financial performance metrics.

Develop deep customer behavior insights to guide experimental design. Since the aim of many business experiments is to influence customers to take more expected actions, firms need to develop intimate knowledge of customers’ behavioral patterns or structure. For example, Pinterest, an image sharing and social media service firm, wanted to achieve a user non-subscription rate of 0.1%, comparable to the industry average. Their customer behavior analysis shows that the majority of users unsubscribed within the first month of sign up, so they developed experiments to enhance customer experience during the first month, which ultimately reduced the overall unsubscribe rate. Had the firm not developing these customer behavior insights to guide its experiments, it could have undertaken a series of experiments without focusing on the key problem of non-subscription in the first month.

Customer behavior insights can be achieved by examining the whole customer journey of interacting with a firm. By looking at customer behavior data at different stages of the customer journey, the firm can identify growth opportunities or customer pain points at different stages of the journey and design relevant experiments that are tailored to different stages of the customer journey.

Develop cross-functional guidance and coordination mechanisms:** Many business experiments are essentially cross-functional. For example, a marketing team often leads efforts to increase product awareness; a sales team finds ways to convert potential customers into paying customers; a customer success team ensures repeat purchases; and a product team is tasked with developing and improving product features. Thus, to avoid disconnected efforts by different functional teams, firms need to set clear strategic priorities and establish effective mechanisms to synchronize different experimental efforts, thus effectively executing growth initiatives by creating synergy. Accordingly, many firms establish growth teams to lead and coordinate such cross-functional growth experiments and efforts.

2. During the Experiments

Follow customer psychology to guide experimental iteration: Since customer behaviors are influenced not only by physical product attributes but also by personal emotions and preferences, firms need to develop in-depth understandings of customer psychological and emotional needs when conducting and iterating business experiments. For example,

Google used different experiments to encourage advertisers to create multiple versions of their same ads, a desired customer behavior. After initial failure, it followed consumer psychology principles to adjust its experiments, which led to a 140% increase in the number of advertisers who created at least three versions of their same ads.

Set a clear time limit for achieving the expected experimental outcomes: Failed business experiments are part of the normal learning process. Firms need to decide how long they are willing to continue with experiments that do not generate the expected results. Facebook Publisher’s Small and Medium Size Business (SMB) group, for example, relied on business experiments to increase its sales. In late 2017, the firm had to experiment for three months before achieving its desired outcomes. During those three months, the SMB group segmented customers into particular tiers, product/industry verticals, and regions to target different customer groups that could present attractive growth opportunities. To evaluate these opportunities, the SMB group’s sales team experimented with different tactics, including cold calling and building customer relationships, carefully analyzed different experimental efforts and the findings, monitored progress and improvements, and shared with Facebook’s leadership.

Managing the scale of the experiments: Although increasing the scale of business experiments improves the credibility of experimental results, firms need well-orchestrated plans to scale up promising experiments or pull back undesirable ones, thus minimizing the potential disruptions of experiments on existing business activities. Firms can adopt structured approaches to manage the scalability of business experiments. YouTube, for example, follows a multistage approach to decide whether to scale up its new product feature experiments. It first evaluates the results of an initial experiment in the “experimentation stage,” and if the results are satisfactory, YouTube moves the experiment to the next phase, known as the “alpha” stage. If the alpha stage is successful, it leads to the “beta” stage. Only once its beta version is successful is a new feature/product made available to the general public. During this process, the firm set very clear criteria to evaluate and “graduate” experiments from one stage to the next.

3. After the Experiments

Evaluate results beyond key objectives and NSMs: Although firms pay close attention to experimental results based on key objectives and NSMs, they should not ignore insights from other experimental results. For instance, Shopkick, a firm that offers a retail shopping loyalty app, launched an experiment on geofencing notifications. Users received these notifications when they were near Shopkick’s partner retail stores. When the firm launched the app, its NSM is that the geofencing notifications would generate a 20-30% increase in visits to its partner stores. However, in reality, it only achieved a 2-3% increase in store visits. Through in-depth analyses, Shopkick discovered that the increase in store visits was far greater for new users, at more than 20%. For existing users, the notifications also had a positive impact on activation by reminding them to use the app every time they visited a store. Thus, firms need to go beyond the preset key metrics, or NSMs, thus uncovering deep insights.

Focus on both percentage gains and the absolute impacts: When analyzing experimental results, firms should focus on both percentage gains in key metrics and on the absolute impacts of the business experiments. As an example, let us say that a firm runs two experiments to increase its active user base. In the first experiment, the firm targets 1,000,000 users, 500,000 of whom are currently active. The experiment helped recruit 15,000 new active users, resulting in a 3% gain. In the second experiment, the firm targets 1,000,000 different users, only 5,000 of whom are currently active. The second experiment results in the acquisition of 3,000 active new users, achieving a 60% increase in total active users. Although the percentage gain in the first experiment is much smaller than that in the second, the number of new active users acquired in the first experiment is much higher, resulting in a larger impact on the firm. Therefore, the first experiment should not be viewed as a failure or less successful.

Consider organic market growth when assessing experimental results: For many experiments aiming to assist business growth, firms need to disaggregate the growth driven by the experimental efforts from those improvements caused by macro market organic growth. If all or most of such growths are due to market upsurges, the business experiments need to be modified.Firms can use A/B tests to disaggregate different sources of growth. For example, Facebook designates 25% of its customers as a control group in a given year and then excludes this group from any experiments and employee efforts. At the end of each year, the performance of the control group can be attributed to organic market conditions or growth. By comparing the growth results of the control group and the test group, Facebook is able to determine the effectiveness of its growth experiments/efforts while excluding organic market growth.

In summary, to benefit from business experiments, firms need to adopt a system of practices before, during and after conducting experiments. At the same time, the effective implementation of these practices requires firms to establish a compatible organizational culture and mindset. For example, it is inconceivable for firms to fully adopt an experiment-enabling activity system without embracing a failure-tolerant and data-driven culture. Together, the business experiment activity system and enabling organizational culture reinforce each other to help firms maximize the benefits of the business experiments.