Mobile App A/B Testing: 5 Mistakes to Avoid

This piece was first published as a guest blog on AppTweak.com
Photo by freestocks.


As marketers, we need to constantly be on our toes. It’s the nature of the role, specifically in the mobile growth realm. This ever-changing ecosystem forces us to work under tight deadlines, constantly readjusting our strategy and reassessing our operational methods. But in this rush for numbers and results, it’s pivotal nothing will get left behind, and no corners will be cut.

After analyzing thousands of app store creative tests, we’ve seen what works and what doesn’t, why some companies are successful and where others fall short. Whether it’s the rush to get results or something else, this list of 5 mistakes to avoid when going through the app store testing process will help you to enjoy better CVR and engagement with your users, while not throwing hard-earned dollars down the drain.

Mistake #1 – Using the same creative on Google Experiments and on the Apple App Store

The Apple App Store and Google Play are fundamentally different platforms, and they should be treated as such.

It’s not only the design itself or the layout differences, but it also has many more implications: different traffic is being driven for each store. The user base is different, apps and games that are popular on one platform could very well not enjoy the same success on the other.

Due to these significant differences between the two stores, it’s crucial to avoid a one-size-fits-all approach. A quick glance at our data below will show the consequences of making this mistake.

Google Play creatives on the App Store

Mistake #2 – Testing without a strategy

We as marketers are always under pressure to show numbers, results and better conversion rates, but there’s one stage in this testing cycle that just can’t be rushed. Conducting proper research and developing strong hypotheses to test with is crucial for your success. Choosing weak hypotheses or too minor and subtle changes to your creatives won’t end up making an impact, it can also harm your efforts, as no valuable insights and no clear direction or hypothesis are driving the test cycle forward.

Below is a good example of a strong hypothesis: running an Icon test in which the variations tested different aspects: Character only, gameplay element only, and combination of character and gameplay element. This way, no matter the winner, it will have a clear insight into what your audience reacts better.

Mistake #3 – Ending and concluding tests too early

Many failed tests are also the end-result of that pressure to show numbers quickly – in this case, you may have a good strategy, but the test itself was closed too early.

it’s more than a statistical aspect. For example, let’s say a mobile game’s highest quality audience is more active on the weekdays than on the weekends. If a test is starting mid-week and closes after only four days in which traffic was received, there’s no doubt the results won’t be accurate, and any conclusion or resolution that will be driven from these results will miss your goals in the long run, as well will damage future tests cycles.

Additionally, it’s rare to see a variation that is the clear winner on every single day of the test. You should always aspire to have a wider date range to give you the confidence of consistency. From our data, gathered from more than 5M users, we recommend running tests for a minimum of 7 days and to push for a longer average of 10-14 days.

Mistake #4 – Measuring results in an invalid way

Bear with us for this one, or go bring your data maven to assist: Even if your test is run flawlessly, it’s easy to make the mistake of measuring results using a model that’s too simplistic.

Some companies, for instance, use pre and post-test analyses that calculate the average CVR two weeks before the change and compare it to the average CVR after the change. This approach, more common than one would think, fails to take into account many variables that affect conversion and app units in the store, such as keyword ranking, overall ranking and category ranking, being featured, the level of user acquisition (UA) spending, competitors’ changes (e.g., creative updates, UA spending, etcetera), and more.

Without accurately measuring the impact, companies can arrive at the wrong conclusions: that creative changes don’t matter, or that conversion rates were hurt when, in reality, they experienced a positive uplift. For more on this subject, read this.

Mistake #5 – Not optimizing for the right audience

Whether you develop a causal game or have a functional app, you need to think about your audience. And different apps and games have very different users coming their way. When you know where your audience is coming from and what they are looking for it’s so much easier to cater to their needs. Let’s assume you have a casual game – mostly attracting decisive users who don’t care to spend additional time or effort researching, it’s easy to figure out you’ll need to spend your money and time optimizing your First Impression Frame – what these decisive users will have a chance to look at in the 3-6 seconds they’ll glance at your page.

Another consideration is dependent on whether your game is available on other markets. If you’re targeting a variety of countries, you can localize or culturalize your creatives, which means adapting your app store assets to the language and culture of different regions.

5 Expert Tips for Your ASO Keyword Research
App Store Marketing in the Wake of COVID-19: Uncertainty is Our Nemesis, Data is our Savior