Complete Guide to iOS 15 A/B Testing in the App Store

Apple’s iOS 15 now provides new tools for ASO and UA specialists: App Store A/B testing. Let our experts guide you in setting up your app A/B tests.

As of December 8, 2021, Apple introduced a line-up of new capabilities – launching the new iOS 15 features, Custom Product Pages and Product Page Optimization. The available tools for ASO and UA managers to optimize conversion rates on the App Store have now changed, with the ability to A/B test in the App Store through these new features. 

This article will map the possible ways to do A/B testing in the App Store. 

How Do You A/B Test on iOS 15 Apps?

Product Page Optimization

In late 2021, Apple started offering a new way to run native App Store product page A/B tests on the App Store. You can now run A/B tests on iOS 15 apps in the App Store with Product Page Optimization, Apple’s native A/B testing solution for your default product page.

You should use iOS A/B testing when you want to run A/B tests on your native organic traffic (both Browse and Search) for screenshots, videos, and icons.

Product Page Optimization: the new iOS 15 feature to run A/B tests in the App Store.

Three crucial insights on the connection between organic mobile installs and paid installs

What billions of data points taught us on paid and organic installs?

    App Store Connect A/B Testing

    iOS 15 A/B testing is conducted through App Store Connect, Apple’s platform for tracking app performance. Within App Store Connect, you can access your app’s analytics, view reports, test your apps, and more. 

    How does App Store Connect A/B testing work?

    App Store Connect allows you to A/B test your product page. Through App Store Connect (ASC) you’ll be able to create up to three additional variations (treatments) to test against your control variation (the current default product page). A test can run for up to 90 days. You’ll also be able to choose the percent of traffic that goes to each variation.

    Each variation will need to go through the App Store review process. You can test different screenshots, app preview videos, or icons. You can submit new variations to the App Store (they need to go through the normal review process) with no new app version update unless you’re also submitting an icon, which in this case you’ll need to include it in the binary of the app as a new app version. This is to ensure users eventually see on their home screen the same app icon they saw on the product page before they installed. 

    You’ll be able to see impressions, conversion rates to install (App Units), and improvement which will show the uplift a certain treatment was able to produce based on an internal Apple Bayesian statistical model.

    Check out our Product Page Optimization Guide for a complete overview of the new iOS 15 feature.

    Custom Product Pages

    A second iOS 15 feature is Custom Product Pages, which is Apple’s solution that allows you to create up to 35 custom App Store product pages, each with different assets (screenshots, app preview videos, and promotional text). 

    Each product page will receive a unique App Store URL you’ll be able to use with any UA campaign or even email marketing campaigns, or mobile websites (or anywhere you can include a link to your product page) to drive a specific audience to. 

    Seven tips to start using PPO with Storemaven’s solution

      Can I Run A/B Tests on Custom Product Pages?

      No, Custom Product Pages do not include a native A/B testing solution. However, you’ll be able to execute a testing methodology by running paid UA campaigns that go to multiple Custom Product Pages, each representing a variation of the test. Then, you will be able to measure the results through the data App Store Connect exposes. 

      This is ideal to use when you’re trying to optimize conversion rates for your UA paid campaigns by measuring which creative and messaging are more effective in driving installs for that specific campaign.

      Run paid campaigns on Custom Product Pages

      How Testing Custom Product Pages Works

      Testing Custom Product Pages works by running multiple campaigns (with the same ad creatives and targeting) and creating custom product pages as test variations, driving traffic from the test campaigns in parallel.

      This requires significant planning and real-time management to ensure the test results are clean and are not biased by inconsistent traffic over time to each variation and inconsistent targeting (as a result of, for example, one campaign that is already optimized by Facebook and is serving higher quality traffic than its counterpart that is yet to be optimized (still in the learning phase) and is driving less quality traffic to another variation. 

      Similar to Product Page Optimization A/B testing, you can submit new custom product pages straight to review without a need for a new app version.

      What can you test with Custom Product Pages?

      With custom product pages, you can test:

      • screenshots
      • app preview videos
      • promotional text

      Icons cannot be tested on CPPs; therefore, you don’t need to go through an app binary review.

      What data do you get from Custom Product Page tests?

      With CPPs you can get more data about the test than what you get with Product Page Optimization A/B tests. In addition to impressions, installs (App Units), and conversion rates, you’ll also be able to see retention data as well as Average proceedings per user for each CPP which will enrich the test data with downstream metrics to gauge the impact of each such funnel and CPP on revenues and retention.

      For example, you might find out that a certain CPP, given it highlights specific features of the app or game is driving new installs coming in through that CPP to use that feature more, and having a higher LTV.

      These issues require a robust testing mechanism and system which we at Storemaven now offer.

      Five things you must know about Custom Product Pages

        Storemaven’s iOS A/B Testing Solution

        In light of the new changes to iOS 15, Storemaven won’t leave you stranded! Storemaven now offers A/B testing solutions: App Store product page A/B tests for rich and advanced A/B testing of Product Pages and Custom Product Page Testing

        On the Storemaven platform you can create replicated App Store product pages as mobile web pages, set up multiple variations of your page to test a certain hypothesis, and receive a unique URL for a test that will randomly and evenly divide traffic among all variations.

        You then are able to use the unique Storemaven test URL with any UA campaign to run a test and collect in-depth data. 

        What can you A/B test?

        You can test any asset on a product page such as the title, subtitle, icon, screenshots, app preview video, description, What’s new text, and even your ratings and reviews. You can also test your App Store product page gallery orientation. 

        Besides impressions, installs, and conversion rates you’ll be able to see for each variation of the test: Video watch rates, screenshot scroll and view rates, page scroll rates, description views, review engagement, and more.

        With this in-depth data you’ll be able to answer questions such as: 

        • In which order should I show the scenes in my app preview video? 
        • In which order should I show the screenshots? 
        • Why do users that explore my page convert much less than people that don’t and how can I improve that?

        When can I use Storemaven’s iOS 15 A/B Testing Solutions?

        There are different times when Storemaven’s A/B testing solutions will come in handy. For example:

        1. A/B testing new pages/apps. When you want to run an App Store product page or a custom product page test for an app that is not live yet, it is a good idea to run A/B testing first to see which pages will perform better. 
        2. Understanding and improving conversion rates. If you wish to dig deeper to understand how to improve conversion rates, try our testing solutions. You can understand why a certain variation has won by looking at on-page engagement analytics, such as:
          1.  video watch rates 
          2. screenshot view rates 
          3. scroll rates
          4. decisive users (users that install after 0-3 seconds without engaging with the page) vs. explorer rates (users that install after 6-12 seconds following some engagement with the page such as scrolling the screenshot gallery, or vertically down the page itself

        Understanding how users consume the information on your App Store product page will allow you to improve conversion rates by understanding which parts of a video are most effective, which screenshots are those that actually drive a user to install.

        In a nutshell, Storemaven install experience testing provides you with the maximum data and insights to make conversion rate improvement on your page. So when you want to make improvement leaps with your conversion rates, give us a call and request a demo

        In Conclusion

        Starting with iOS 15 you’ll have multiple tools to leverage when you are trying to improve organic and paid conversion rate on the App Store. 

        The teams behind the most successful apps and games in the App Store are always leveraging all tools they have and develop a testing culture that involves CPP testing (for paid traffic), PPO testing (for organic traffic) and Storeamven install experience testing for guidance on creating improvements on both the paid and organic side. 

        We have built the only ASO platform in the world to plan, manage, test, analyze, and make decisions around your App Store product pages, allowing you to leverage all testing options in the most accurate way possible, leading to repeatable improvements to the KPIs you’re measured on. And we even have the creative studio to back you up. Our product page design services will happily create data-based designs for your A/B tests.

        To see our platform in action, take a look at this page and feel free to ask for a call with our ASO consulting team to walk you through how you can start implementing this today.

        Jonathan Fishman
        About Jonathan Fishman
        Jonathan is Storemaven's VP of Marketing and Growth. Before joining Storemaven he spent ten years commanding tanks, working on Wall St., consulting high-growth companies, and exploring Black Rock City. In his spare time, he likes building things from wood, listening to Frank Zappa, and spending time with his daughter.

        Join 10,000 other mobile marketers and stay on top of your craft with the mobile growth newsletter