All the Ways to A/B Test your Product Page for iOS 15

If you’re in ASO and UA, Apple are presenting you with a great gift: iOS 15. You’ll have many new tools added to your arsenal, but harnessing their capabilities won’t be easy. Let the experts guide you.

Coming later this year (2021), the available tools for App Store Optimization (ASO) managers and User Acquisition (UA) managers to optimize conversion rates on the App Store & Google Play will change. 

As Apple introduced a line-up of new capabilities on iOS 15, this article should organize your thoughts and map all the possible ways to A/B test App Store moving forward. 

Apple launched Custom Product Pages and Product Page Optimization on December 8th, 2021.

On Apple iOS 15

In late 2021 there will be a new way to run native App Store product page A/B tests on the App Store. The second capability Apple is introducing, custom product pages (CPPs) will not include native A/B testing but will allow mobile marketing teams to manually run tests. 

As a reminder, according to the new setup, if you leverage Custom Product Pages (CPPs), and drive all of your paid UA traffic to one of the available 35 custom product pages, the remaining portion of traffic, the organic side (Browse and Search traffic) will land on your default product page.

More on Custom Product Pages and how to set them up.

Three crucial insights on the connection between organic mobile installs and paid installs

What billions of data points taught us on paid and organic installs?

    Product Page Optimization (PPO)

    All the Ways to A/B Test your Product Page for iOS 15 - 1
    • What is it: Apple’s native A/B testing solution for your default product page
    • When to use it: When you want to run A/B tests on your native organic traffic (both Browse and Search) for screenshots, videos, and icons. 
    • How does it work: Through App Store Connect (ASC) you’ll be able to create up to three additional variations (treatments) to test against your control variation (the current default product page). A test can run for up to 90 days. You’ll also be able to choose the percent of traffic that goes to each variation. 
    • What you can test: You can Each variation will need to go through the App Store review process. You can test different screenshots, app preview videos, or icons. You can submit new variations to the App Store (they need to go through the normal review process) with no new app version update unless you’re also submitting an icon, which in this case you’ll need to include it in the binary of the app as a new app version. This is to ensure users eventually see on their home screen the same app icon they saw on the product page before they installed. 
    • What data do you get about the test: You’ll be able to see impressions, conversion rates to install (App Units), and improvement which will show the uplift a certain treatment was able to produce based on an internal Apple Bayesian statistical model.

    Custom Product Pages (CPP)

    All the Ways to A/B Test your Product Page for iOS 15 - 2
    • What is it: Apple’s solution that allow you to create up to 35 custom App Store product pages, each with different assets (screenshots, app preview videos, and promotional text). Each product page will receive a unique App Store URL you’ll be able to use with any UA campaign or even email marketing campaigns, or mobile websites (or anywhere you can include a link to your product page) to drive a specific audience to. 

      This solution does not include a native A/B testing solution, but you’ll be able to execute a testing methodology against it by running paid UA campaigns that go to multiple Custom Product Pages, each represents a variation of the test, and then measure the results through the data App Store Connect exposes. 
    • When to use it: When you’re trying to optimize conversion rates for your UA paid campaigns by measuring which creative and messaging are more effective in driving installs for that specific campaign. 
    • How does it work: By running multiple campaigns (with the same ad creatives and targeting) and creating custom product pages as test variations, driving traffic from the test campaigns in parallel. This requires significant planning and real-time management to ensure the test results are clean and are not biased by inconsistent traffic over time to each variation and inconsistent targeting (as a result of, for example, one campaign that is already optimized by Facebook and is serving higher quality traffic than its counterpart that is yet to be optimized (still in the learning phase) and is driving less quality traffic to another variation. 

      These issues require a robust testing mechanism and system which we at Storemaven are offering, to check this out visit this page. 
    • What you can test: Similar to Product Page Optimization testing, you can submit new custom product pages straight to review without a need for a new app version. As you can test only screenshots, app preview videos, and promotional text, and an icon can’t be tested on CPPs you don’t need to go through an app binary review. 
    • What data do you get about the test: With CPPs you can get more data about the test than what you get with Product Page Optimization. In addition to impressions, installs (App Units), and conversion rates, you’ll also be able to see retention data as well as Average proceedings per user for each CPP which will enrich the test data with downstream metrics to gauge the impact of each such funnel and CPP on revenues and retention. 

      For example, you might find out that a certain CPP, given it highlights specific features of the app or game is driving new installs coming in through that CPP to use that feature more, and having a higher LTV.

    On Storemaven A/B Testing

    Install Experience Testing

    • When to use it: When you want to run an App Store product page or a custom product page test for an app that is not live yet, and whenever you wish to dig deeper to understand how to improve conversion rates by understanding why a certain variation has won by seeing on-page engagement analytics such as video watch rates, screenshot view rates, scroll rates, decisive (users that install after 0-3 seconds without engaging with the page) vs. explorer rates (users that install after 6-12 seconds following some engagement with the page such as scrolling the screenshot gallery, or vertically down the page itself. 

      Understanding how users consume the information on your App Store product page will allow you to improve conversion rates by understanding which parts of a video are most effective, which screenshots are those that actually drive a user to install. 

      So when you want to make improvement leaps with your conversion rates and 
    • How does it work: On the Storemaven platform you can create replicated App Store product pages as mobile web pages, set up multiple variations of your page to test a certain hypothesis, and receive a unique URL for a test that will randomly and evenly divide traffic among all variations. 

      You then are able to use the unique Storemaven test URL with any UA campaign to run a test and collect in-depth data. 
    • What you can test: You can test any asset on a product page such as the title, subtitle, icon, screenshots, app preview video, description, What’s new text, and even your ratings and reviews. You can also test your App Store product page gallery orientation. 
    • What data do you get about the test: Besides impressions, installs, and conversion rates you’ll be able to see for each variation of the test: Video watch rates, screenshot scroll and view rates, page scroll rates, description views, review engagement, and more. 

      With this in-depth data you’ll be able to answer questions such as: “In which order should I show the scense in my app preview video? In which order should I show the screenshots? Why users that explore my page convery much less than people that don’t and how can I improve that?

      In a nutshell, Storemaven install experience testing provides you with the maximum data and insights to make conversion rate improvement on your page.

    In Conclusion

    Starting with iOS 15 you’ll have multiple tools to leverage when you are trying to improve organic and paid conversion rate on the App Store. 

    The teams behind the most successful apps and games in the App Store are always leveraging all tools they have and develop a testing culture that involves CPP testing (for paid traffic), PPO testing (for organic traffic) and Storeamven install experience testing for guidance on creating improvements on both the paid and organic side. 

    We have built the only ASO platform in the world to plan, manage, test, analyze, and make decisions around your App Store product pages, allowing you to leverage all testing options in the most accurate way possible, leading to repeatable improvements to the KPIs you’re measured on. And we even have the creative studio to back you up. Our product page design services will happily create data-based designs for your A/B tests.

    To see our platform in action, take a look at this page and feel free to ask for a call with our ASO consulting team to walk you through how you can start implementing this today.

    Jonathan Fishman
    About Jonathan Fishman
    Jonathan is Storemaven's VP of Marketing. Before joining Storemaven he spent ten years commanding tanks, working on Wall St., consulting high-growth companies, and exploring Black Rock City. In his spare time, he likes building things from wood, listening to Frank Zappa, and spending time with his daughter.

    Ready to see our platform in action?
    Request a demo