On June 7th, Apple revealed two iOS 15 new features to the App Store during their annual World Wide Developer Conference (WWDC) that are coming with the new operating system iOS 15, sometime in the fall of 2021.
These features, Product Page Optimization and Custom Product Pages, will change the lives of mobile marketers, turning App Store Optimization (ASO) into a powerful, not-to-be-ignored aspect of any app or game developer’s strategy.
In this instantaneous guide, we’ll discuss one of these features – Product Page Optimization, also known as PPO. You can check out our second guide, looking into iOS 15 Custom Product Pages (CPP).
Apple launched Custom Product Pages and Product Page Optimization on December 8th, 2021. Read more here.
What is iOS 15 Product Page Optimization?
Product Page Optimization for iOS 14 is a brand new feature where you can test your Product Page App Creatives in the App Store with three different treatments (variations) and your control App Store page.
In each PPO treatment, you’ll be able to choose a different icon, screenshots, and app preview video. According to Apple’s statement, all creative assets will need to go through the normal App Store review process before you can use them. For each test, you will be able to decide the portion of traffic which should be allocated to each treatment. For example, 90% Control, 10% for each of the three treatments.
Once your test is live, you’ll be able to compare the performance of each variant by examining the impression volume and the install volume (and see the conversion rate of each App Store page). Each PPO test can run within a specific localization. Meaning, you can test specific treatments for specific countries.
Apple makes it clear that the intention is for developers to use this new mechanism to optimize for Search and Browse traffic (aka “Organic”), while Paid UA (referral traffic) has a dedicated solution in Custom Product Pages – CPP (For more information, check out our full guide to iOS 15 Custom Product Pages).
You’ll also have the ability to run tests in only specific localizations.
For example, if your product page is currently localized in English, Japanese and Korean, you can choose to run your tests in Japanese only.
This means that any users that see the English or Korean localized versions of your product page will not be included in the test. This gives you the flexibility to focus your test if your treatment is only relevant for a subset of your localizations.
Let’s break things down further:
- What does Product Page Optimization mean for ASO?
Only Organic. If you are fully adopting Apple’s new features and using custom store pages for all your paid UA traffic, the only traffic that will reach your default App Store product page is organic traffic, composed of search and browse traffic.
This means that setting up three variations on your default App Store product page and comparing their performance vs. the control variation will actually allow you to optimize your conversion rates for real organic traffic.
- What are the challenges of APP Store Product Page Optimization?
A testing tool alone isn’t enough, you need a decision-making protocol.
In-Depth Analysis: Apple doesn’t seem to provide any real testing functionality. Yes, they’re about to provide you with a way to compare the performance of each variation but they’re not going to provide you with statistical insights regarding which variation is really performing better.
Here at Storemaven we know from six years of testing experience that the testing method itself, i.e the way in which you make a decision as to which variation is really performing better, is 10x more important than the testing technology.
In other words, the fact you have access to run a basic test doesn’t mean you are going to be able to positively increase your conversion rates.
- Will the Google Experiments conundrum repeat itself?
Google Experiments – Google’s own tool that helps developers A/B test its app store – have been available for years. But, from heavy research we did into the platform, scanning all global Google Experiments and detecting those that concluded with a winner that’s not the control, we found that less than 15% of such experiments/tests end with a winner.
Why? There are a few reasons. One is that most test owners under-invest in the research & hypothesizing phase, the creative design phase, and the analysis phase. Whether it’s lack of time, resources, or understanding, this is the norm.
Another example comes from web A/B testing that has been available for around two decades. According to research from AppSumo, less than 12.5% of tests produce results.
Another issue is called “Peeking”. Let’s explain: Within Apple’s new iOS 15 app A/B testing feature, you have the ability to run a test for up to 90 days. During this time you will be able to compare the performance of each variant and you’re supposed to decide when to stop the test and implement the winner so that all your (organic) traffic will go to it.
This opens up a huge issue that Evan Miller, one of the world’s top experts on A/B testing & statistics, wrote extensively about. “Peeking”. What does it mean? Essentially, someone will start a test, look at the result every few hours or every day and once they see the results they want to see (one of the variants beat the control) they’ll conclude and stop the test.
But once the “winner” is called and applied to all 100% of the traffic, our “peeker” is usually disappointed to find that at best, no real improvement in conversion rates occurred, and at worst, conversion rates dropped. This is due to the fact that the test results had very low statistical significance.
Let’s recap: All the official details about Product Page Optimization
Similarly to Custom Product Pages, Apple has provided more color on Product Page Optimization (PPO), or the capability to run native a/b tests on your default App Store Page.
As a reminder, PPO only runs on your default product page, that will, if you leverage Custom Product Pages properly by driving all of your paid UA traffic to them, receive solely organic traffic. So PPO in a nutshell is a way to A/B test your “organic” App Store product page for organic search and browse traffic.
- Product Page Optimization a/b tests will be shown only to iOS 15 users (or later)
First a side note that’s important to take into account. Only iOS 15 users will be “eligible” to view a PPO test. As of now, iOS 15 adoption stands at about 56% and I expect it to soon cross the 70% mark. As adoption continues to rise, more sample traffic will be available for testing.
Source: https://mixpanel.com/trends/#report/ios_15/from_date:-93,report_unit:day,to_date:0
- Your can test your screenshots, app preview videos, and icons through up to 3 different treatments (or different product page variations)
Source: https://developer.apple.com/app-store/product-page-optimization/
For PPO a/b tests, you can test your product page screenshots, app preview videos and icons. For icons though, you’ll have to submit a new app version with the icons you want to test in the binary of the app, as Apple will ensure that a user who downloaded your app through a variation with a certain icon will also see that icon on their homepage.
If you’re applying a winner to all 100% of users and that winner had a different icon than the one in your control variant, you’ll have to include that new icon in the next app version submission.
- You can test up to 3 different treatments for each a/b test and choose the traffic proportion
Source: https://developer.apple.com/app-store/product-page-optimization/
You can set up to three different treatments for each PPO a/b test and you can choose the proportion of your overall traffic that gets to the test, in increments of 3%.
For example, if you allocate 40% of your traffic for the test, then 60% of the traffic will continue to the original product page. The 40% will be split for each treatment (if you have two treatments each will get 20% of your overall traffic).
Source: https://help.apple.com/app-store-connect/#/dev811bef199
As a reminder, if you configured your mobile marketing and UA in a way that all your paid UA traffic reaches Custom Product Pages, then the traffic eligible for the test will be solely organic search and browse traffic.
- You can choose the localizations you want to include in the test
For each test, you can configure the localizations you want to be included in the tested audience. This means that if you choose all localizations (the default) for your treatments, then users will see a localized version in their language.
Source: https://help.apple.com/app-store-connect/#/dev811bef199
If you excluded a certain localization from the test, the users in that localization will only see your original product page and won’t be included in the test.
5. You can estimate your test duration and if it’s feasible, you’ll reach at least 90% confidence in the results
There is a pretty helpful feature that allows you to estimate the duration of your test. It allows you to choose a desired improvement to CVR and see as a result the amount of impressions you’ll need to reach at least 90% confidence, and if you have sufficient traffic. Tests will be closed automatically after 90 days regardless of reaching confidence or not.
6. Custom Product Pages might appear in featured placements
According to the announcement, although Custom Product Pages won’t appear when a user searches the App Store, they might appear in the Today, Games, and Apps tabs if the editorial team finds it suitable. I don’t expect this to happen a lot anyway, but take into account it might.
7. When you apply a treatment as a winner while the test is still running, it’ll automatically end the test
This action can’t be undone, so be careful when applying a treatment as a winner. When you apply a treatment, only the screenshots and preview videos will be implemented. If the winning treatment has an icon, you have to include it with your next app version as the default app icon. Check out our complete guide to iOS app icon requirements.
8. More color on what the monitoring and results of the tests look like
Source: https://developer.apple.com/app-store/product-page-optimization/
A few cool things in the PPO a/b test results screen are:
- You can see the lift from the baseline CVR for each treatment and you can choose a different baseline (as one of the treatments) which allows you to “mark the control”. You’ll see if you managed to reach confidence between two treatments and not just between a treatment and the control.
- When sufficient data is collected, you’ll start seeing more information in the treatments CVR cards such as: performing worst, performing better, or the most interesting one “likely to be inconclusive” which predicts that a variant is unlikely to produce a CVR change either way by the time the tests will end.
- You can see the CVR by treatment in a time-series graph that’ll help you understand seasonal (most likely weekly seasonality) effects on the results of the tests.
Last important notice: You can now submit multiple entities for review besides your app version and your default product page metadata. In-App Events, Custom Product Pages, Product Page Optimization treatments. When you’re awaiting review for these entities, your ability to submit more assets (including your app version) will be on hold until the App Store Review team finishes the review. So be diligent in your submission planning, especially with time-sensitive entities.
Conclusion
Although Product Page Optimization (PPO) is a nice to have testing tool, whether running natively on the App Store or not, it will need a great decision-making protocol to enable the deployment of better converting App Store product pages. Get early access to our Growth Intelligence Platform, and specifically our full App Store Product Page Planning services.
