You’ve worked hard to develop an app that your audience will love. Unfortunately, creating a world-class product is only half the battle. You also need to effectively promote your app to its target market. You can use influencers, run ads, and use other marketing tricks to make this happen. But high traffic metrics will only benefit your business if they convert into high quality installs.
Enter A/b testing App Store pages.
In this article, we’ll discuss what app store testing is, how you can a/b test your App Store Page and your Google Play store listing page, the elements of your app store page that should be tested on a regular basis, which tests to run, and a whole lot more.
Let’s dive in!
What is App Store Testing?
App store testing is: the act of A/B testing an app store page in Apple App or Google Play stores to improve the app page conversion rates.
Testing your app store page on a regular basis, can ensure you’re capitalizing on your marketing efforts that drive impressions to your page, by converting the most impressions to installs, maximizing your conversion rates, and lowering your user acquisition costs.
Let’s dive deeper into the benefits of App Store A/B testing…
What Problem Does App Store Testing Solve?
If you don’t understand what drives your target market to install your app , or more importantly what pushes them away from your page without installing, you’ll never be able to:
- Increase both organic and paid conversion rates
- Lower user acquisition costs, by improving the conversion rates from a paid ad to install.
Without gaining this understanding not only that you’ll leave growth on the table, you’ll allow your competitors to gain an edge on you. Think about it this way, users searching for apps around a specific topic (let’s say, Travel apps) will be more likely to install your competitor’s apps, and more likely to find it in the store as the higher conversion rates will lead to better ranking and discoverability.
Moreover, your competitors’ UA teams will be able to acquire the same users for a lower cost as their paid funnel will be significantly more converting, effectively lowering their cost per install (CPI).
Getting App Store testing right you’ll be able to:
- Get to Know Your Users: By testing different creatives and product page copy, you’ll learn what your unique audience responds to best. You can then use this information to craft better product pages and marketing content.
- Increase Organic Conversion Rates: A proper keyword strategy will enable your app to appear in Apple App and Google Play searches. But if you can’t convert that organic traffic into paying customers, what’s the point? App store testing will help you determine what your organic traffic is looking for so that you can give it to them.
- Increasing Paid Conversion Rates: Similarly, the perfect advertisement and marketing budget are useless if the traffic they drive to your product pages never end up becoming paid users. App store testing gives you the tools you need to learn why paid users don’t convert so that you can adjust your app pages accordingly.
- Decreasing User Acquisition Costs: When you understand your users and know how to increase both organic and paid conversion rates, your user acquisition costs will naturally go down. Lower acquisition costs mean greater profitability and more growth for a fixed budget.
How to run App Store & Google Play product page A/B Tests
Five specific app store testing methodologies:
So far we’ve covered why app store testing is important. Now we’ll cover the kind of A/B tests and other tests that can be run in the app stores.
1. Storemaven App Store & Google Play A/B Testing
The Storemaven app a/b testing platform uses a replicated app store environment. A replicated test is one that happens within a simulated environment. The simulated app store environment mirrors the actual app store while allowing an app developer to test new app page elements without harming real-world traffic and gaining important on-page behavioral insights about the app page.
How does a Storemaven A/B Test work?
With this type of app a/b test, you’ll create a replicated app store environment, and create several variants of your app store page each with different creatives and messaging, run dedicated traffic (a sample that represents the users you want to optimize for) through a paid ad (usually Facebook ads) where each user will randomly land on each variant, and measure how users interact with the different variants.
During the test, a sophisticated algorithm, StoreIQ, will automatically close low-performing app page variants as soon as it detects they have a low chance of winning the a/b test. This will radically reduce the amount of traffic needed for the test compared to other testing methods, and thus, the overall test traffic cost.
When the app a/b test concludes you’ll have access to deep insights that’ll uncover why a certain variation won and you’ll be able to both get a great direction for what to improve and test next as well as what to implement in the real app store to enjoy the conversion rate lift.
Other app store a/b testing options:
2. Google Play Store Listing Experiments on Android
App developers have access to Google Play’s Store Listing Experiments in the Google Play Store, which allow you to a/b test app icons, images, videos, and descriptions for your Google Play app product page.
What’s missing in Google Play store listing experiments?
You will not get data on which traffic source and audience drove the increase in conversion nor why a certain variant of the app a/b test won as it doesn’t include any on-page behavior insights when running which will show what pushed users away or drove them to install. So as you can see, when running a Google Play store listing experiment you may be missing some key insights of your app a/b test.
When should you use Google Play store listing experiments?
- If you’re just getting started with ASO Creative Optimization, and don’t have the resources for dedicated traffic for app store tests, Google Play store listing experiments can be a great tool to get some insights on your Google Play page.
- If your app is almost exclusively focused on the Google Play store with your distribution strategy.
- To amplify your iOS App Store testing, validate hypotheses that were already tested through a full testing solution.
3. App Creative Set Tests
App developers also have specific testing options within the iOS Search Ads ecosystem. These are called “Creative Sets” and will allow you to test different sets of screenshots and App Previews for your Apple Search Ads. By testing and optimizing your ads, you can improve their click through rates (CTRs) and direct response ad conversion rates.
What’s missing?
This type of test misses many of the flows users go through when discovering and installing apps. The different variants are only apparent through Search Ads, and not through the full product page. While this is a valid tool to increase Return on Ad Spend (ROAS) for Search Ads, it is unlikely to uncover insights that’ll lead to better conversion rates for organic and paid traffic.
4. Pre-Post Analysis Tests
A pre-post test is akin to a before-after picture. You simply measure a specific aspect of your app store page before making a creative change, and then measure it again after the change has been implemented, usually 2-weeks before and after the change.
What’s missing?
As it’s impossible to isolate all the different changes in your marketing efforts and their impact on conversion rates, these types of tests usually lead to the wrong conclusion. For example, the conversion rate might have dropped because of a change in the UA mix, as opposed to a real drop driven by a creative change.
5. Focus Groups and Survey Tests
Finally, you can attempt to perform app store testing by showing your product to a focus group and evaluating the responses you get from a post-group survey. This kind of test has the promise to get you detailed, real-world feedback.
What’s missing?
In this method, it’s extremely challenging to draw conclusions for your overall audience and traffic. The focus group is not capturing the natural flow of how a user would stumble upon an app and evaluate it on a product page. Users simply don’t make decisions in a way that’s comparable with the way the focus group participants will.
What Does a Good App Store Test Look Like?
Unfortunately, many app developers run app store tests that aren’t conducive to the above goals. To make sure your testing methods are up to par, look for the following in every app store test you run:
- Access to On-Page Behavior Statistics: Knowing how many users visit your app store pages and the percentage of them that convert is somewhat useful. But this information alone won’t help you improve your conversion rates. Make sure your app store testing efforts include behavior statistics so you know which aspects of your app store pages are turning users away, or driving them towards the install button
- The Ability to Run Tests Without Harming Real-World Traffic: What if you run a test and your conversion rates go down instead of up? If you run tests on actual users, you could lose a substantial amount of revenue.Imagine 50% of your actual traffic landing on a variant that converts 20% worse than your control variant. That’s why it’s essential to test your app store changes in ways that don’t harm your real-world traffic.
- Your Results Are Accurate and Trustworthy: Your testing efforts will be wasted if you can’t trust the information you receive. The results you glean from your tests should be reflected in the Apple App and Google Play stores once you implement changes.
If your app store testing methods don’t allow you to access on-page behavior statistics, run tests without harming real-world traffic, or trust the results, you should consider adjusting your approach immediately.
The Current State of App Store Testing
After facilitating thousands of app store tests, we believe we’re in a prime position to speak about the current state of app store testing. We have three observations to share in this regard:
- Most app developers quickly lose sight of why they’re running app store tests and oftentimes develop a low-quality test addiction.
- It’s all too easy for app developers to run pseudo app store tests that never contribute to the metrics they’re trying to improve upon.
- App conversion rate optimization in both the Apple App and Google Play stores has become a key growth strategy. Since App Store Testing is essential to proper ASO, mistakes in this regard will inevitably harm an app’s growth.
Don’t run an app a/b test just for the sake of running tests or simply to only satisfy the need for a “rubber stamp” to justify a certain decision. It’s not the number of tests that you run on your app, it’s the actual install growth you’re able to drive and show, and achieving real results can only stem from using a methodology that works.
We encourage you to keep these three observations in mind when thinking about how to test your app store pages! If not, you’ll likely waste time and money on faulty app tests that won’t move the needle.
What App Store & Google Play Page Elements to Test
It’s clear that app store testing is vital to the success of your app. But which elements of your app product page should you test? Are some elements more important than others? That’s what we’ll cover in this section.
In the spirit of focusing your efforts where you have the most potential to have an impact, you should always focus on the elements that have a better potential to increase conversion rates. Don’t spend your time on elements that can’t move the needle.
The key here is to understand how users interact with app store pages and you can take a look at this report we’ve made that analyzes more than 500M user sessions on the app store.
Apple App Store A/B Testing
Obviously, the Apple App and Google Play stores operate differently.
More importantly, the two platforms attract inherently different people (Android vs. iOS) which will 100% have different preferences.
That’s why it’s important to test different elements for each store. Here’s what we recommend app developers test in the iOS ecosystem:
A/B test your App Store Icon
Your app’s icon is usually the first thing a potential user sees in regards to your product. As such, it’s imperative that you make a good first impression and create an icon that conveys the value your app brings to the table.
Test different icon variations to see which one is preferred by your target audience. Just be careful. You don’t want to change your icon so much that company branding is affected.
You can expect a CVR uplift of up to 18% when optimizing for the right app store icon. Read more about the icon’s role in your ASO strategy.
A/B test Your App Store Screenshots
In the Apple App Store product page as well as in search results, users will be able to view one to three of your app’s screenshots in the first impression, depending on their orientation (portrait or landscape.) This fact makes the screenshots you choose (Apple allows a total of 10 screenshots) incredibly important.
You’ll want to choose images that clearly show how your app is different and unique and any exciting value propositions it has. Test multiple screenshots to ensure the ones you use catch the attention of your ideal users.
You can expect a CVR uplift of up to 28% when optimizing for the right app store screenshots. This piece will help you know better how to orchestrate your screenshot methodology.
A/B test Your App Store Videos
Once a user visits your App Store product page, they’ll be able to view any videos you’ve created for it. These videos are known as “App Previews” in the Apple App Store and you can include a maximum of three 30 second clips. It’s important to note that App Previews play autoplay on mute once a user views your page, meaning the first few seconds of your videos need to be visually compelling if you hope to grab your audience’ attention.
We suggest using App Preview footage to demonstrate key value propositions, show your app’s most exciting usability, etc., though you’ll want to test your videos to see which ones best connect with users.
our data tells us that CVR rates can be increased by 20%-30% when videos are done right. Have a look into our comprehensive video guide to know more.
A/B test Your App Store Titles
Your app’s name is absolutely crucial to its discoverability in the Apple App Store. You’ll want to choose a name that’s simple, memorable, leverages your brand capital, and at least hints at the function it performs.
Additionally, you’ll want to include your highest value keywords as Apple gives the most weight to keywords located in the title section. Test various app names and keyword combinations to find one that drives both a high volume of organic traffic and boosts conversion rates.
App developers also have access to a Subtitle section in the iOS ecosystem. This area will allow you to further explain the value of your app and include additional keywords.
As the title and subtitle appear in the first impression they do have the potential to increase/harm conversion rates.
You can increase CVR by up to 8% when you have the right text elements optimization strategy in place. Search this article to know more.
A/B test Your App Store Descriptions
If a user is enticed by your app’s icon, screenshots, and App Previews, they may decide to read its description as well. This is where you should highlight your app’s key capabilities and functionality. Apple recommends starting with a brief, but engaging paragraph followed by a bullet list of top features. As less than 1%-2% of users will ever read the full description this is not the lowest hanging fruit in your journey to increase conversion rates.
You can increase CVR by up to 8% with the right text elements in place. Click here to know more.
Google Play Store A/B Testing
We covered a/b testing in IOS, now let’s dive into google play store a/b testing.
A/B Test App icon in Google Play
The app icon is probably the first encounter users have with your app in the Google Play Store. Making a good first impression is very important, so creating an app icon that conveys the value your app brings to the table should be a top priority — at least if you want to increase your app traffic and conversions.
Keeping brand best practices in mind, run multiple a/b tests with different app icon variations to see which one is most effective in driving installs.
You can expect a CVR uplift of up to 11% when optimizing for the right icon in the Google Play Store. Read more about the icon’s role in your ASO strategy.
A/B Test app Screenshots in Google Play
As you’ve probably noticed by now, the visuals you include in your app store listings are incredibly important. So it should come as no surprise to learn that app screenshots in the Google Play Store should be tested and optimized on a regular basis.
Though screenshots are displayed more prominently in the Apple App Store and therefore have a greater effect on conversion rates, your app Screenshots shouldn’t be overlooked in Google.
Choose up to eight screenshots that display prominent app features. Then test your app Screenshots to ensure you are enticing the right kind of users to download your app. We also suggest experimenting with text captions on images to better convey your marketing messages as well with orientation.
You can expect a CVR uplift of up to 13% when optimizing for the right screenshots in Google Play. This piece will help you know better how to orchestrate your screenshot methodology.
A/B Test app videos in Google Play
Videos are an important component of Google Play app listings as well, though it should be noted that videos on Google’s platform don’t autoplay in search results like they do on iOS devices. Still, your video’s thumbnail image will be visible to searchers. That being said, Videos do auto plays within the Store Listing page on most Android devices.
Additionally, the Google Play store only allows app developers to include one video per app and video content is sourced directly from YouTube.
While there are many differences between video on each of the major two app stores, the reason for including them on your app’s product page is the same: to convey a compelling story to users that’ll drive them to install your app.
To achieve this, make sure your video showcases top value propositions and the most exciting aspects of your appin an appealing and engaging way. Then optimize your videos via in-depth tests.
our data tells us that CVR rates can be increased by 20%-30% when videos are done right. Have a look into our comprehensive video guide to know more.
A/B Test Your App Title in Google Play
Your app’s title has a major effect on your app search ranking and conversion rate potential. Your Google Play store app title should include your highest value keywords. Optimizing your app title will help drive organic traffic to your app.
Keep in mind, the best app store titles are short, clear, and easy to remember. Google Play allows app titles of up to 50 characters and we suggest using the character limit Google Play Store provides to the fullest inorder to maximize your app ranking potential.
Like every other aspect of your app store page, your app’s title needs to be tested to ensure it’s driving the maximum volume quality installs.
You can increase CVR by up to 8% when you have the right text elements optimization strategy in place. Search this article to know more.
A/B Test App Short Description in Google Play
App developers have two description areas to worry about in the Google Play store: the Short Description and the Long Description. The short description is always visible and doesn’t require the user to do anything to view it, while the Long Description requires the user to tap to read more before it’s visible.
The Short Description is an 80-character field that’s automatically visible to users once they arrive on your app’s product page. The Short Description can be expanded into the Long Description, which gives app developers 4,000-characters to describe their solution to users.
When optimized, both the short and long app description areas will help you drive organic traffic to your app and increase conversion rates.
It’s important to note, both the Apple App Store and Google Play Long Description are only read by less than 2% of users which means it’s not the most impactful element to focus on.
You can increase CVR by up to 8% with the right text elements in place. Click here to know more.
How Often Should You Run App Store Tests?
There’s no hard and fast rule for app testing, so the answer is “it depends.”
Here are a few guidelines to follow:
- Quality Over Quantity:
It’s better to run a few good app tests than it is to run many mediocre tests. Why? Because mediocre tests have the potential to give you inaccurate and/or irrelevant results that won’t help you achieve your goals. Instead, run only the right tests on your app as often as you need to. - Avoid Frivolous Testing:
Similarly, it won’t do you any good to run frivolous tests. A frivolous test is any test that’s run “just because,” rather than for a specific purpose, or in other words, to test a specific hypothesis. Commit to only running tests that will help you achieve, validate or disprove something specific on your app store page.
The key to proper app store testing is regularity and quality.
Running app store tests regularity: You want to test your app page often enough to ensure your app is always performing well, and that you avoid the widely known phenomena of conversion rate decay (a decrease in CVRs driven by using the same app store creatives for too long)Ensure your app tests are quality tests: Make sure that every test you run is giving you quality information that will help you either achieve or maintain your goals.
A Quick Word About Statistics and A/B Testing
App store testing is necessary, but it’s easy to get it wrong. In most cases, A/B tests fail because the test planner doesn’t take into account a few important factors.
First, it’s important to design tests with clearly defined statistical models and stopping rules. In other words, you need to begin your tests with assumptions, goals, and end points. That way your results are accurate and can be acted on in a timely manner.
And second, it’s vital that you sample the right kind of traffic.
Imagine testing a new icon in the Google Play Store. Your designers work to create a compelling image and you spend money driving traffic to your test page. Then, based on the data you acquire, you finalize your icon, thinking it’s perfectly optimized.
But if you tested your icon on traffic that doesn’t match your ideal user, your experiment was flawed from the beginning and the results you achieved will likely harm your organic traffic and conversion metrics rather than help them. Learn more about Storemaven’s statistical methodology.
In Conclusion:
App store a/b testing is vital to the success of your app in both the Apple App and Google Play stores. More than just focusing on absolute conversion rates figures, nailing App Store Testing will have ripple effects that’ll amplify most of your marketing effort’s ability to drive installs.
If you want to leverage the platform that 80% of top apps use to boost their mobile growth in the app stores, let’s talk. Storemaven streamlines the app store testing process which enables you to see which aspects of your app product pages need to be optimized.
