How to A/B Test your App Store and Google Play Store Pages

More than just focusing on absolute conversion rates figures, nailing App Store Testing will have ripple effects that’ll amplify most of your marketing endeavours. Learn more with this guide.

You’ve worked hard to develop an app that your audience will love. Unfortunately, creating a world-class product is only half the battle. You also need to effectively promote your app to its target market. You can use influencers, run ads, and use other marketing tricks to make this happen. But high traffic metrics will only benefit your business if they convert into high quality installs.

Enter app store testing.

In this article, we’ll discuss what app store testing is, how you can a/b test your App Store Page and your Google Play store listing page, the elements of your app store page that should be tested on a regular basis, which tests to run, and a whole lot more.

Let’s dive in!

ASO is more than SEO

The Creative Optimization side of ASO, short for App Store Optimization, is the process of adjusting an app’s product page to improve visibility and conversion rates in the Apple App and Google Play stores.

When people first learn about ASO, many assume it’s the same as website SEO and completely reliant on keyword strategy. While keywords do play an important role in ASO, they are far from the only factor. Every aspect of your app’s product page falls under the ASO umbrella, including its icon, screenshots, videos, and copy.

Both the Apple App and Google Play stores have said that an app’s app store product page overall performance, from its keywords to its conversion rate, will impact its ranking in search results.

What is App Store Testing?

App store testing is exactly what it sounds like: the act of A/B testing an app store page in either the Apple App or Google Play stores to improve its ability to convert users, as measured by App Store and Google Play conversion rates.

By testing your app store pages on a regular basis, you can make sure that you’re capitalizing on your marketing efforts that drive impressions to your page, by converting the most impressions to installs, maximizing your conversion rates, and lowering your user acquisition costs. 

Let’s dive deeper into each of these important benefits…

What Problem Does App Store Testing Solve?

If you don’t understand what drives your target market to install your app , or more importantly what pushes them away from your page without installing, you’ll never be able to:

  • Increase both organic and paid conversion rates
  • Lower user acquisition costs, by improving the conversion rates from a paid ad to install. 

Without gaining this understanding not only that you’ll leave growth on the table, you’ll allow your competitors to gain an edge on you. Think about it this way, users searching for apps around a specific topic (let’s say, Travel apps) will be more likely to install your competitor’s apps, and more likely to find it in the store as the higher conversion rates will lead to better ranking and discoverability. 

Moreover, your competitors’ UA teams will be able to acquire the same users for a lower cost as their paid funnel will be significantly more converting, effectively lowering their cost per install (CPI).

Getting App Store testing right you’ll be able to: 

  • Get to Know Your Users: By testing different creatives and product page copy, you’ll learn what your unique audience responds to best. You can then use this information to craft better product pages and marketing content.
  • Increase Organic Conversion Rates: A proper keyword strategy will enable your app to appear in Apple App and Google Play searches. But if you can’t convert that organic traffic into paying customers, what’s the point? App store testing will help you determine what your organic traffic is looking for so that you can give it to them.
  • Increasing Paid Conversion Rates: Similarly, the perfect advertisement and marketing budget are useless if the traffic they drive to your product pages never end up becoming paid users. App store testing gives you the tools you need to learn why paid users don’t convert so that you can adjust your app pages accordingly.

Decreasing User Acquisition Costs: When you understand your users and know how to increase both organic and paid conversion rates, your user acquisition costs will naturally go down. Lower acquisition costs mean greater profitability and more growth for a fixed budget.

How to run App Store & Google Play product page A/B Tests

Five specific testing methodologies:

So far we’ve covered why app store testing is important. Now we’ll cover the kind of A/B tests and other tests that can be run in the app stores.

1. Storemaven App Store & Google Play A/B Testing

The Storemaven testing platform uses a replicated app store environment. A replicated test is one that happens within a simulated environment. The simulated app store environment mirrors the actual one, but allows an app developer to test new app page elements without harming real-world traffic while gaining important on-page behavioral insights.

How does it work? 

How to A/B Test your App Store and Google Play Store Pages - 1

With this type of test, you’ll create a replicated app store environment, and create several variants of your app store page each with different creatives and messaging, run dedicated traffic (a sample that represents the users you want to optimize for) through a paid ad (usually Facebook ads) where each user will randomly land on each variant, and measure how users interact with the different variants. 

During the test, a sophisticated algorithm, StoreIQ, will automatically close low-performing variants as soon as it detects they have a low chance of winning the test. This will radically reduce the amount of traffic needed for the test compared to other testing methods, and thus, the overall test traffic cost. 

When the test concludes you’ll have access to deep insights that’ll uncover why a certain variation won and you’ll be able to both get a great direction for what to improve and test next as well as what to implement in the real app store to enjoy the conversion rate lift.

Other options include:

2. Google Play Store Listing Experiments on Android

As an app developer, you have access to Google Play’s Store Listing Experiments in the Google Play Store, which will allow you to test app icons, images, videos, and descriptions for your app product page.

What’s missing?

In Google Play store listing experiments you won’t be able to get any data around which traffic source and audience drove the increase in conversion, nor why a certain variant won as it doesn’t include any on-page behavior insights which will show what pushed users away or drove them to install.

Why use it?

  1. If you’re just getting started with ASO Creative Optimization, and don’t have enough resources for dedicated traffic for app store tests, this can be a great tool to still get insights on your Google Play page. 
  2. If you’re almost exclusively focused on the Google Play store with your distribution strategy. 
  3. To amplify your iOS App Store testing, validate hypotheses that were already tested through a full testing solution. 

3. Creative Set Tests

App developers also have specific testing options within the iOS Search Ads ecosystem. These are called “Creative Sets” and will allow you to test different sets of screenshots and App Previews for your Apple Search Ads. By testing and optimizing your ads, you can improve their click through rates (CTRs) and direct response ad conversion rates.

What’s missing?

This type of test misses many of the flows users go through when discovering and installing apps. The different variants are only apparent through Search Ads, and not through the full product page. While this is a valid tool to increase Return on Ad Spend (ROAS) for Search Ads, it is unlikely to uncover insights that’ll lead to better conversion rates for organic and paid traffic. 

4. Pre-Post Analysis Tests

A pre-post test is akin to a before-after picture. You simply measure a specific aspect of your app store page before making a creative change, and then measure it again after the change has been implemented, usually 2-weeks before and after the change. 

What’s missing?

As it’s impossible to isolate all the different changes in your marketing efforts and their impact on conversion rates, these types of tests usually lead to the wrong conclusion. For example, the conversion rate might have dropped because of a change in the UA mix, as opposed to a real drop driven by a creative change. 

5. Focus Groups and Survey Tests

Finally, you can attempt to perform app store testing by showing your product to a focus group and evaluating the responses you get from a post-group survey. This kind of test has the promise to get you detailed, real-world feedback.

What’s missing?

In this method, it’s extremely challenging to draw conclusions for your overall audience and traffic. The focus group is not capturing the natural flow of how a user would stumble upon an app and evaluate it on a product page. Users simply don’t make decisions in a way that’s comparable with the way the focus group participants will.

What Does a Good App Store Test Look Like?

Unfortunately, many app developers run app store tests that aren’t conducive to the above goals. To make sure your testing methods are up to par, look for the following in every app store test you run:

  • Access to On-Page Behavior Statistics: Knowing how many users visit your app store pages and the percentage of them that convert is somewhat useful. But this information alone won’t help you improve your conversion rates. Make sure your app store testing efforts include behavior statistics so you know which aspects of your app store pages are turning users away, or driving them towards the install button
  • The Ability to Run Tests Without Harming Real-World Traffic: What if you run a test and your conversion rates go down instead of up? If you run tests on actual users, you could lose a substantial amount of revenue.Imagine 50% of your actual traffic landing on a variant that converts 20% worse than your control variant. That’s why it’s essential to test your app store changes in ways that don’t harm your real-world traffic.
  • Your Results Are Accurate and Trustworthy: Your testing efforts will be wasted if you can’t trust the information you receive. The results you glean from your tests should be reflected in the Apple App and Google Play stores once you implement changes.

If your app store testing methods don’t allow you to access on-page behavior statistics, run tests without harming real-world traffic, or trust the results, you should consider adjusting your approach immediately.

The Current State of App Store Testing

After facilitating thousands of app store tests, we believe we’re in a prime position to speak about the current state of app store testing. We have three observations to share in this regard:

  1. Observation #1: Most app developers quickly lose sight of why they’re running app store tests and oftentimes develop a low-quality test addiction.
  2. Observation #2: In a similar vein, it’s all too easy for app developers to run pseudo app store tests that never contribute to the metrics they’re trying to improve upon.
  3. Observation #3: Lastly, conversion rate optimization in both the Apple App and Google Play stores has become a key growth strategy. Since App Store Testing is essential to proper ASO, mistakes in this regard will inevitably harm an app’s growth.

Many organizations fall into the convention of running many tests for the sake of running tests, many times using methodologies that only satisfy the need for a “stamp” behind certain decisions. It’s not the number of tests that you run, it’s the actual install growth you’re able to drive and show, and achieving real results can only stem from using a methodology that works. 

We encourage you to keep these three observations in mind when thinking about how to test your app store pages! If you don’t, you’ll likely waste time and money on faulty tests that don’t move the needle on the KPIs you’re measured on and trying to improve.

What App Store & Google Play Page Elements to Test

It’s clear that app store testing is vital to the success of your app. But which elements of your app product page should you test? Are some elements more important than others? That’s what we’ll cover in this section.

In the spirit of focusing your efforts where you have the most potential to have an impact, you should always focus on the elements that have a better potential to increase conversion rates. Don’t spend your time on elements that can’t move the needle.

The key here is to understand how users interact with app store pages and you can take a look at this report we’ve made that analyzes more than 500M user sessions on the app store. 

Get the App Store Behavioral Report

Know which assets drive installs in the App Store




Apple App Store A/B Testing

Obviously, the Apple App and Google Play stores operate differently. 

More importantly, the two platforms attract inherently different people (Android vs. iOS) which will 100% have different preferences.

That’s why it’s important to test different elements for each store. Here’s what we recommend app developers test in the iOS ecosystem:

A/B test your App Store Icon

Your app’s icon is usually the first thing a potential user sees in regards to your product. As such, it’s imperative that you make a good first impression and create an icon that conveys the value your app brings to the table.

Test different icon variations to see which one is preferred by your target audience. Just be careful. You don’t want to change your icon so much that company branding is affected.

You can expect a CVR uplift of up to 18% when optimizing for the right app store icon. Read more about the icon’s role in your ASO strategy.

A/B test Your App Store Screenshots

In the Apple App Store product page as well as in search results, users will be able to view one to three of your app’s screenshots in the first impression, depending on their orientation (portrait or landscape.) This fact makes the screenshots you choose (Apple allows a total of 10 screenshots) incredibly important.

You’ll want to choose images that clearly show how your app is different and unique and any exciting value propositions it has. Test multiple screenshots to ensure the ones you use catch the attention of your ideal users.

You can expect a CVR uplift of up to 28% when optimizing for the right app store screenshots. This piece will help you know better how to orchestrate your screenshot methodology.

A/B test Your App Store Videos

Once a user visits your App Store product page, they’ll be able to view any videos you’ve created for it. These videos are known as “App Previews” in the Apple App Store and you can include a maximum of three 30 second clips. It’s important to note that App Previews play autoplay on mute once a user views your page, meaning the first few seconds of your videos need to be visually compelling if you hope to grab your audience’ attention. 

We suggest using App Preview footage to demonstrate key value propositions, show your app’s most exciting usability, etc., though you’ll want to test your videos to see which ones best connect with users.

our data tells us that CVR rates can be increased by 20%-30% when videos are done right. Have a look into our comprehensive video guide to know more.

A/B test Your App Store Titles

Your app’s name is absolutely crucial to its discoverability in the Apple App Store. You’ll want to choose a name that’s simple, memorable, leverages your brand capital, and at least hints at the function it performs. 

Additionally, you’ll want to include your highest value keywords as Apple gives the most weight to keywords located in the title section. Test various app names and keyword combinations to find one that drives both a high volume of organic traffic and boosts conversion rates.

App developers also have access to a Subtitle section in the iOS ecosystem. This area will allow you to further explain the value of your app and include additional keywords. 

As the title and subtitle appear in the first impression they do have the potential to increase/harm conversion rates.

You can increase CVR by up to 8% when you have the right text elements optimization strategy in place. Search this article to know more.

A/B test Your App Store Descriptions

If a user is enticed by your app’s icon, screenshots, and App Previews, they may decide to read its description as well. This is where you should highlight your app’s key capabilities and functionality. Apple recommends starting with a brief, but engaging paragraph followed by a bullet list of top features. As less than 1%-2% of users will ever read the full description this is not the lowest hanging fruit in your journey to increase conversion rates.

You can increase CVR by up to 8% with the right text elements in place. Click here to know more.

How to A/B Test your App Store and Google Play Store Pages - 2

Google Play Store A/B Testing

Now that we’ve thoroughly covered the Apple App Store page elements worth testing and optimizing, we’ll do the same for the Google Play Store. 

Let’s dive in!

A/B Test Your Google Play Icon

Just like in the Apple App Store, your app’s icon is probably the first encounter users will have with your app in the Google Play Store. Making a good first impression is very important, which means creating an icon that conveys the value your app brings to the table should be a top priority — at least if you want to increase traffic and conversions.

Keeping brand best practices in mind, test multiple icon variations to see which one is most effective in driving installs.

You can expect a CVR uplift of up to 11% when optimizing for the right icon in the Google Play Store. Read more about the icon’s role in your ASO strategy.

A/B Test Your Google Play Screenshots

As you’ve probably noticed by now, the visuals you include in your app store listings are incredibly important. So it should come as no surprise to learn that app screenshots in the Google Play Store should be tested and optimized on a regular basis.

Though screenshots are displayed more prominently in the Apple App Store and therefore have a greater effect on conversion rates, they shouldn’t be overlooked in Google.

Choose up to eight screenshots that display prominent app features. Then test them to ensure the images you use are enticing the right kind of users to download your app. We also suggest experimenting with text captions on images to better convey your marketing messages as well with orientation.

You can expect a CVR uplift of up to 13% when optimizing for the right screenshots in Google Play. This piece will help you know better how to orchestrate your screenshot methodology.

A/B Test Your Google Play Videos

Videos are an important component of Google Play app listings as well, though it should be noted that videos on Google’s platform don’t autoplay in search results like they do on iOS devices. Still, your video’s thumbnail image will be visible to searchers. That being said, Videos do auto plays within the Store Listing page on most Android devices

Additionally, the Google Play store only allows app developers to include one video per app and video content is sourced directly from YouTube.  

While there are many differences between video on each of the major two app stores, the reason for including them on your app’s product page is the same: to convey a compelling story to users that’ll drive them to install your app. 

To achieve this, make sure your video showcases top value propositions and the most exciting aspects of your appin an appealing and engaging way. Then optimize your videos via in-depth tests.

our data tells us that CVR rates can be increased by 20%-30% when videos are done right. Have a look into our comprehensive video guide to know more.

A/B Test Your Google Play Title

Finally, we have your app’s title, which will have a major effect on its search ranking and conversion rate potential. In the Google Play store, it’s important to include your highest value keywords in the title. This will help drive organic traffic to your app.

Keep in mind, the best app store titles are short, clear, and easy to remember. Google Play allows app titles of up to 50 characters and we suggest using most of them so that you can use as many keywords as possible.

Like every other aspect of your app store page, your app’s title needs to be tested to ensure it’s driving the maximum volume quality installs.

You can increase CVR by up to 8% when you have the right text elements optimization strategy in place. Search this article to know more.

A/B Test Your Google Play Short Description

App developers have two description areas to worry about in the Google Play store: the Short Description and the Long Description. The short description is always visible and doesn’t require the user to do anything to view it, while the Long Description requires the user to tap to read more before its visible.

The Short Description is an 80-character field that’s automatically visible to users once they arrive on your app’s product page. The Short Description can be expanded into the Long Description, which gives app developers 4,000-characters to describe their solution to users.

When optimized, both description areas will help you drive organic traffic to your app and increase conversion rates. Similar to the App Store, the Google Play Long Description is only read by less than 2% of users which means it’s not the most impactful element to focus on.

You can increase CVR by up to 8% with the right text elements in place. Click here to know more.

How to A/B Test your App Store and Google Play Store Pages - 3

How Often Should You Run App Store Tests?

The answer to that question is the ever infuriating “it depends.” There’s no hard and fast rule for testing, but there are a few guidelines:

  • Quality Over Quantity: It’s better to run a few good tests than it is to run many mediocre tests. Why? Because mediocre tests have the potential to give you inaccurate and/or irrelevant results that won’t help you achieve your goals. Instead, run only the right tests as often as you need to.
  • Avoid Frivolous Testing: Similarly, it won’t do you any good to run frivolous tests. A frivolous test is any test that’s run “just because,” rather than for a specific purpose, or in other words, to test a specific hypothesis. Commit to only running tests that will help you achieve, validate or disprove something specific.

The key to proper app store testing is regularity and quality. You want to test often enough to ensure your app is always performing well, and that you avoid the wildly known phenomena of conversion rate decay (a decrease in CVRs driven from using the same app store creatives for too long) but you want to make sure that every test you run is giving you quality information that will help you either achieve or maintain your goals.

A Quick Word About Statistics and A/B Testing

App store testing is necessary, but it’s easy to get it wrong. In most cases, A/B tests fail because the test planner doesn’t take into account a few important factors.

First, it’s important to design tests with clearly defined statistical models and stopping rules. In other words, you need to begin your tests with assumptions, goals, and end points. That way your results are accurate and can be acted on in a timely manner.

And second, it’s vital that you sample the right kind of traffic.

Imagine testing a new icon in the Google Play Store. Your designers work to create a compelling image and you spend money driving traffic to your test page. Then, based on the data you acquire, you finalize your icon, thinking it’s perfectly optimized.

But if you tested your icon on traffic that doesn’t match your ideal user, your experiment was flawed from the beginning and the results you achieved will likely harm your organic traffic and conversion metrics rather than help them. Learn more about Storemaven’s statistical methodology here.

In Conclusion:

App store testing is vital to the success of your app in both the Apple App and Google Play stores. More than just focusing on absolute conversion rates figures, nailing App Store Testing will have ripple effects that’ll amplify most of your marketing effort’s ability to drive installs. 

If you want to leverage the platform that 80% of top apps use to boost their mobile growth in the app stores, let’s talk. Our platform streamlines the app store testing process and will allow you to easily see which aspects of your product pages need to be optimized, to truly allow you to run an ongoing optimization routine.

Request a Storemaven Demo

Please fill in your details




Jonathan Fishman
About Jonathan Fishman
Jonathan is Storemaven's Director of Marketing. Before joining Storemaven he spent ten years commanding tanks, working on Wall St., consulting high-growth companies, and exploring Black Rock City. In his spare time, he likes building things from wood, writing, and listening to Frank Zappa.