Developing a new mobile product is no simple feat. You have to deal with a highly saturated (and ever-growing) mobile app ecosystem in which millions of apps and games are already available in both the Apple App Store and Google Play Store. Plus, the process of developing an app or game involves a multitude of decisions and unknowns. So how do companies gain a competitive advantage and identify which ideas will (or won’t) be well-received by their market before they invest valuable time and resources into development? It’s the first stage in the pre-launch process, and it’s called concept testing.
Many people assume that app store creative testing is only applicable to apps or games that are already being developed or are currently listed in the App Store or Google Play. However, there’s a huge opportunity to utilize app store optimization (ASO) prior to development in order to solidify an idea with the highest chance of success.
One of the things companies get wrong from the beginning, though, is not fully understanding their audiences. The sooner you use testing to uncover insights about your potential audience’s unique preferences and interests, the sooner you can move forward with a mobile app or game concept that is likely to be a lucrative opportunity.
Even though concept testing is more frequently conducted by game developers, it’s a universal activity that is relevant to every mobile developer who wants to launch a new product and gain a competitive advantage. Using our experience helping leading mobile publishers run concept tests, this post will cover why it should be an essential part of your mobile app or game development process and how you can leverage it effectively.
Why Mobile App Concept Testing is Important
On the surface, the benefit of concept testing is that it helps you discover a potential product opportunity in the highly saturated mobile app marketplace.
Companies are already investing resources in this area and using alternative research methods, such as focus groups, market research, and surveys, to formulate hypotheses on the best direction to take a new product. Many even have dedicated consumer insights and research teams supporting and driving these initiatives. However, these methods aren’t accurate enough. They fail to replicate the environment in which real potential users discover your game or app and then decide whether or not to install. App store testing offers a unique opportunity to uncover unbiased and significantly more detailed data to support these hypotheses or even reveal an alternative direction.
Most importantly, it helps you answer these essential business questions:
- Is it worth adding a new franchise or new app?
- What is the best combination of IP and game mechanic? Best combination of app features with app concept/theme?
- What app concept or game will have the largest reach?
- What is the new app or game’s potential ROI?
- How do different demographics and interest groups respond to each concept?
- What do users expect to see after they decide to download (but before they open the app/game)?
- How does your existing network respond to this concept?
If we bring it into the real world, many major developers own multiple apps or games in the same theme or franchise. A major question that these developers have is whether they should come up with an entirely novel concept and expand to a new category, or stick with what’s proven successful and use their current franchise(s) or brands.
For example, Supercell currently has two games in their successful “Clash” franchise—Clash of Clans and Clash Royale. Their situation would be the ideal use case for concept testing, as it would help them determine if they should add a third title to this franchise or develop an entirely unrelated game that would enable them to reach a new market.
As you can see, the business questions you choose to answer lay the foundation for your entire concept test strategy.
Concept Testing Process
Concept testing in the app stores may appear straightforward—you just solidify different app/game concepts, create mock app store pages for each, and then send traffic to see which concept drives the most installs, right? It’s actually not so simple. In order to extract the most value from this activity, you should follow a strategic process and fully understand how to accurately measure and use the results to your advantage.
Determine the Factors to Test
So you’ve identified the business questions you want to answer. Now you can start narrowing down the options for what type of app or game to develop.
First, you should define your different audiences. You can segment them by factors such as:
- If they’re existing users of an app/game you already have vs. a broader audience
- Age (e.g.,18-24, 25-35, 35-44, etcetera)
- Interests (e.g., Game of Thrones, casual games, etcetera)
- Gender
- Location
At this stage, you can begin to identify the target audience and create a user persona. A major benefit of defining your audiences early on is that it enables you to understand if it’s an audience worth targeting. For example, if you were deciding whether you should build a tower defense game using an established IP (e.g., Game of Thrones), you’d first want to look at the reach on Facebook (i.e., how big the audience is for people interested in those two genres).
Continuing with the IP example, there would be two audiences you should look at: 1) the IP (in this case, Game of Thrones) audience and 2) the broader tower defense genre-loving audience. The goal is to determine how attractive this concept would be among people who are fans of multiple genres. If these two audiences complement each other and have a wide reach, then you can be more certain it’s a good route and can begin concept testing. If the overall reach is low, then you might want to reconsider this concept.
As we’ll discuss in the analysis section, the goal is to find an idea that will appeal to the largest audience with the lowest cost per install (CPI). The metrics used to determine this is ad reach, overall conversion rate (CVR), and (of course) CPI.
Once you segment your audience, you should then identify the specific concepts (e.g., different game mechanics, app functionality, storylines, themes, etcetera) you want to test. These concepts should be distinct enough so that when a “winner” is found, it cannot be confused with another concept.
Here’s an example of an initial concept test for a new game. The defined audience segments are directed to variations that showcase different game mechanics (in this example, match 3, interactive story, tower defense, or role-playing game (RPG) strategy). This first test is crucial to determine the type of game that will appeal to the widest potential users, and the results will guide the rest of the concept testing process.
Analyze the Results
Once the test ends, you can determine which concept drove the most interest. Research groups can reveal qualitative user preferences on a smaller scale, but testing reveals both quantitative data (e.g., potential CPI and CVR) and qualitative data (e.g., which exact messages in the Gallery drove people to install or drop) from thousands of highly targeted and quality users.
If we continue with our example, you can see that the tower defense game mechanic won because it had the highest CVR and lowest CPI. Based on the results of this first test, you can more accurately verify that resources will be invested in the correct place and that the game will be developed to match the widest possible audience preferences.
To further enhance your learnings, we recommend adding a survey that users have the chance to complete after clicking “install” on the test app store page. The questions can range from specific (e.g., which character do you like best, which features interest you) to more broad, behavioral questions (e.g., have you spent money in a game before, how often do you play games). These survey questions are invaluable to:
- Determine the quality of the traffic segment
- Find most beloved (or hated) characters
- Identify aspects or features that could drive the most monetization and return on investment (ROI)
- Pinpoint the most preferred (or disliked) features
Based on concept tests we’ve run with leading developers, we’ve seen completion rates of up to 70%.
Finalize Concepts in Subsequent Tests
Using the winner of the first test, each subsequent test will help you determine a different aspect or feature of the app/game. The hypotheses become more specific, and the audience is more representative of your real audience.
For example, the hypothesis driving the second test above is which theme will appeal to the most users: Aquatic, Futuristic, Middle Ages, or Tropical?
The rest of the concept testing process helps you hone in on the granular specifics of the app or game. For apps, it could be which features or branding style piques the most interest. For games, it could be the characters, game setting, or storylines that appeal to the most users.
Each test also uncovers the unique app store engagement data. Among different target audiences, how do their preferences vary? How many visitors scrolled through my Gallery? Which Screenshots had the most impact? How many visitors dropped after seeing the First Impression (everything above the fold)?
But, why are these questions important? They represent another crucial step in the product development process in which you’re trying to understand which messages and creatives drive more people to install and how best to convey the unique selling points of your game/app. This enables you to identify the most impactful marketing messages to highlight in your broader mobile marketing and UA campaigns once development is complete.
Overall, concept testing offers you a data-backed way to better hedge your bets and find a product opportunity with a high chance of success. As you can see above, the tower defense genre is already saturated with different themes. Without concept testing, we would’ve spent valuable time and resources developing yet another tower defense game without fully understanding if it would appeal to a wide audience or even which theme to use in order to stand out among the current competitors. Through the testing process, we identified an opportunity to develop an aquatic-themed tower defense game that found a niche, profitable place in the mobile game ecosystem.
What Comes After Concept Testing?
It’s no secret that the app store platforms are becoming more competitive every day. The best way to gain an advantage in this saturated space is to more accurately identify which concepts will appeal to the widest audience. Testing helps you make these important product development decisions using concrete data, and it plays a critical role in revealing how people make their own decisions and which messages convince them to install.
Despite these benefits, most developers think of the app stores as the final stage in their launch plan. By bringing it back to the beginning of the process with concept testing, it gives them a unique glimpse into how potential audiences will react, if there’s an opportunity for a new app/game, and if it even makes business sense to invest resources into the idea.
After concept testing, the next step is to soft launch in smaller markets that aren’t your main GEO. This enables you to further optimize your marketing messages and finalize specific features before fully launching in your main market. Keep an eye out for our follow-up post that will cover best practices for soft-launch testing.
StoreMaven is the leading ASO platform that helps global mobile publishers like Google, Zynga, and Uber test their App Store and Google Play creative assets and understand visitor behavior. If you’re interested in concept testing or would like to optimize your current app store creatives, schedule a demo with us.
