Why Creative Sets are Great for Optimizing ROAS but Risky for Product Page Optimization

Creative Sets are not actually an a/b testing solution for App Store creatives. Instead, they are Apple’s attempt to allow Search Ads users to dynamically serve different creatives in different ad groups.

What are Creative Sets?

When Apple introduced its Search Ads product, allowing marketers to directly advertise on the App Store search results page, it also left a wide gap in its product. 

Compared to other advertising products, such as Google AdWords and Facebook mobile install campaigns, Apple didn’t allow users to test their ad creatives. Knowing that user acquisition budgets usually pour towards ads with the highest return on ad spend (ROAS), Apple decided to allow marketers and user acquisition teams the option to test their Search Ads creatives. 

They called the testing solution “Creative Sets.” 

Creative Sets are collections of app screenshots and previews that you upload to App Store Connect. After setting up your app and assets on App Store Connect, you can create and link Creative Sets to ad groups through the Application Programming Interface (API) or through the Apple Search Ads User Interface (UI).

Creative Sets enable you to test ad variations in groups and optimize for different keywords, devices, and display sizes. 

These sets can be applied to change the screenshots used for a specific keyword (or keyword group). For example when a food delivery app is targeting the term “burger,” it can use screenshots of scrumptious burgers, and when targeting “sandwiches” it can use images of mouth-watering sandwiches.

Why Creative Sets are Great for Optimizing ROAS but Risky for Product Page Optimization - 1

What Apple Search Ads Creative Sets Are and What They Are Not

Creative Sets are not actually an a/b testing solution for App Store creatives. Instead, they are Apple’s attempt to allow Search Ads users to dynamically serve different creatives in different ad groups. It also optimizes the tap-through and install rate by matching assets and messaging to users based on keywords. 

Similar to how Facebook, Google, and other mobile ad platforms provide advertisers with testing tools to optimize their ad creatives, Apple created a great tool for optimizing ROAS. 

As with any tool, it was designed to do a very specific job (optimizing ROAS for Search Ads), and utilizing it to perform other jobs can be tricky at best and dangerous for App Store Key Performance Indicators at worst. 

The Risk of Testing Creative Assets on Creative Sets and Implementing Them on the App Store

One of the most common mistakes we see on the App Store is marketers utilizing Creative Sets to test creatives, using these creatives to update their page, and then expecting an uplift in conversion rates for browse, search, and referral (paid) traffic. 

In many cases, doing so will lead to an increase in performance for Search Ads but a decrease in performance for other channels, including organic search. Let’s examine why this is happening. 

Reason 1: The User Discovery Journey is Different

Users in the App Store discover apps through three main journeys. Let’s analyze the mindset users have throughout each journey and the creatives they are exposed to in different stages of that journey. 

App Store Browse Journey

  1. The journey

Users go into the App Store and find apps that are featured either on the Today tab or the Games/Apps tabs. Users also browse the Top Charts, either for general apps/games or overall categories. 

  1. The mindset 

Users in this journey have a very low intent mindset. They aren’t looking for anything specific, but are browsing the store opportunistically for great apps and games to download. 

  1. The creatives

In most instances, including the Top Charts, users are exposed to the app’s icon, title, and subtitle. In some situations, they are also exposed to the product page screenshots.

Why Creative Sets are Great for Optimizing ROAS but Risky for Product Page Optimization - 2

Referral Journey

  1. The journey

Users land directly on a product page (either from a paid ad or any other type of link, for example, from a brand’s mobile website). 

  1. The mindset 

Users in this group have a very high intent to download the specific app/game they were exposed to. An ad convinced them to tap on it and explore further. 

  1. The creatives

Referral users are exposed to the ad creative (usually on other platforms, such as Facebook, Google, or a different app/game through in-app advertising). They have then been exposed to the App Store product page where most users will never go past the “First Impression” or the above-the-fold area of the product page (which includes the icon, title, subtitle, first one or two screenshots, and (if it exists) an autoplaying video). 

Why Creative Sets are Great for Optimizing ROAS but Risky for Product Page Optimization - 3

Search Journey

  1. The journey

Users use the App Store search tab for either brand-related terms (such as “McDonald’s”) or intent-related terms (such as “burgers”). In most cases, these users will also be presented with an ad. Some users will tap on the ad: this group is categorized as paid search. Some users will tap on an organic search listing and be categorized as organic search.

  1. The mindset 

Users in this group have a mid-high intent mindset. They are either looking for a specific app they heard about from a different source (word-of-mouth, seeing an ad, reading content about the app, etc.) or are looking in general terms, as they know what type of app/game they’re looking for but don’t know which one specifically. 

  1. The creatives

Search users are exposed to the app icon, title, subtitle, and the first one to three screenshots from the product page gallery (that may include an autoplaying video). If they choose to install directly from the search results, they won’t be exposed to further creatives; but, if they choose to tap through to the full product page, they’ll be exposed to the entire product page before they make the decision whether to install.

Why Creative Sets are Great for Optimizing ROAS but Risky for Product Page Optimization - 4

So, using Creative Sets, you’ll be optimizing for a particular group of userspaid search traffic. You won’t be optimizing your creatives for the other groups of users, such as browse, referral, and organic search. It’s merely a different journey, and the creatives you’re optimizing (the screenshots that appear within a search ad) would be the right ones to attract that paid search audience. As the mindset of these users is different (shopping for apps/games vs. looking for a variety of options), the creatives that perform better will, by definition, be nonoptimal to the other groups of users. 

For these groups, you need to use other messaging to fit their mindset and motivations, or at least messaging that doesn’t deter them from installing. 

Lastly, in most cases for non-paid search traffic, creatives should work together with the screenshots to tell one cohesive story. By updating the winning screenshots from the Creative Set test, you could actually hurt the performance of these other traffic sources by creating a detachment between the screenshots and the other elements that work together. 

For example, it could be that for browse traffic, one of the most important dynamics of the page is that users see a character in the icon, get intrigued, and explore the screenshot gallery to find more assets relating to that character which excite them and drive them to install. 

Let’s assume that a Creative Sets test showed that a certain screenshot set has won, and you updated your product page with it. These winning screenshots lack that character (because it was less important for paid search traffic), so the conversion rate for browse traffic (that could represent 30-40% of all traffic) drops.

Reason 2: The Test Audience and Method Are Skewed 

Another factor to consider is that Apple won’t randomly serve the Creative Set to 50% of the traffic at any given time, and the percent of the traffic that each creative gets will change over time. This is because the solution is designed to be more of a dynamic optimization and personalization tool than an a/b testing tool. 

When traffic isn’t being randomly exposed to each variant of the Creative Set test at any given time, results can be dramatically skewed as they don’t follow the principle of using a randomized sample. 

This will render results invalid in most cases, and once the results are implemented, you’ll experience a very different KPI profile (conversion rates, tap through rates) as the winning variant will be shown to 100% of the paid search traffic all the time. 

In Conclusion: Why Should You Use Creative Sets? 

So, to recap, Creative Sets are a great way to optimize ROAS for paid Search Ads. But using them to test App Store screenshots and expecting an uplift in conversion rates for the other channels is risky, as it can drive a decrease in performance overall. 

Given that, in order to run an effective Search Ad test, you need to invest enough in the ad to drive a sufficient sample size. Since Search Ads have an extremely high cost per install, we recommend reconsidering. 

Although Search Ads might be a good signal for what messaging and creatives will work well with mid-high intent search users in general, it will most likely create problems elsewhere (in the form of lower conversion rates for referral, organic search, and browse) which could be extremely costly.

Jonathan Fishman
About Jonathan Fishman
Jonathan is Storemaven's Director of Marketing. Before joining Storemaven he spent ten years commanding tanks, working on Wall St., consulting high-growth companies, and exploring Black Rock City. In his spare time, he likes building things from wood, writing, and listening to Frank Zappa.