Updated: December 6, 2018
App Store Testing plays a significant role in your App Store Optimization (ASO) strategy. Done right, it can double your conversion rates and cut the cost of each new install.
At StoreMaven, our goal is to make App Store Testing accurate, quick and more affordable. Since testing multiple store pages involves sending dedicated traffic to your tests, it’s crucial to minimize the required sample as much as possible without harming the accuracy of the results.
In early 2014, we started experimenting with implementations of machine-learning predictive algorithms in App Store Testing. The result of that experiment yielded our proprietary app store predictive testing algorithm – StoreIQ™.
StoreIQ™ is based on a well-known algorithm called Multi-Armed Bandit and is designed to save you between 30-50% of the cost of each test by concluding tests faster and with fewer samples.
The good news is that all StoreMaven tests utilize StoreIQ™, constantly saving you precious budget and time.
Predictive testing allows StoreMaven to quickly identify app store variations that are under-performing and have very low chances of winning. Once a variation hits a certain threshold, it will be closed and will not receive additional traffic. This is where the money-saving kicks in – with StoreIQ™, traffic is automatically redirected from variations that have slim chances of winning.
Once you start sending test traffic to different app store pages, StoreIQ™ will assign each of your test variations “Winning Probabilities”. Winning Probability is the probability of a variation being the top performing variation in your live store.
In traditional A/B testing, you are required to send an equal amount of traffic to all test variations until the end of the test. This means that for each test, your total sample size is much higher and, as a result, so is your test cost.
On average, StoreIQ™ will cut the required sample size for any given test by 40%. Simply put, if a specific 4-variation A/B test requires a total of 7500 clicks, the equivalent test using StoreIQ™ will require only 4000 clicks, saving you thousands of dollars per test. The more variations you test, the more you save.
In many cases, you will want to get test results as fast as possible. Concluding tests with significant results at a faster pace will help you meet your version release deadline, as well as shorten your design cycles. This will allow you to respond more quickly to how users engage with your app store creatives (icons, screenshots and videos).
StoreIQ™ is able to conclude tests significantly faster than traditional AB testing as it takes into consideration multiple factors when analyzing the results beyond the conversion metric. These other engagement actions on the App Store are ignored in traditional AB tests. In most cases, tests on StoreMaven will run for between 7-14 days.
Multi-Armed Bandit experiments (which StoreIQ™ is based on) are optimal for multivariate testing environments. In such experiments, StoreIQ™ will quickly identify combinations that perform better and close all weak variations. While you can save a huge amount in MVT testing since variations are closed down early, you will also benefit by using StoreIQ™ for two-variation tests. This is because they are likely to conclude faster and will take other actions that users completed on the page into consideration.
Once a given variation reaches a certain negative threshold, it will be closed and will not receive any additional traffic during the test. All of the data collected for this variation will still be available on the StoreMaven dashboard.
The threshold we use to determine if a variation can be closed is dynamic and changes per test. It includes the variation performance in comparison to the other variations, the length of the test, traffic source (i.e. Facebook / Google), the “Potential Value Remaining” and dozens of additional factors. These factors are based on tens of millions of unique users who have come through our experiments.
The ‘Potential Value Remaining’ in an experiment is determined not only by the chance a variation has of winning but also by the CVR amount by which the variation would win. For example, let’s say your winning variation has a conversion rate of 4%. You also have a second variation that has a 2% chance of beating the winner – however, if it does beat the winner, it would have a conversion rate of 10%. We would keep that variation open since the potential gain is so high, even if the chances are slim.
We will only close a variation when there is at least a 95% probability that the value remaining in the experiment is less than 1%. In the above example, the experiment would only be closed if we are 95% sure that no other variation would have more than a 4.04% CVR.
We highly recommend turning Autopilot ON from your StoreMaven Test Lab dashboard. Once turned ON, StoreIQ™ will automatically close low-performing variations in real time and make sure you are optimizing for savings and quicker results.