A/B Testing Statistics


What are A/B Testing Statistics
A/B testing statistics refers to the statistical model used in the conduction of an A/B test (controlled experiment) which in the App stores is the comparison in performance between two variations of an App store page. An A/B test will be used to approve or disprove a hypothesis by testing just a sample of the entire population in the live store, before using the observations collected to predict with a reasonable level of accuracy how the entire population in the live App stores will behave.
Every statistical model has a number of pre-requisites (test parameters) that need to be met in order to conduct a reliable test that shows which iteration performs better.
There are three A/B testing statistical methods that can be used in different ways. The first is the ‘frequentist’ approach which ignores any previous findings or knowledge from similar tests, using only data from the current experiment. ‘Sequential’ testing doesn’t limit sample size so the data is evaluated as it’s collected and testing can be stopped once enough data is collected. The third is the ‘Bayesian’ method of testing, used by third-party A/B testing platforms like Storemaven. This method works on the statistical theory that it’s possible to calculate the conversion rate probability of a variation based on having a model that constantly evolves and is taking into consideration the incoming data throughout the test. This results in a more accurate model that allows you to reach a conclusion with a high degree of certainty.
Why A/B Testing Statistics are Important
Statistics is vital to the process of planning, running and evaluating A/B tests.
Simply put: failing to use the right statistical model when A/B testing would be a waste of time and money. The effective implementation of A/B testing statistics should translate to an increase in installs for the tested audience once the better performing page is applied to the entire population in the live App stores.
A/B Testing Statistics and ASO
ASO teams need to be aware of any statistical model being used in A/B testing to ensure that it’s one that can be trusted to best serve the purpose of the test. It must produce results that can be implemented with confidence in the real App Store or Google Play Store and provide tangible benefits. Using the wrong statistical method that isn’t suited to the metrics being measured could result in having to endure the frustrating process of running tests, implementing results but without gaining any usable results initially hoped for.
Get the ultimate ASO conversion rate optimization eBook
Everything you need to know about icons, screenshots, videos, app reviews & ratings, localization, and seasonality.
Navigate between the letters to explore additional glossary terms:
- a
- b
- c
- d
- e
- f
- g
- h
- i
- j
- k
- l
- m
- n
- o
- p
- q
- r
- s
- t
- u
- v
- w
- x
- y
- z
- User Segmentation
- User Acquisition for Mobile Apps
- Paid App User Acquisition
- Organic User Acquisition
- Explorative Users
- Decisive Users
- App Store Update
- Supply Side Platform
- Software Development Kit (SDK)
- Soft Launch
- SKAdNetwork
- Sessions Per Active Device
- Sales Per Paying User
- Google Play Search
- Apple Search Ads
- App Store Subtitle
- App Store Seasonality
- App Store Search
- App Store Screenshots
- App Store Screenshot View Rate
- App Store Sales
- App Store Analytics: Standard Downloads
- App Store Analytics: Source Type
- App Store Analytics: App Store Sessions
- App Size
- Rewarded Video Ads
- Return on Ad spend (ROAS)
- Replicated App Store Page
- Real-Time Bidding (RTB)
- Re-engagement
- Google Play Third-party Referral
- Google Play Ranking
- App Store Web Referral
- App Store Reviews
- App Store Review Engagement Rate
- App Store Redownloads
- App Store Ratings
- App Store Ranking
- App Store App Referral
- App Store Analytics: Region
- App Retention Rate
- Product Page Conversion Rate
- Proceeds Per Paying Users
- Preloaded Apps
- Pre/Post App Store Tests
- Post-Install Event
- Playable Mobile Ads
- Performance Marketing
- Pay Per Install
- Google Play Product Page
- Google Play Personalization
- Apple’s Product Page Optimization (PPO)
- App Store Product Page Views (Unique Devices)
- App Store Product Page Views
- App Store Product Page
- App Store Proceeds
- App Store Preview Videos
- App Store Pre-Orders
- App Store Pre-Launch
- App Store Personalization
- App Store Paying Users
- App Store Page Scroll Rate
- App Store Analytics: Purchasable Item
- App Store Analytics: Product Page
- App Store Analytics: Pre Orders
- App Store Analytics: Platform Version
- App Store Analytics: Page Type
- Monthly Active Users (MAU)
- Mobile Growth
- Mobile Game Monetization
- Mobile Game Marketing
- Mobile Game Business Model
- Mobile Delivery Platform
- Mobile Bid Adjustment
- Mobile Attribution Window
- Mobile Attribution Manipulation
- Mobile App Retention
- Mobile App Onboarding
- Mobile App Monetization
- Mobile App Advertising
- Mobile Ad Viewability
- Mobile Ad Fraud
- Mobile App Growth
- Average Revenue Per Paying User (ARPPU)
- Average Revenue Per Daily Engaged User (ARPDEU)
- Average Revenue Per Daily Active User (ARPDAU)
- Attribution
- ARPU – Average Revenue Per User
- Apple Arcade
- Apple App Store Connect
- App Store Optimization (ASO)
- App Store Audience Testing
- App Store Analytics: App Version
- App Store Analytics: App Store Active Devices
- App Store Analytics: App Referrer
- App Store Analytics: Active In Last 30 Days
- App Store Acquisition Channels
- App Store A/B testing
- Ad Unit
- Ad Server
- Ad Revenue
- Ad Network
- Ad Mediation
- Ad Impression
- Ad Exchange
- Keyword Localization
- K-Factor
- Google Play Keywords
- App Store Keywords Research
- App Store Keywords
- App Store and Google Play Keywords Optimization
- Interstitial Ads
- Interactive Ads
- Installs Per Mille (IPM)
- In-App Events
- In-App Bidding
- In-App Advertising
- Identifier for Advertisers (IDFA)
- App Store In-App Purchases
- App Store Impressions (Unique Devices)
- App Store Impressions
- App Store Icon
- App Store Analytics: Installations
- App Install Rates
- App Influencer Marketing
- Growth Loop
- Google Play Points
- Google Play Pass
- Google Play Experiments
- Google Play Developer Console
- Google Advertising ID (GAID)
- Gametech (Game Technology)
- Game Bundles
- Google Play Short Description
- Google Play Long Description
- Deep Linking
- Daily Active Users (DAU)
- App Store Description Read Rate
- App Store App Description
- App Store Analytics: Device
- App Store Analytics: App Store Deletions
- App Store Analytics: App Download Date
- Google Play Creatives
- CTR (Click Through Rate)
- Cross Promotion
- Cost Per View (CPV)
- Cost Per Mille (CPM)
- Cost per Install
- Cost Per Engagement (CPE)
- Cost Per Completed View (CPCV)
- Cost Per Click (CPC)
- Cost Per Action (CPA)
- Churn Rate
- Casual Games
- Apple’s Custom Product Pages
- App Store Culturalization
- App Store Creatives
- App Store Conversion Rate
- App Store Categories
- App Store Analytics: Campaign
- App Store Analytics: App Store Crashes
- App Conversion Rate
- App Clip Sessions
- App Clip Installations
- App Clip Crashes
- App Clip Card Views (Unique)
- App Clip Card Views
- App Clip Active Devices
Related Terms
-
Mobile Growth
Mobile growth is the science of growing and gaining ground in reaching and retaining users in the App Store or Google Play.
-
App Store Web Referral
App Store web referral is a traffic source which includes all visitors that arrive at an app listing from an external source (mobile web), most commonly the iOS Safari browser.
Related Resources

Why Save the Girl Broke the App Stores?

Google Play is a Step Closer Towards Disrupting the Mobile Marketing Industry
