Firebase A/B Testing — what it is, experiment types and how to configure

What is A/B Testing in the context of mobile applications

A/B testing (split testing) is a comparative analysis method in which two groups of users (control and experimental) see different versions of the same app element, after which the impact of each version on the selected metric is measured. In mobile development, A/B tests are used to test hypotheses about UI changes, onboarding, monetization mechanics, push notifications and recommendation algorithms.

The key difference between A/B testing and simple observation is causality. If after changing the checkout screen the conversion rate increased by 15%, an A/B test proves that this particular change caused the growth, not an external factor (holiday, advertising campaign, seasonality). Without an A/B test, you cannot claim a causal relationship — only correlation. According to Optimizely (2025), companies that regularly conduct A/B tests increase conversion by an average of 30% per year.

To conduct a quality A/B test, four components are needed: a hypothesis (what we change and why), a metric (how we measure the effect), a sample size (how many users are needed for a reliable result) and a duration (how long to collect data). Firebase A/B Testing covers all four components automatically, but understanding each of them is necessary for correct interpretation of the results.

Why A/B tests are important for mobile applications

Mobile applications have specific features that make A/B testing especially valuable. First, high competition: there are over 3 million apps on Google Play, and every UI decision affects retention and conversion. Second, a long release cycle: publishing a change through the app store can take 1 to 7 days for review. An A/B test allows you to test a hypothesis without a release (via Remote Config) and apply the change only when effectiveness is confirmed.

Audience segmentation is another advantage of A/B tests. A change that works for new users may be harmful for existing ones. Firebase A/B Testing allows you to segment audiences by app version, country, language, registration date and user properties. This makes it possible to test changes on a specific subgroup before global rollout.

Difference between A/B test and feature flag (Remote Config)

Feature flag is a simple enabling or disabling of a feature for all users or a percentage of them. An A/B test is a structured experiment with metric measurement and statistical significance calculation. A feature flag does not answer the question “did the change affect the metrics?” — it only manages feature availability. In Firebase A/B Testing, Remote Config is used as a value delivery mechanism, but adds a layer of analytics and statistics.

In practice: if you just want to gradually roll out a new feature to 20% of users and make sure it doesn’t crash — use Remote Config with a random_percent condition. If you want to prove that a new feature increased the conversion rate by 10% — use Firebase A/B Testing, which will automatically measure the metrics and show the p-value.

How Firebase A/B Testing Works

Firebase A/B Testing is an overlay on top of Remote Config and Cloud Messaging, providing a unified interface for creating and monitoring experiments. Architecturally, the service consists of three components: the management console (A/B Testing section in Firebase Console), the distribution engine (assigns users to groups based on a given percentage) and the statistical engine (analyzes the difference in metrics between groups).

When the experiment creator publishes changes, Firebase saves a new version of the Remote Config template but applies different parameter values for different user groups. The client app, after executing fetchAndActivate, receives the value corresponding to its group. Firebase Analytics collects events from all groups and passes them to the statistical engine, which daily updates the report with p-value and confidence intervals.

Statistical model Firebase A/B Testing uses the Frequentist approach with a t-test for comparing mean metric values. For binary metrics (conversion, retention) — a two-sample z-test of proportions. The default significance level (alpha) is 0.05. Firebase adjusts for multiple comparisons using Bonferroni correction if several primary metrics are selected. Important: statistical significance does not guarantee practical significance — even with p-value < 0.05, the absolute improvement may be economically unfeasible.

Distribution of users into groups

Firebase A/B Testing uses deterministic distribution based on the user identifier (Analytics App Instance ID). This means that the same user always falls into the same group on repeated experiment launches, provided the experiment configuration hasn’t changed. Determinism is important for consistency of the user experience: a user should not see different interface versions every time they launch the app.

Percentage distribution is set when creating the experiment: for example, 50% control group, 50% experimental group. Firebase distributes users evenly using a random seed, ensuring balanced group sizes. When using multiple experimental groups (A/B/n), the percentage is divided equally among them. Important: the distribution percentage cannot be changed after the experiment starts — to change the percentage, you need to stop the experiment and create a new one.

Integration with Remote Config and Cloud Messaging

Remote Config serves as the source of values for parameters modified in the experiment. When creating an A/B test, you select a Remote Config parameter and set its value for each group. Firebase automatically creates a temporary Remote Config template branch with experimental values. After stopping the experiment in favor of one group, its value can be applied as the production value through the Firebase console.

Cloud Messaging is used to send push notifications that are part of the experiment. Firebase A/B Testing supports creating experiments with different texts, images and timing of push notifications. The service automatically distributes notifications across groups and measures the impact on metrics: open rate, post-click conversion, uninstall rate. This allows you to find optimal communication mechanics with users without manual A/B testing of mailings.

Creating and Configuring an Experiment

Creating an A/B test in Firebase Console is done in the A/B Testing section via the “Create experiment” button. The creation wizard includes several steps: selecting the experiment type (Remote Config or Notification), specifying the parameter and its values for the control and test groups, defining the target audience (by attributes) and selecting metrics for measurement. After completing the setup, the experiment is published and starts collecting data.

Choosing the experiment type: Remote Config experiment — for changing any app parameter (UI, content, logic); Notification experiment — for comparing the effectiveness of different push notifications. Remote Config experiments require a previously created parameter in Remote Config. Notification experiments are created independently — Firebase will automatically prepare and send push notifications for each group without writing client-side code.

Defining the audience is a critically important step. By default, the experiment runs on all app users. To narrow the audience, use filters: app version, country, language, OS version, Analytics user properties. For example, changing onboarding only makes sense to test on new users (first_open within 7 days). Testing on an irrelevant audience gives a “blurred” result, hiding the real effect of the change.

Experiment duration and sample size

Minimum duration of an experiment in Firebase A/B Testing is 3 days (including a full weekend, as user behavior on weekdays and weekends differs). Firebase automatically calculates the recommended duration based on traffic and the specified Minimum Detectable Effect (MDE). The default MDE is 5% relative change in the metric. If current traffic is insufficient to detect a 5% effect within 4 weeks, Firebase will warn about it.

Sample size is calculated based on: baseline metric (current value), MDE, significance level (alpha = 0.05) and statistical power (power = 0.8). For a typical app with 50,000 MAU and a baseline conversion rate of 10%, detecting a 5% relative change will require about 30,000 users in each group (60,000 total). If the sample size is insufficient, the result may not reach statistical significance even if the change was effective (type II error).

Working with multiple variants (A/B/n)

Multi-variant experiments (A/B/n) allow comparing 3 or more versions of one parameter. Firebase supports up to 10 variants in one experiment. The more variants, the more users are needed to achieve statistical significance. Rule: for each additional variant, the sample size increases by 20–30% relative to a two-variant test. If traffic is limited, sequential two-variant tests are preferable to a single multi-variant test.

Bonferroni correction — Firebase automatically applies a correction for multiple comparisons when there are multiple variants or metrics. The gist: if you test 5 hypotheses with alpha = 0.05, the probability of at least one false positive result is 1 — (0.95)^5 ≈ 22.6%. Bonferroni correction divides alpha by the number of comparisons: for 5 hypotheses, alpha = 0.01. This makes effect detection more conservative but reduces the risk of false positives.

Metrics, Result Analysis and Decision Making

Choosing metrics is the most important stage that determines the quality of the experiment. Firebase A/B Testing offers several categories of metrics: engagement (daily active users, session duration, screens per session), monetization (revenue, purchases, subscriptions), retention (Day 1, Day 7, Day 28), conversion (conversion rate for the selected event). Custom metrics based on any Firebase Analytics events are also available.

Primary metric — the single metric on which the success of the experiment is judged. The choice of primary metric should be made before the experiment starts based on the hypothesis. If the hypothesis is “New onboarding will increase the conversion rate for registration,” then the primary metric is the conversion rate of the sign_up_completed event. Secondary metrics — additional indicators for analyzing side effects: whether retention decreased, whether revenue dropped.

Interpreting results: Firebase displays a table with metric values for each group, percentage difference from the control group, p-value and 95% confidence interval. If p-value < 0.05 and the confidence interval does not include 0 — the difference is statistically significant. If p-value > 0.05 — the result is inconclusive, and the experiment should be extended or stopped as indeterminate.

Making a decision based on results

Firebase A/B Testing offers three options after the experiment ends: apply the winning variant to all users, continue the experiment (if data is insufficient) or stop the experiment without applying (if all variants are worse than the control or the result is inconclusive). Applying the winner automatically updates the Remote Config template with the production value of the winning variant.

Caution: sometimes a statistically significant result has no practical meaning. For example, the test showed an increase in conversion rate of 0.5% (p = 0.03), but the new UI version requires 2 weeks of development. The cost-benefit ratio may be unfeasible. Make decisions based on business impact, not just statistical significance. Firebase shows not only the p-value but also the absolute change in the metric, which helps assess practical significance.

Advanced metrics: retention and LTV

Retention is one of the most important metrics for mobile applications as it is directly related to long-term user value (LTV). Firebase A/B Testing automatically calculates Day 1, Day 7 and Day 28 retention for each group. However, reliable retention measurement takes time: Day 7 retention can be assessed 7 days after the experiment starts, Day 28 retention — after 28 days. Plan the experiment duration considering the time needed to collect retention data.

LTV (Lifetime Value) is a more complex metric that requires integration of Firebase with Google Analytics for Firebase and, if necessary, with an attribution platform (Adjust, AppsFlyer). Firebase A/B Testing allows using LTV as a metric, but to calculate it, you need to set up data import for purchases and user acquisition costs. Without attribution, LTV may be inaccurate since Firebase does not see the cost of installs from advertising sources.

Setting Up an A/B Test via Remote Config

To conduct an A/B test via Firebase A/B Testing, no special client-side code is required — the entire experiment is configured in the Firebase console. However, the client code must correctly use Remote Config parameters so that the values assigned by the experiment are applied properly. Consider an example: an A/B test of a new subscription price, where the control group sees the old price ($9.99) and the experimental group sees the new one ($7.99).

In the Firebase console, we create a Remote Config parameter subscription_price with a default value of “9.99”. Then we create an A/B test where we specify the value “7.99” as the winning variant for 50% of users. Firebase automatically assigns each user to a group and delivers the corresponding value via Remote Config. The client code uses the standard getString to get the price.

Client code for applying an A/B test

The client code does not know about the experiment — it simply gets the parameter value from Remote Config. The Firebase SDK handles grouping on the server side. This is the main advantage of Firebase A/B Testing: the developer doesn’t need to write conditional distribution logic. The only requirement is that the app must regularly call fetchAndActivate to get current values.

class SubscriptionFragment : Fragment() {

    private fun loadPrice() {
        val remoteConfig = Firebase.remoteConfig
        val priceStr = remoteConfig
            .getString("subscription_price")
        val price = priceStr.toDoubleOrNull() ?: 9.99
        priceView.text = "$$price/month"
    }

    override fun onViewCreated(...) {
        super.onViewCreated(...)
        loadPrice()
    }
}

In the example, loadPrice gets the value of the subscription_price parameter via Remote Config. The Firebase SDK automatically returns the value corresponding to the user’s group within the active A/B test. If the experiment is not active or the user is not in a group — the default value is returned. This makes the code completely independent of the presence or absence of experiments.

Logging analytics events for metrics

For Firebase A/B Testing to work correctly, the app needs to log the events selected as experiment metrics. The Firebase Analytics SDK automatically collects standard events (first_open, session_start, in_app_purchase, etc.), but for custom metrics, logging needs to be added. The example below logs the subscription_started event when a user attempts to subscribe.

private fun onSubscribeClick() {
    // Logging the event for an A/B test
    val bundle = Bundle().apply {
        putString(
            FirebaseAnalytics.Param.PRICE,
            remoteConfig.getString("subscription_price")
        )
    }
    FirebaseAnalytics.getInstance(requireContext())
        .logEvent("subscription_started", bundle)

    // Starting the payment flow
    startBillingFlow()
}

Important: the subscription_started event must be registered in Firebase Analytics as a custom event (for reports) or be a standard event used by Firebase A/B Testing. Firebase automatically links the event to the experiment group via the Analytics App Instance ID. No additional tagging is needed — all the magic happens on the Firebase server side.

Common Mistakes When Conducting A/B Tests

Peek effect error — stopping the experiment at the first appearance of statistical significance without considering the planned duration. If you check p-value daily and stop as soon as p < 0.05, the probability of a false positive result increases from 5% to 30–40%. Firebase A/B Testing recommends a fixed experiment duration. Do not look at the results before the estimated period ends.

Unaccounted external factors — seasonality, advertising campaigns, OS updates, competitor releases. If during an A/B test you launched an advertising campaign that changed the traffic composition, the test result may be skewed. It is recommended not to conduct A/B tests simultaneously with major marketing activities. If unavoidable — make sure that traffic from advertising is evenly distributed between groups.

Segmental effect (Simpson’s Paradox) — a situation where the overall result shows no effect, but within individual segments the effect exists and is opposite. For example, a test showed that the new checkout design did not change the overall conversion, but when broken down into iOS and Android, it turned out: on iOS conversion increased by 20%, while on Android it dropped by 15%. Always check results by key segments (platform, country, app version).

Multiple metrics problem

Multiple comparison problem arises when many metrics are used in an experiment. If you check 20 metrics with alpha = 0.05, the probability of finding at least one falsely significant difference (false positive) is 1 — (0.95)^20 ≈ 64%. Firebase uses Bonferroni correction for several primary metrics but not for secondary ones. Conclusion: choose one primary metric before the experiment starts and ignore p-values of secondary metrics when making decisions.

Novelty effect — users may react differently to a new change simply because it is new, not because it is better. The first days of an experiment may show false growth (users click on a new button out of curiosity), which subsides over time. The minimum experiment duration of 3 days partially solves this problem, but for UI changes, a duration of 7–14 days is recommended to allow the novelty effect to stabilize.

Interference between experiments

Network effect — a problem where the behavior of users in one group affects users in another group. For example, an A/B test of a news feed algorithm change: if the experimental group receives better recommendations, they create more content that users in the control group also see, skewing the results. In such cases, use social graph isolation or conduct the test at the country/region level.

Simultaneous experiments on the same Remote Config parameter are another source of interference. Firebase A/B Testing does not allow launching a second experiment on an already occupied parameter, but if experiments affect different parameters yet influence the same metric, cross-effect is possible. It is recommended to run no more than 2–3 active A/B tests simultaneously and ensure they do not affect the same user scenarios.