Firebase Test Lab — what it is, device testing types and automation

What is Firebase Test Lab

Firebase Test Lab is Google's cloud infrastructure for automated testing of mobile applications on physical devices located in Google's global data centers. Unlike emulators on a local machine, Test Lab runs tests on real Android devices and iOS devices with various operating system versions, screen resolutions, RAM sizes and API levels. The service automatically processes test results, providing execution videos, step screenshots and detailed logs.

Benefits of Cloud Testing

Maintaining your own device farm requires significant capital expenditure on purchasing and maintaining physical devices. Firebase Test Lab provides access to thousands of devices on a pay-per-use basis with a free limit on the Spark plan. Test Lab automatically updates the device pool to current models, including the latest flagship Google Pixel, Samsung Galaxy, Xiaomi and iPhone devices. Tests run in parallel on multiple devices, reducing the overall application verification time.

Supported Platforms

For Android, Test Lab supports testing on devices with API levels from 21 to the latest stable Android version. iOS support includes real iPhone and iPad devices with iOS versions from 15 to the latest stable version. Robot testing and XCTest are available for iOS. Each test can be run on physical devices or emulators — for the most accurate results, Google recommends using physical devices as they reveal issues with sensors, camera and network interfaces.

Test Types in Firebase Test Lab

Firebase Test Lab supports several test types, each solving its own task in the development cycle: robot tests for quick stability checks, instrumentation tests for functional verification, and screenshot tests for detecting visual regressions. The choice of test type depends on the development stage and goals of a particular run.

Robot Tests (Robo tests)

Robot tests are a unique feature of Firebase Test Lab that does not require writing test code. Firebase Crawler automatically explores the application: finds all interactive UI elements, clicks on them, fills forms with test data, navigates between screens and records any failures (crash, ANR, exceptions). Robot tests detect up to 80% of critical bugs without a single line of test code, making them an ideal tool for assessing the basic stability of an application in the early stages of development.

Instrumentation Tests

Instrumentation tests are tests written by the developer using Espresso (Android) or XCTest (iOS) frameworks. Test Lab runs the APK with tests on the target device and collects execution results. Tests can be simple (checking element display) or complex (end-to-end scenarios: login, add to cart, checkout). Firebase Test Lab supports Android Test Orchestrator for isolating tests from each other, preventing one failed test from affecting the others.

Screenshot Tests (Game loop and screenshots)

Screenshot tests are used to detect visual regressions in the application. Game loop is a special test type that passes a list of scenarios to the application via intent, and the application must execute these scenarios and take screenshots of key screens. Test Lab collects screenshots from all devices in the matrix and allows comparing them with reference images to detect differences. This test type is especially popular in gaming applications for checking rendering on different GPUs.

Device Matrix and Configurations

Testing matrix in Firebase Test Lab is a set of device configurations on which the selected test is executed. Each configuration includes the device model, OS version, screen orientation (portrait/landscape) and regional settings (locale, language). The test runs in parallel on all selected configurations, significantly reducing the total execution time compared to sequential running on each device.

Choosing Devices for the Matrix

Google recommends including devices with different characteristics in the matrix: a flagship model (Google Pixel, Samsung Galaxy S), a budget device (Moto G, Xiaomi Redmi), a device with the minimum API version supported by the application. An optimal matrix includes 5–10 configurations covering all OS versions that the application supports. For iOS, iPhone and iPad models of different generations are selected (e.g., iPhone SE, iPhone 14, iPad Air) with different iOS versions.

Test Grouping and Parallelism

Test Lab automatically distributes tests across selected devices, running each test on each configuration. The execution time of the entire matrix equals the execution time of the slowest test on the slowest device — all others run in parallel. If one test fails on all devices, Test Lab stops its execution on the remaining configurations to save resources. Results are grouped by test and by device, allowing quick identification of whether an error is specific to a particular configuration.

Robot Testing: Automatic Exploration

Robot testing is a flagship feature of Firebase Test Lab that allows you to automatically explore an application without writing a single test. Firebase Crawler (robot) analyzes the application's UI hierarchy, finds all interactive elements (buttons, text fields, lists, menus) and interacts with them, simulating real user behavior. Robot testing is especially effective in the early stages of development, when tests have not yet been written, or for checking legacy code.

Setting Up a Robot Test

To run a robot test, simply upload an APK or IPA to Firebase Console and select the Robo test option. The robot can automatically log into the application through pre-configured credentials (login/password) to access screens hidden behind authentication. You can also set the exploration depth (max depth) to limit the number of robot steps, or specify specific elements that the robot should click on with priority.

<!-- res/xml/robo_directives.xml -->
<!-- The robot will fill in the login and password fields -->
<?xml version="1.0" encoding="utf-8"?>
<robo-directives
    xmlns:android="http://schemas.android.com/apk/res/android">
    <directive
        android:resourceName="@id/et_email"
        android:inputValue="test@example.com" />
    <directive
        android:resourceName="@id/et_password"
        android:inputValue="password123" />
    <directive
        android:resourceName="@id/btn_login"
        android:clickAction="CLICK" />
</robo-directives>

Limitations of Robot Tests

Robot tests do not replace manual testing and written instrumentation tests for complex scenarios. The robot cannot check business logic: it clicks on all elements in sequence, but does not know if the result is correct. Also, the robot cannot interact with custom views that do not inherit from standard ViewGroup, and does not handle WebView. For these cases, instrumentation tests or Game loop must be used.

CI/CD and Gradle Integration

Firebase Test Lab integration with CI/CD systems allows you to automatically run tests with every push to the repository. Test Lab supports Gradle tasks for Android and Fastlane for both platforms. Setting up a CI/CD pipeline includes building the application, uploading the APK/IPA to Google Cloud Storage, running tests via Firebase CLI and getting reports back to the CI system for displaying build status.

Setup via Gradle (Android)

Google provides the gcloud-test-lab plugin for Gradle, which automates test uploading and execution. The assembleAndroidTest task builds the APK with tests, and gcloud tasks upload it to Test Lab and run the matrix. Test results are returned to Gradle as pass/fail, allowing Test Lab integration into any CI pipeline that supports Gradle. A Firebase service account with Test Lab permissions is used for authorization.

// build.gradle (module: app)
task runTestLab() {
    exec {
        commandLine "gcloud", "firebase", "test", "android", "run",
            "--type", "robo",
            "--app", "build/outputs/apk/debug/app-debug.apk",
            "--device", "model=Pixel7,version=33",
            "--device", "model=GalaxyS23,version=34",
            "--device", "model=RedmiNote12,version=31",
            "--timeout", "15m",
            "--results-dir", "firebase/test-lab/robo"
    }
}

Setup via Fastlane

Fastlane provides the firebase_test_lab plugin for integration with iOS and Android tests. A Fastlane lane performs the build, runs tests and processes results. For iOS XCTest, Test Lab requires an xctestrun file, which is created automatically when building the test target. Fastlane can also upload test results to Slack or other messengers to notify the team about testing status after each run.

Test Results Analysis

After test completion, Firebase Test Lab provides a comprehensive report available in Firebase Console and via gcloud CLI. The report includes a brief summary (number of passed, failed and skipped tests), detailed information for each test on each device, execution videos, screenshots of key steps and device logs (logcat for Android, system log for iOS). For failed tests, a stack trace with the exact error location is available.

Interpreting Robot Test Results

For a robot test, the report differs from an instrumentation one: instead of pass/fail per test case, the report shows exploration statistics. Key metrics — the number of found and visited screens, the number of robot interactions (clicks, swipes, text input), detected crashes and ANRs, and the path traversed through the application. The robot execution video allows you to visually assess which screens were tested and which remained uncovered.

Exporting Results

Test results are stored in Google Cloud Storage in the project's associated bucket. The data is available for export and integration with external analytics systems. For long-term storage and comparison of results between runs, it is recommended to set up BigQuery export or use the Firebase Test Lab API for programmatic result retrieval. Google provides the ability to download a ZIP archive with the full set of results for each test run.