Screenshot Test: What It Is, Types and How It Works in Testing

What is Screenshot Test and why is it needed?

Screenshot Test is an end-to-end user interface testing where the test opens an application screen, performs actions (taps, text input, scroll) and takes a screenshot of the resulting state. The screenshot is compared with a baseline stored in the repository. If the screenshots differ — the test fails. Screenshot tests detect visual regressions that unit tests cannot see: incorrect margins, overlapping elements, wrong colors.

Why do we need screenshot tests if we have golden tests — golden tests check components in isolation: one button, one card, one text. Screenshot tests check an entire screen in an environment as close to production as possible: real navigation, real data (or maximally realistic mocks), real system fonts, real status bar. Only a screenshot test will show that a button overlaps with another element on a real device.

Business value of screenshot tests

Business value — according to Google (2023), visual bugs account for 15-25% of all mobile application bugs. Screenshot tests automate visual quality checking that used to be done manually by QA engineers. One screenshot test replaces 5-10 minutes of manual testing of one screen. For an application with 50 screens, savings: 4-8 person-hours per regression run. Screenshot tests pay off in 2-3 release cycles.

Screenshot Test vs Golden Test: comparison of approaches

Golden tests are faster and simpler: rendering a component in off-screen buffer takes milliseconds, does not require a device, and is stable on CI. Screenshot tests are more realistic: they capture a real screen with system elements, support animations and navigation, and work on real devices. The choice depends on the goal: fast feedback for the developer (golden) or maximum realism before release (screenshot).

Coverage strategy: golden + screenshot

Golden + Screenshot — use golden tests for each UI component in the component library (Design System). 80% of visual regressions are caught at the component level. Screenshot tests — for critical user paths: onboarding, login, payment flow, shopping cart. 20% of regressions related to component integration on a real screen are only caught by screenshot tests. At IT Sectr we use an 80/20 ratio: 400 golden + 100 screenshot.

When a screenshot test is not needed — if the screen consists of static content without interactivity, a golden component test provides the same level of verification at a lower cost. If the screen changes dynamically (feed, chat), a screenshot test requires complex data setup and wait times. In such cases, use screenshot for the baseline state (empty list, loading) and golden for individual cards in the list.

UI Automator and Firebase Test Lab for Android

UI Automator is an Android framework for cross-application UI testing. It allows taking screenshots via UiDevice.takeScreenshot(). Unlike Espresso (works inside a single application), UI Automator can interact with system dialogs (permissions, notifications) and other applications. A UI Automator screenshot test: open the app, wait for loading, take a screenshot, compare with baseline.

class LoginScreenScreenshotTest {

    @get:Rule
    val rule = ComposeTestRule.createAndroidComposeRule<MainActivity>()

    @Test
    fun login_screen_default() {
        val device = UiDevice.getInstance(
            InstrumentationRegistry.getInstrumentation()
        )

        // Waiting for the screen to load
        IdlingRegistry.getInstance().waitForIdle()

        // Taking a screenshot
        val screenshot = device.takeScreenshot()
        val golden = loadGolden("login_default.png")

        // Comparing with the reference
        val diff = ImageComparator.compare(screenshot, golden)
        assertTrue(diff.similarity > 0.98)
    }
}

Firebase Test Lab is a Google Cloud service for running instrumented tests on hundreds of real devices in parallel. Screenshot tests on Firebase Test Lab capture screenshots on different devices (Pixel 7, Galaxy S24, Xiaomi 14) and compare with baselines. Advantage: one test checks UI on 20 devices in 10-15 minutes. Disadvantage: cost ($1-5 per test on 20 devices). Firebase Test Lab integrates with CI via gcloud CLI or Gradle plugin.

Shot: library for simplifying screenshot tests

Shot is a library for screenshot testing on Android that simplifies creating and comparing screenshots. Shot works on top of Espresso and UI Automator, adding manage golden (create, update, delete), comparison with threshold (pixels or percentages) and HTML report generation. Shot is suitable for projects that want to quickly implement screenshot testing without writing their own image comparison infrastructure.

XCUITest and Xcode Cloud for iOS

XCUITest is Apple’s framework for UI testing of iOS, iPadOS and tvOS applications. Screenshot tests on XCUITest use XCUIScreen.main.screenshot() for screen capture and XCAttachment for saving screenshots. XCUITest simulates user actions: tap, swipe, typeText, and takes screenshots after each step. In Xcode 16+, built-in screenshot comparison with baselines through XCTAttachment is added.

final class LoginScreenScreenshotTests: XCTestCase {

    var app: XCUIApplication!

    override func setUp() {
        super.setUp()
        app = XCUIApplication()
        app.launch()
    }

    func test_login_initial_state() {
        let loginButton = app.buttons["login_button"]
        XCTAssertTrue(loginButton.exists)

        // Taking a screenshot
        let screenshot = app.screenshot()
        let attachment = XCTAttachment(screenshot: screenshot)
        attachment.name = "Login-Screen-Initial"
        attachment.lifetime = .keepAlways
        add(attachment)

        // Comparison with the reference (requires XCTAttachment + golden)
        assertScreenshot(
            screenshot: screenshot,
            goldenName: "login_initial_state"
        )
    }
}

Xcode Cloud is Apple’s cloud CI for building and testing iOS applications. Xcode Cloud supports running XCUITest tests on simulators. Screenshot tests can be run on multiple simulators in parallel (iPhone 15, iPhone 15 Pro Max, iPad Pro). Results: XCResult Bundle with attachments. Xcode Cloud is not built into GitHub/GitLab — use Xcode Cloud Webhooks for integration. Alternative: GitHub Actions with macos-14 and xcodebuild.

Comparison frameworks — iOSSnapshotTestCase (Uber) also works for screenshot tests if run on a simulator. SwiftSnapshotTesting (pointfree) is more oriented toward component golden tests. For iOS screenshot tests, use built-in XCUITest tools + XCTAttachment + custom ImageComparator (Pixelmator or AImage). On CI use simulator — on real devices screenshot tests only work via Device Farm (AWS Device Farm).

Screenshot test automation process in CI

Baseline management — baseline screenshots are stored in the repository (Git LFS) or in S3. Each screenshot is named by template: {testName}_{device}_{orientation}_{locale}.png. Example: loginScreenPixel7PortraitRu.png. When adding a new device or locale, a new baseline is created. When changing the UI, old baselines are replaced with new ones after code review. Baseline is part of the code base, like test sources.

CI Pipeline — (1) Build the application. (2) Run screenshot tests on emulators/simulators. (3) Compare screenshots with baselines. (4) On mismatch — generate diff image. (5) Upload diff artifacts (actual, expected, diff — three files). (6) Publish HTML report with results table. (7) If threshold exceeded — test fails. (8) Reviewer reviews diff artifacts and makes a decision: approve (update baseline) or reject (fix code).

Threshold and tolerance — absolute pixel-by-pixel comparison is too strict. Use SSIM (Structural Similarity Index) or MSE (Mean Squared Error). SSIM 0.98 = 98% structural similarity — a good threshold. Different screens may require different thresholds: dark theme (more black — higher accuracy), gradients (more noise — lower accuracy). Configure threshold per-test via parameter: @ScreenshotTest(threshold = 0.99).

Device Farm vs Simulator — tests on real devices (Firebase Test Lab, AWS Device Farm) provide maximum realism but are slow and paid. Tests on simulators/emulators are fast and free but do not show real-device features (different GPUs, display color reproduction, pixel density). Strategy: simulator for pre-merge check (5 minutes), Device Farm for nightly (30 minutes, 20 devices). At IT Sectr we use Firebase Test Lab for nightly runs on top-10 Android devices.