ARCore: what it is, features, and augmented reality technology

What is ARCore?

ARCore is Google's augmented reality platform, announced in 2018 as a successor to Project Tango in a lighter and more accessible form. Unlike Tango, which required specialized depth sensors, ARCore works on standard Android smartphones using only the camera and inertial sensors (IMU).

ARCore uses three key technologies: Motion Tracking for device position tracking, Environmental Understanding for detecting horizontal and vertical planes, and Light Estimation for adjusting virtual object lighting to match the real environment. These three technologies allow placing AR objects that look natural on any surface.

According to Google I/O 2024, ARCore is supported on over 1 billion Android devices, including smartphones from Samsung, Xiaomi, Huawei, OnePlus, Google Pixel, and others. The platform is integrated with Google Play Services for AR, allowing ARCore to be updated via Google Play Store independently of the Android version.

Evolution of ARCore

Over six versions, ARCore has undergone significant evolution: ARCore 1.0 — basic tracking and plane detection; ARCore 1.2 — vertical planes and Augmented Images; ARCore 1.7 — Cloud Anchors for multiplayer sessions; ARCore 1.10 — Augmented Faces for facial expression tracking.

ARCore 1.20+ includes Scene Semantic Understanding — recognition of object categories (floor, ceiling, wall, furniture, plant, door) via an on-device TensorFlow Lite ML model. The Street View Geometry API allows using Google Street View depth maps for outdoor AR navigation.

// Initializing an ARCore session
val session = Session(context)
val config = Config(session)
config.lightEstimationMode =
    Config.LightEstimationMode.ENVIRONMENTAL_HDR
config.planeFindingMode =
    Config.PlaneFindingMode.HORIZONTAL_AND_VERTICAL
session.configure(config)

// Frame processing
val frame = session.update()
val pointCloud = frame.acquirePointCloud()
val planes = session.getAllTrackables(
    Plane::class.java
)

How ARCore works

ARCore uses a hybrid approach: the IMU (Inertial Measurement Unit) predicts device position at high frequency (up to 1000 Hz), while visual odometry corrects errors based on feature point analysis from the camera at 30–60 Hz. This approach provides tracking accuracy of approximately 1–2 centimeters in stationary conditions.

The Environmental Understanding process works in three stages: first, ARCore finds feature points in the camera image, then identifies clusters of points lying on the same plane and approximates them into a polygon. After plane recognition, ARCore creates a Trackable object of type Plane, which updates as new frames are received.

According to Google, ARCore detects a plane as small as 10x10 centimeters at a distance of up to 1 meter from the camera. For medium-sized rooms (20 m²), full plane scanning takes about 2–4 seconds. When using the Depth API with ToF sensor support, recognition accuracy improves by 40%.

Depth API and Scene Mesh

Depth API is an ARCore technology that builds a scene depth map based on stereo image analysis from the camera. On devices with a ToF sensor (Time-of-Flight), the depth map is obtained instantly with an accuracy of up to 2 cm. On devices without a depth sensor, the Depth API uses machine learning to estimate depth from a single frame, providing accuracy of about 10 cm.

Scene Mesh is a 3D mesh of the environment that ARCore builds in real time. The mesh is used for collision physics, occlusion of virtual objects, and shadow rendering. Scene Mesh is available on devices with OpenGL ES 3.1 and Depth API support, generating up to 500 thousand polygons per frame.

// Getting the depth map
val depthImage = frame.acquireDepthImage16Bits()

// Placing an object on a detected plane
val hitResult = frame.hitTest(x, y)
    .firstOrNull { it.trackable is Plane }

hitResult?.let { result ->
    val anchor = session.createAnchor(
        result.hitPose
    )
    val model = ModelRenderable.builder()
        .setSourceId(R.raw.my_model)
        .build()
        .exceptionally { null }

    AnchorNode(anchor).apply {
        this.renderable = model.get()
    }
}

ARCore vs ARKit: comparison

Although ARCore and ARKit solve the same tasks — position tracking, plane detection, and lighting estimation — their approaches and ecosystems differ. ARCore prioritizes cross-platform compatibility, whereas ARKit uses deep integration with Apple hardware.

Key differences: ARCore works on Android and iOS via the ARCore SDK, while ARKit works only on iOS. ARCore uses Cloud Anchors for synchronization via Firebase, while ARKit uses ARWorldMap for direct device-to-device exchange. ARCore with Depth API supports ToF sensors, while ARKit with LiDAR provides laser scanning with up to 3 cm accuracy.

According to Google 2024 benchmark tests, ARCore tracking accuracy on devices with gyroscope and ToF sensor is comparable to ARKit on iPhone with LiDAR — error does not exceed 2 cm at a distance of 2 meters. On devices without ToF, ARCore lags behind ARKit in initialization speed: 3 seconds versus 0.5 seconds.

The choice between ARCore and ARKit depends on the target audience: if the app is iOS-only, ARKit delivers better performance and access to exclusive LiDAR features. If cross-platform support is needed, ARCore with the ArcoreiOS SDK allows using a single codebase with minimal platform differences, though with some loss of hardware-specific capabilities of each platform.

Integrating ARCore into an app

ARCore integration starts with adding the dependency for Google Play Services for AR in build.gradle: com.google.ar:core. The app should check ARCore availability on the device via ArCoreApk.checkAvailability() — if ARCore is not installed, prompt the user to install it from Google Play Store.

A basic ARCore project includes: creating a Session and Config, setting up a SurfaceView for camera display, starting the session via session.resume(), and processing frames via session.update(). For 3D object rendering, the Sceneform SDK (Google's library based on Filament) or Unity/Unreal for complex scenes is used.

Google recommends Sceneform for Kotlin apps — it provides ready-made components: ModelRenderable for loading 3D models (OBJ, glTF, FBX), ViewRenderable for embedding Android Views into an AR scene, and MotionEvent for touch handling. Sceneform automatically manages the AR session lifecycle and resource cleanup.

Handling tracking failures

ARCore may lose tracking under poor lighting, uniform surfaces, or rapid device movements. Tracking state is available via frame.getTrackingState() — values are TRACKING, PAUSED, and STOPPED. When transitioning to PAUSED, it is recommended to pause rendering and show user guidance: "Move the device more slowly" or "Move to a brighter area."

For automatic recovery after failure, ARCore supports relocalization based on a saved feature point map. Developers can save the map via session.serialize() and restore it via session.deserialize() — useful for long-running AR experiences where the user returns to the same physical space. Map restoration takes 1–2 seconds provided the environment has not changed significantly.

class ARCoreActivity : AppCompatActivity() {

    private lateinit var session: Session

    override fun onResume() {
        super.onResume()
        when (ArCoreApk.getInstance()
            .checkAvailability(this)
        ) {
            ArCoreApk.Availability.SUPPORTED_INSTALLED -> {
                session = Session(this)
                session.resume()
            }
            else -> {
                showInstallPrompt()
            }
        }
    }
}