ML Kit: Libraries and ML for Android and iOS

What is ML Kit: Capabilities and Architecture

ML Kit is a Firebase-compatible SDK for mobile platforms providing 14 ML APIs for Android and iOS. ML Kit replaced Firebase ML (old name) in 2020 and is now available as a standalone SDK via Google Play Services (Android) or CocoaPods/SPM (iOS). The key advantage is that all models work on-device, ensuring data privacy and offline operation.

ML Kit Architecture is built around two modes: bundled (model embedded in APK/IPA) and downloaded (model downloaded via Google Play Services on first call). Bundled mode provides instant startup but increases app size by 5–20 MB. Downloaded mode saves space but requires internet on first launch. Google recommends downloaded for APIs not used on every screen (Object Detection, Pose Detection).

Customization via TFLite — ML Kit allows you to load your own TensorFlow Lite model through the Custom Model API. This provides flexibility — use ready-made APIs for standard tasks and your own model for specific ones. ML Kit provides a universal Input/Output handler working with ByteBuffer and FloatArray. According to Google (2025), 40% of ML Kit projects combine ready-made APIs and custom models.

// ML Kit Text Recognition setup (Android)
val recognizer = TextRecognition.getClient(TextRecognizerOptions.DEFAULT_OPTIONS)

val image = InputImage.fromBitmap(bitmap)
recognizer.process(image)
    .addOnSuccessListener { result ->
        for (block in result.textBlocks) {
            Log.d("MLKit", block.text)
        }
    }
    .addOnFailureListener { error ->
        // Processing error
    }

Firebase vs standalone — ML Kit can be used with Firebase (Cloud functions, Analytics, Crashlytics) or as a standalone SDK without Firebase. Standalone mode connects via Google Play Services (Android) without setting up a Firebase account. Cloud APIs are available via Firebase (Cloud Vision, Natural Language) for tasks requiring neural networks on Google's server — but these features are paid and not on-device.

ML Kit Text Recognition: Recognizing Text

ML Kit Text Recognition — API for optical character recognition (OCR) on images. Supports two modes: Latin-based (Latin, Cyrillic, digits — 45 languages) and Chinese/Japanese/Korean (CJK — ideographic languages). Latin recognition uses V2 model (faster) or V1 (high accuracy). V2 delivers 10–30 ms per frame, V1 — 50–100 ms.

Text Recognition Capabilities include recognizing printed and handwritten text, detecting text orientation, finding lines, words and characters, determining text language. The API returns a structure: Text → TextBlock → Line → Element (Word/Symbol). Each element has bounding box, confidence and recognized text. According to Google (2025), ML Kit Text Recognition V2 accuracy on Latin script is 96%, on Cyrillic — 91%.

Real-time Text Recognition — using CameraX or Camera2 for video stream. Create an ImageAnalysis.Analyzer and pass frames to ML Kit. For performance, reduce frame resolution to 480p (640x480) — this is the optimal balance between speed and accuracy. ML Kit automatically handles image rotation, but for maximum speed pass the image in the correct orientation.

// Text Recognition with CameraX Analyzer
class TextAnalyzer : ImageAnalysis.Analyzer {
    private val recognizer = TextRecognition.getClient(
        TextRecognizerOptions.DEFAULT_OPTIONS
    )

    override fun analyze(image: ImageProxy) {
        val inputImage = InputImage.fromMediaImage(
            image.image, image.imageInfo.rotationDegrees
        )
        recognizer.process(inputImage)
            .addOnSuccessListener { /* text found */ }
            .addOnCompleteListener { image.close() }
    }
}

Text Recognition Optimization: use ImageDecoder to improve recognition of blurry or dark images. For printed text — TextRecognizerOptions.DEFAULT_OPTIONS (V2 model). For handwritten text — TextRecognizerOptions.SCRIPT_LATIN (more accurate but slower). In IT Sectr projects we use V2 for document recognition and Latin model for checks and receipts.

ML Kit Face Detection: Detecting Faces and Contours

ML Kit Face Detection — API for detecting faces in images and video. Detects face bounding box, eye, nose, mouth, ear coordinates (468 contour points) and basic expressions: smile, open mouth, closed eyes. Face Detection is one of the fastest ML Kit APIs: 5–15 ms per frame depending on face count and contour points.

Face Detection Modes: contour mode (468 face contour points for precise mask/filter overlay), classification mode (detecting smile, open eyes), landmark mode (key points without contour). Modes are combined in FaceDetectorOptions. Enabling all modes slows detection by 2–3x — use the minimum required set for your task.

Face Detection in AR Applications — based on contour points ML Kit builds a face mask for Snapchat-like filters. ML Kit returns 468 points with x, y, z coordinates (z = depth). Points update 30–60 times per second on modern devices. For AR overlay use FaceMeshDetector (specialized version) — it also returns mesh triangles for texturing.

// Face Detection with contour points
val options = FaceDetectorOptions.Builder()
    .setPerformanceMode(FaceDetectorOptions.PerformanceMode.FAST)
    .setContourMode(FaceDetectorOptions.ContourMode.ALL)
    .setClassificationMode(FaceDetectorOptions.ClassificationMode.ALL)
    .build()
val detector = FaceDetection.getClient(options)

detector.process(inputImage)
    .addOnSuccessListener { faces ->
        faces.forEach { face ->
            val bounds = face.boundingBox
            val smileProb = face.smilingProbability
        }
    }

Face Detection Limitations: the API does not recognize who a face belongs to (face recognition) — it only detects faces. For identity recognition use FaceNet or ArcFace on a custom TFLite model. ML Kit works correctly with faces at angles up to 45°, with glasses and partial masks. For profile angles (90°) accuracy drops to 40–60%.

ML Kit Barcode Scanning: Reading All Code Formats

ML Kit Barcode Scanning — API for reading and decoding 1D (EAN, UPC, Code 128) and 2D (QR, Data Matrix, PDF417) barcodes. Barcode Scanning detects the code in the image — unlike ZXing which requires the code to occupy almost the entire frame. ML Kit detects codes of any size and orientation, including multiple codes in one frame (multi-barcode mode).

Supported Formats: EAN-8, EAN-13, UPC-A, UPC-E, Code 39, Code 93, Code 128, ITF, Codabar, QR, Data Matrix, PDF417, Aztec. ML Kit automatically determines the format — no need to specify the code type. In production use setBarcodeFormats() to limit supported formats — this speeds up scanning by 30–50% by excluding unnecessary decoding algorithms.

Real-time Barcode Scanning — similar to Text Recognition, integration with CameraX. ML Kit processes frames at up to 60 FPS. For maximum speed enable setPerformanceMode(PerformanceMode.FAST). ML Kit Barcode Scanning is 2–3x faster than ZXing and more accurate at detecting blurry or partially covered codes. According to Google (2025), ML Kit Barcode Scanning has 98.5% accuracy under standard lighting.

// Barcode Scanning with format filter
val options = BarcodeScannerOptions.Builder()
    .setBarcodeFormats(Barcode.FORMAT_QR_CODE,
        Barcode.FORMAT_EAN_13)
    .build()
val scanner = BarcodeScanning.getClient(options)

scanner.process(inputImage)
    .addOnSuccessListener { barcodes ->
        barcodes.forEach { barcode ->
            val value = barcode.rawValue
            val type = barcode.format
        }
    }

Comparison with ZXing: ML Kit is faster, more accurate and easier to integrate. ZXing is open-source, works fully offline, does not require Google Play Services. ML Kit requires Google Play Services (Android) or CocoaPods (iOS). For apps running on devices without Google (Huawei, Amazon Fire), use ZXing as fallback. For others — ML Kit, as it provides better UX through multi-code detection.

Object Detection and Pose Detection in ML Kit

ML Kit Object Detection — API for detecting and tracking objects in images. Detects objects from 1000+ categories (ImageNet classes) — people, animals, vehicles, household items. Object Detection works in two modes: single-image (single photo) and streaming (video stream with tracking). Streaming mode tracks objects between frames, assigning each a unique track ID.

Pose Detection — API for determining human pose using 33 key points (skeleton): head, shoulders, elbows, wrists, hips, knees, feet. Determines coordinates (x, y, z) and confidence for each point. Pose Detection is used in fitness trackers (rep counting, form checking), AR applications (avatar mimics movements) and sports analytics. Speed — 10–30 ms per frame depending on person count.

Object Tracking — ML Kit Object Detection with inter-frame tracking uses an Optical Flow-based tracker. When an object is detected, the tracker follows it without re-detection, saving CPU/GPU resources. If the tracker loses the object (fast motion, occlusion), ML Kit restarts detection. According to Google (2025), the tracker holds the object in 95% of frames at speeds up to 30 km/h.

// Pose Detection (human skeleton)
val poseDetector = PoseDetection.getClient(
    PoseDetectorOptions.Builder()
        .setDetectorMode(PoseDetectorOptions.STREAM_MODE)
        .build()
)

poseDetector.process(inputImage)
    .addOnSuccessListener { pose ->
        pose.allPoseLandmarks.forEach { landmark ->
            val type = landmark.landmarkType // LEFT_SHOULDER, RIGHT_ELBOW...
            val position = landmark.position3D
        }
    }

Selfie Segmentation — API for segmenting a person from an image (separating person from background). Returns a confidence mask for each pixel. Selfie Segmentation is used for blurring or replacing backgrounds in video calls, photo editors and AR applications. The mask is returned as UInt8Array sized to the original image — each pixel contains a value from 0.0 (background) to 1.0 (person).

Custom TFLite Models in ML Kit

Custom Model API — the ability to load and run your own TensorFlow Lite models through the ML Kit interface. ML Kit provides a unified Input/Output API: ByteBuffer as input, FloatArray/TensorImage as output. This allows leveraging ML Kit advantages (model management, image processing, CameraX integration) with custom models for face recognition, object classification, NLP and more.

Loading a Model — place the .tflite file in assets/ (Android) or bundle (iOS) and load via CustomModelDownloadConditions or Firebase Model Manager. For downloading large models (5+ MB) use download conditions: Wi-Fi, charged, background. Google recommends downloading the model on first app launch, not at the moment of first use.

// Loading custom TFLite model
val conditions = CustomModelDownloadConditions.Builder()
    .requireWifi()
    .build()
val remoteModel = CustomRemoteModel.Builder("my_model")
    .setRemoteModelName("my_model_v2")
    .build()

remoteModel.download(conditions)
    .addOnSuccessListener { /* model ready */ }
    .addOnFailureListener {
        // Use bundled model from assets
    }

Custom Model Optimization: use TFLite Converter with delegate compatibility. For maximum performance quantize the model (INT8) — this reduces model size by 4x and speeds up inference by 2–3x through XNNPACK Delegate. ML Kit Custom Model API supports Dynamic Shape (batch = 1), Image Input (UInt8, Float32) and Tensor Output.