Object Detection: What It Is, Algorithms and How It Works

What Is Object Detection: How It Works

Object Detection solves two tasks simultaneously: what is in the image (classification) and where (coordinate regression). The model divides the image into a grid, predicts a bounding box (x, y, width, height) and class probabilities for each cell. Non-Maximum Suppression (NMS) removes duplicate boxes for a single object, keeping the most confident prediction. Modern architectures are divided into two-stage (Faster R-CNN — first region proposals, then classification) and one-stage (YOLO, SSD — everything at once).

Evaluation Metrics: mAP (mean Average Precision) — the main quality metric for Object Detection. mAP@0.5 — accuracy at IoU threshold 0.5, mAP@0.5:0.95 — average across thresholds from 0.5 to 0.95. FPS — frames per second (inference speed). For mobile applications, the mAP/FPS balance is critical: YOLOv8-Nano delivers 42% mAP@0.5:0.95 at 50+ FPS, SSD MobileNet — 22% mAP at 60+ FPS. For real-time > 20 FPS, for interactive use 5–15 FPS.

IoU (Intersection over Union) — a metric for overlap between predicted and ground truth bounding box. IoU = intersection area / union area. The IoU threshold for NMS is typically 0.5–0.7. For object tracking between frames (re-identification), use Deep SORT or ByteTrack with IoU matching. For counting objects in video, BoT-SORT provides 85% MOTA (Multiple Object Tracking Accuracy).

Anchor Boxes — predefined bounding box shapes that the model adjusts. YOLOv8 uses anchor-free detection, which simplifies training and speeds up inference. SSD and older YOLO require anchor tuning for each dataset. For mobile development, anchor-free architectures (YOLOv8, FCOS) are preferable — they do not depend on object size and are easier to customize.

ML Kit Object Detection and Tracking

ML Kit Object Detection and Tracking — Google's library for on-device object detection and tracking. Supports two modes: single-image detector (for photos) and streaming detector (for video). Streaming mode automatically tracks objects between frames using optical flow, reducing latency to 5–10 ms between frames after initial detection. Categories: fashion, food, home, places — 10–20 classes each.

ML Kit API: ObjectDetector (options: detectorMode, enableClassification, enableMultipleObjects) → InputImage → DetectedObject (trackingId, boundingBox, classificationCategory, classificationConfidence). TrackingId allows tracking the same object across frames. MultipleObjects = true — detects up to 5 objects per frame. Classification — enables object category recognition (default false for speed). Streaming mode is recommended for real-time: 20–30 FPS on Pixel 6.

val options = ObjectDetectorOptions.Builder()
    .setDetectorMode(ObjectDetectorOptions.STREAM_MODE)
    .enableClassification()
    .enableMultipleObjects()
    .build()

val detector = ObjectDetection.getClient(options)

detector.process(inputImage)
    .addOnSuccessListener { objects ->
        objects.forEach { obj ->
            val box = obj.boundingBox
            val trackingId = obj.trackingId
            val category = obj.classificationCategory()
            drawBoundingBox(box, trackingId, category)
        }
    }

Object Tracking: after detecting an object in the first frame, ML Kit tracks it through the video stream without re-detection (based on optical flow + feature tracking). This reduces CPU/GPU load: initial detection 30–60 ms, subsequent tracking frames 2–5 ms. If the object leaves the frame or rotates > 30°, the tracker resets and requires re-detection. TrackingId persists while the object is in frame. For counting unique objects, use trackingId as an identifier.

YOLO: You Only Look Once on Mobile Devices

YOLOv8-Nano — the smallest version of YOLOv8 (Ultralytics) for edge devices. 2.6M parameters, 5.9 MB (FP16), 42% mAP@0.5:0.95 on COCO. YOLOv8 uses anchor-free detection + C2f backbone + TaskAlignedAssigner for prediction matching. For mobile development, export is available to TFLite (INT8 — 2.0 MB, latency 8–20 ms) and Core ML (4.5 MB, latency 5–15 ms on iPhone 15 Pro). YOLOv8-Nano is the best choice for on-device real-time Object Detection.

YOLOv8 Export to TFLite: Ultralytics provides CLI: yolo export model=yolov8n.pt format=tflite int8. The output is a TFLite INT8 model optimized for GPU Delegate via NNAPI. For custom datasets (your own classes, not COCO) — training via Ultralytics HUB on 50–500 images per class. RT-DETR (Real-Time Detection Transformer) — an alternative based on Transformer architecture, 48% mAP at 60 FPS on NVIDIA Jetson, but for mobile devices it still lags behind YOLOv8-Nano in speed.

# YOLOv8 export to TFLite
from ultralytics import YOLO

model = YOLO("yolov8n.pt")
model.export(format="tflite", int8=True, imgsz=320)

# Inference with TFLite on Android
val interpreter = Interpreter(loadFile("yolov8n_int8.tflite"))
val output = Array(1) { Array(8400) { FloatArray(6) } }
interpreter.run(inputBuffer, output)

YOLO vs ML Kit on Mobile Devices: ML Kit Object Detection is easier to integrate (10 lines of code), requires no ML expertise, but is limited to standard classes (fashion, food, home, places). YOLOv8-Nano requires custom export but provides full control: any classes, input size, architecture. For rapid prototyping — ML Kit. For production with non-standard objects — YOLOv8-Nano. For iOS: YOLOv8-Core ML via model export from Ultralytics + VNCoreMLRequest.

SSD and Other On-Device Architectures

SSD (Single Shot Multibox Detector) — a classic one-stage architecture for mobile devices. SSD MobileNetV2 — the de facto standard in 2020–2023: 22% mAP@0.5:0.95 on COCO, 15–40 ms on Pixel 6. SSD uses a feature pyramid (different MobileNet layers for detecting objects of various sizes) and default boxes (anchor boxes) for each feature map cell. Pros: maximum speed for its time. Cons: low accuracy on small objects (10–30 px).

EfficientDet-Lite — a family from Google (2020+), optimized for TFLite. EfficientDet-Lite0: 3.9M parameters, 35% mAP, 20–50 ms. EfficientDet-Lite2: 8.1M, 42% mAP, 40–80 ms. EfficientDet uses BiFPN (Bidirectional Feature Pyramid Network) for multi-scale feature fusion — this provides a 2–4% mAP gain without increasing latency. For mobile development, Google recommends EfficientDet-Lite as the primary detector for TFLite Task Vision.

MediaPipe Object Detector — a ready-made implementation based on EfficientDet-Lite as part of MediaPipe Tasks Vision. Provides a unified API for Android, iOS, Python, Web. MediaPipe Object Detector supports COCO classes (80 classes), custom models, streaming mode, and CPU/GPU delegates. For fast Object Detection integration in a cross-platform project, MediaPipe is the optimal choice: one code for all platforms.

// MediaPipe Object Detection
val options = ObjectDetectorOptions.Builder()
    .setBaseOptions(BaseOptions.builder()
        .setDelegate(BaseOptions.Delegate.GPU)
        .build())
    .setMaxResults(5)
    .setScoreThreshold(0.5f)
    .build()

val detector = ObjectDetector.createFromOptions(context, options)

val image = MPImage.fromBitmap(bitmap)
val result = detector.detect(image)
result.detections.forEach { detection ->
    val box = detection.boundingBox
    val category = detection.categories.first()
}

Choosing an Architecture for a Mobile Application: for > 30 FPS on mid-range devices — YOLOv8-Nano (INT8) or SSD MobileNetV2. For accuracy > 40% mAP — YOLOv8-Small (11.2M) or EfficientDet-Lite2 (8.1M). For cross-platform — MediaPipe Object Detector. For simplicity — ML Kit. For iOS-first — Core ML + YOLOv8 .mlpackage. For Android-first — TFLite Task Vision (ObjectDetector API). Training: Roboflow (dataset) + Ultralytics YOLOv8 (training) + export to TFLite/Core ML.

TensorFlow Lite Task Vision API

TFLite Task Vision ObjectDetector — a unified API from Google for running Object Detection models on Android and iOS. Task API automatically handles image preprocessing (scaling, normalization, color space conversion) and post-processing (NMS, thresholding). The developer only needs to pass a .tflite model and receive a list of Detections with bounding box + category + score. Task API supports COCO-compatible models (80 classes).

Converting a Custom Model to Task API: the TFLite model must have input [1, height, width, 3] (float32/uint8) and output: detection_boxes[1,N,4], detection_classes[1,N], detection_scores[1,N], num_detections[1]. Export from TensorFlow Object Detection API with post-processing ops included (SSD Postprocess). For YOLOv8 — TFLite export with NMS via ops-compat. Task API on iOS uses Core ML Delegate for acceleration on ANE.

// TFLite Task Vision Object Detector
val options = ObjectDetectorOptions.Builder()
    .setScoreThreshold(0.5f)
    .setMaxResults(10)
    .setDelegate(Delegate.GPU)
    .build()

val detector = ObjectDetector.createFromFileAndOptions(
    context, "custom_model.tflite", options
)

val image = TensorImage.fromBitmap(bitmap)
val results = detector.detect(image)
results.forEach { detection ->
    onObjectDetected(
        detection.boundingBox,
        detection.categories.first().label,
        detection.categories.first().score
    )
}

Task API Performance: GPU Delegate on Android provides 3–5x acceleration compared to CPU (XNNPACK). NNAPI Delegate — another 1.5–2x on devices with NPU (Pixel 8, Snapdragon 8 Gen 3, Dimensity 9300). Hexagon, DSP — up to 10x on DSP accelerators. For iOS, Core ML Delegate — 4–6x vs CPU. Task API automatically selects the available hardware accelerator, but you can force a delegate via DetectorOptions.

Object Detection Use Cases in Mobile Applications

People and Object Counting — retail analytics: detecting visitors in a store, queue counting, determining shelf stock levels. An Object Detection people counter in a frame helps estimate foot traffic and conversion. ML Kit Object Detector delivers up to 30 FPS — sufficient for real-time counting. For zone crossing — a combination of Object Detection + ByteTrack tracking. Counting accuracy: 92–95% in good lighting conditions.

Defect Detection in Manufacturing — Object Detection identifies defects on a conveyor: scratches, chips, cracks, incorrect assembly. A custom YOLOv8-Nano (INT8) model runs on an Android tablet connected to a USB camera. Latency: 20–40 ms per frame (25–50 FPS). For complex defects (micro-cracks) — increase input resolution to 640x640 (latency 40–80 ms). Training: Roboflow — dataset of 200–1000 defect images + Ultralytics YOLOv8.

Real-Time AR Object Marking — ARCore (Android) and ARKit (iOS) use Object Detection for recognizing flat surfaces, vertical planes, and 3D objects. ML Kit Object Detection + ARCore Scene Semantics provides object segmentation: wall, floor, ceiling, furniture. For custom AR marking (e.g., recognizing a specific sofa model and overlaying AR information) — YOLOv8-Nano + SceneKit/SceneForm. Real-time requirement: > 20 FPS.

// ARKit + Object Detection
let request = VNCoreMLRequest(model: objectDetector) { req, _ in
    guard let results = req.results as? [VNDetectedObjectObservation] else { return }
    for result in results where result.confidence > 0.6 {
        let rect = result.boundingBox
        let worldPos = arView.hitTest(rect.center)
        arView.addAnchor(ARAnchor(name: label, transform: worldPos))
    }
}

Medical Imaging — Object Detection for analyzing X-rays, MRIs, CT scans. Detection of tumors, fractures, pathologies on a doctor's mobile device. YOLOv8-Nano on an Android tablet detects lung nodules in 15–30 ms with 89% sensitivity and 93% specificity (NIH ChestX-ray14 data). Requirements: INT8 quantization (data privacy), high recall (missing a pathology is more dangerous than a false positive), confidence threshold < 0.4. On-device processing ensures medical data privacy.