Face Detection in Apps: What It Is, Methods and Libraries

What is Face Detection: Principles and Algorithms

Face Detection is a computer vision task of detecting the presence and location of human faces in an image. The algorithm returns a bounding box (rectangle around the face) and a confidence score (probability that it is a face). Modern algorithms also return facial landmarks — key points (eyes, eyebrows, nose, mouth, face contour). Face Detection is the first step for many ML tasks: person recognition, AR filters, attention analytics.

History of algorithms began with Viola-Jones (2001) — a cascade of Haar-like features on CPU. In the 2010s, HOG + SVM (Histogram of Oriented Gradients) dominated. Since 2016, all modern algorithms are based on convolutional neural networks (CNN): MTCNN, RetinaFace, SSH, MobileNet-SSD, YOLO-Face. CNN algorithms on GPU provide 95–99% accuracy compared to 85–90% for classical methods. According to the WiderFace benchmark (2025), RetinaFace achieves 96.3% mAP on the hardest subset.

MTCNN (Multi-Task Cascaded CNN) is a classic three-stage detector: P-Net (Proposal Network) finds candidates, R-Net (Refine Network) filters them, O-Net (Output Network) refines the bounding box and outputs landmarks. MTCNN runs on CPU at 30–50 ms per frame at 640x480 resolution. For mobile devices, MTCNN offers an optimal balance of speed and accuracy without GPU.

Real-time Face Detection requires 30+ FPS for video. This is achievable on modern smartphones thanks to NPU (Neural Processing Unit) — Qualcomm Hexagon, Apple Neural Engine, MediaTek APU. NPU performs detection in 2–5 ms with power consumption of 50–100 mW, while GPU consumes 500–2000 mW. For always-on detection (camera always active), NPU is the only practical option without device overheating.

ML Kit Face Detection on Android and iOS

ML Kit Face Detection is the most popular SDK for face detection on mobile devices. ML Kit offers two modes: contour mode (468 face contour points for precise mask overlays) and classification mode (emotion detection: smile, eyes open, mouth open). Modes are combined in FaceDetectorOptions. ML Kit uses Google's MobileNetV2-SSD neural network with optimization via NNAPI/GPU Delegate.

ML Kit Face Detection configuration: setPerformanceMode(FACE_DETECTION_FAST / ACCURATE), setLandmarkMode (key points), setContourMode (contour points), setClassificationMode (emotions). For maximum speed use FAST + LANDMARKS_ONLY + no contour. For AR filters — ACCURATE + ALL_CONTOURS. According to Google (2025), FAST mode gives 98% accuracy at 5 ms, ACCURATE — 99% at 15 ms.

// ML Kit Face Detection with contours
val options = FaceDetectorOptions.Builder()
    .setContourMode(FaceDetectorOptions.ContourMode.ALL)
    .setClassificationMode(FaceDetectorOptions.ClassificationMode.ALL)
    .setPerformanceMode(FaceDetectorOptions.PerformanceMode.FAST)
    .enableTracking()
    .build()
val detector = FaceDetection.getClient(options)

detector.process(inputImage)
    .addOnSuccessListener { faces ->
        faces.forEach { face ->
            val trackingId = face.trackingId
            val smile = face.smilingProbability
            val leftEyeOpen = face.leftEyeOpenProbability
        }
    }

Face Tracking in ML Kit — a unique feature for video streams. Each detected face is assigned a tracking ID (integer) that persists across frames. This allows AR objects to be attached to a specific face without re-detection. The tracker uses Optical Flow and retains the ID even with partial face occlusion. Tracking ID resets when the face leaves the frame for more than 1 second.

Apple Vision Framework: VNDetectFaceRectanglesRequest

Apple Vision Framework is a native iOS framework for Face Detection that runs through Core ML on Apple Neural Engine. Vision provides VNDetectFaceRectanglesRequest (bounding box only) and VNDetectFaceLandmarksRequest (with contour points). Vision Framework has been built into iOS since iOS 11 and requires no additional SDKs — it is part of Foundation.

Advantages of Vision Framework: zero dependency on third-party libraries, runs through Apple Neural Engine (ANE) on A12+ chips, automatic image orientation handling via exifOrientation, support for CIDetectorAccuracy for speed/accuracy tuning. Vision returns VNFaceObservation with bounding box (normalized coordinates 0..1) and landmarks (VNFaceLandmarkRegion2D).

Vision vs ML Kit on iOS: Vision is faster on older devices (A12–A15) as it uses native Apple neural engines. ML Kit uses Google models and may be more accurate on complex angles (profile, strong tilt). For simple detection (camera, photos) — use Vision. For AR filters and contour masks — use ML Kit (468 points vs 76 points in Vision).

import Vision

let request = VNDetectFaceRectanglesRequest { request, error in
    guard let observations = request.results as? [VNFaceObservation] else { return }
    for face in observations {
        let rect = face.boundingBox // normalized 0..1
        let confidence = face.confidence
    }
}

let handler = VNImageRequestHandler(
    cgImage: cgImage, orientation: .up, options: [:]
)
try handler.perform([request])

VNDetectFaceLandmarksRequest is the extended version for key points. Returns VNFaceLandmarkRegion2D for each point group: leftEye, rightEye, nose, faceContour, innerLips, outerLips. Each group is an array of CGPoint (normalized). Vision does not return 468 points like ML Kit — only 76 key points. For precise face masks, ML Kit is preferable; for basic detection, Vision suffices.

OpenCV Haar Cascade: The Classic Approach

OpenCV Haar Cascade is a classic Face Detection algorithm based on a cascade of Haar-like features (Viola-Jones, 2001). Despite its age, Haar Cascade is still used in resource-constrained projects: Raspberry Pi, IoT devices, embedded systems. The algorithm does not require GPU or NPU — it runs on any CPU at 5–15 ms per VGA-resolution frame.

LBP Cascade (Local Binary Patterns) is an alternative to Haar Cascade in OpenCV. LBP is faster (2–5 ms) and less sensitive to lighting, but produces more false positives. Haar is more accurate (85–90% vs 80–85% for LBP) but sensitive to glare and shadows. For mobile apps, OpenCV Face Detection is inferior to neural network methods in accuracy but wins in compatibility — it works on any device.

// OpenCV Face Detection via CascadeClassifier
val cascadeFile = File(context.filesDir, "haarcascade_frontalface_default.xml")
val faceDetector = CascadeClassifier(cascadeFile.absolutePath)

val gray = Mat()
Imgproc.cvtColor(colorFrame, gray, Imgproc.COLOR_RGBA2GRAY)
val faces = MatOfRect()
faceDetector.detectMultiScale(gray, faces)

faces.toList().forEach { rect ->
    // rect = face bounding box
}

OpenCV limitations: high false positive rate (up to 15%), poor performance on profile angles (>30° — accuracy drops to 50%), no landmarks without an additional model, no inter-frame tracking. For modern mobile apps, OpenCV Face Detection serves as a fallback for devices without Google Play Services (Huawei, Amazon Fire). For primary scenarios — use ML Kit or Vision.

Face Detection vs Face Recognition: The Difference

Face Detection — determining the presence and position of a face in an image. Result: bounding box and key points. Face Recognition — identifying a person by their face: determining that this face belongs to a specific individual. Face Recognition uses embeddings (a vector of numbers representing a face) and compares them against a database of known faces. Face Detection is a mandatory first step for Face Recognition.

Face Recognition technologies: FaceNet (Google, 2015) — triplet loss for training 128–512 float-value embeddings, ArcFace (2019) — additive angular margin loss, best accuracy (99.83% on LFW). For mobile apps, Face Recognition requires a custom TFLite model — ML Kit does not provide a ready API for person identification (only detection).

Privacy on mobile devices: Face Detection is safe — it does not store user data. Face Recognition requires storing embeddings or photos for comparison. Apple requires explicit user consent for Face Recognition and prohibits its use for advertising. Google ML Kit intentionally does not include Face Recognition to avoid privacy risks. For detection without identification — there are no restrictions.

Face Liveness Detection — an additional check to ensure the face is real (not a photo/video). Uses eye movement analysis (blink detection), micro-expressions, depth map (3D camera). Liveness Detection is mandatory for banking and payment applications. ML Kit does not have built-in liveness — use a custom model or Google Play Integrity API for verification.

Face Detection Use Cases in Mobile Apps

AR filters and masks — the most popular application of Face Detection. Based on facial contour points (468 points), 3D masks, hats, glasses, mustaches are overlaid. ML Kit Face Detection + SceneKit (iOS) or Filament (Android) creates a real-time AR experience. Snapchat and Instagram use their own detectors based on MobileNet + MTCNN. For custom AR filters, ML Kit and a 3D engine are sufficient.

Photo editors and retouching — Face Detection identifies the face area for automatic correction: skin color balancing, blemish removal, eye and teeth enhancement. OpenCV + ML Kit Face Detection provides coordinates for Selective Gaussian Blur (skin smoothing) and eye contrast. Real-world applications — Facetune, BeautyPlus, YouCam Makeup.

Attention monitoring — determining gaze direction (gaze detection). Face Detection returns bounding box and eye landmarks. The pupil position relative to the eye determines where the user is looking: at the screen, left, right, up. Used in e-learning (monitoring whether students are watching the lecture), advertising (Attention metrics), driver assistance (fatigue monitoring).

// ARKit + Vision for AR filter
let faceLandmarksRequest = VNDetectFaceLandmarksRequest { req, _ in
    guard let face = req.results?.first as? VNFaceObservation else { return }
    if let leftEye = face.landmarks?.leftEye {
        // AR object overlay on left eye
    }
}

let handler = VNSequenceRequestHandler()
for pixelBuffer in videoStream {
    try handler.perform([faceLandmarksRequest], on: pixelBuffer)
}

Medicine and wellness — Face Detection is used for facial asymmetry analysis (diagnosing neurological disorders), pulse estimation from skin micro-vibrations (photoplethysmography via camera), skin hydration assessment. ML Kit Face Detection + OpenCV provide sufficient accuracy for preliminary screening without medical certification. For production medicine, FDA/CE certification is required.