Pose Detection: What It Is, Models and How It Works

What is Pose Detection: Basics and Metrics

Pose Detection (also Pose Estimation) refers to the task of determining semantic key points of the human body from an image or video. Top-down approach first detects the person (Object Detection), then determines their pose. Bottom-up approach finds all landmarks in the image, then groups them by person (OpenPose). For mobile devices, bottom-up is used — it is faster with multiple people in the frame. ML Kit and BlazePose use a single-stage approach — detection + pose in one pass.

COCO Keypoints — a standard set of 17 points: nose, eyes (2), ears (2), shoulders (2), elbows (2), wrists (2), hips (2), knees (2), ankles (2). MediaPipe BlazePose uses 33 points, adding: toes, heels, torso center, face points. The more points, the more accurate the pose analysis, but higher the computational load. For fitness apps, 17–20 points are sufficient. For full analysis (yoga, dancing) — 33 points. For AR filters — 33 points + 468 face points (MediaPipe Face Mesh).

PCK (Percentage of Correct Keypoints) — Pose Detection accuracy metric. It calculates the proportion of points predicted within a distance from ground truth (usually 0.05–0.15 of the person's bounding box size). ML Kit Pose Detection: 94% PCK@0.1. BlazePose Full: 96% PCK@0.1. MoveNet Lightning: 91% PCK@0.1. OKS (Object Keypoint Similarity) — a metric used in COCO that accounts for person scale and point difficulty. mAP@OKS — standard metric for benchmarks.

Keypoint Visibility: each landmark has a score (0..1) indicating how confident the model is about the point and whether it is visible. Points with score < 0.5 are considered invisible (occluded, out of frame). For pose analysis, use only visible points. For alignment assessment — calculate confidence-weighted distances, where points with low score have less weight in the metric.

ML Kit Pose Detection on Android and iOS

ML Kit Pose Detection — a library from Google for on-device pose detection. Supports 33 landmarks, including face points, toes and heels. Modes: Accurate Mode (14 MB model, 15–30 ms, high accuracy) and Streaming Mode (5 MB, 5–10 ms, for real-time). Streaming Mode uses a lightweight model with tracking — after the first frame, it tracks movement without re-detecting exact positions, achieving up to 60 FPS.

ML Kit Pose Detection API: PoseDetector (options: setDetectorMode, setPerformanceMode) → InputImage → Pose (List<PoseLandmark>). Each PoseLandmark has: landmarkType (38 types), position (PointF3D), inFrameLikelihood (0..1). Pose also provides: person's boundingBox, list of landmarks, getLandmark(type) method. For pose classification, use angles between bones: shoulderAngle, elbowAngle, hipAngle — calculated via Vector3D between adjacent landmarks.

val options = PoseDetectorOptions.Builder()
    .setDetectorMode(PoseDetectorOptions.STREAM_MODE)
    .setPerformanceMode(PoseDetectorOptions.PERFORMANCE_MODE_FAST)
    .build()

val detector = PoseDetection.getClient(options)

detector.process(inputImage)
    .addOnSuccessListener { pose ->
        val leftElbow = pose.getLandmark(PoseLandmark.LEFT_ELBOW)
        val leftShoulder = pose.getLandmark(PoseLandmark.LEFT_SHOULDER)
        val leftWrist = pose.getLandmark(PoseLandmark.LEFT_WRIST)
        val elbowAngle = calculateAngle(
            leftShoulder.position, leftElbow.position, leftWrist.position
        )
    }

ML Kit on iOS: Pose Detection is available via ML Kit CocoaPods. API is identical to Android: PoseDetector → UIImage → Pose → PoseLandmark. On iOS, ML Kit uses Core ML and ANE (A12+), achieving 10–20 ms latency in Accurate Mode on iPhone 15 Pro. Streaming Mode — 3–8 ms, up to 120 FPS. For iOS, ML Kit Pose Detection is faster than MediaPipe BlazePose (Core ML acceleration), but falls behind BlazePose in FPS on older devices (iPhone X–11).

MediaPipe BlazePose: Architecture and 3D

MediaPipe BlazePose — a high-performance Pose Detection model from Google Research. Uses a hybrid architecture: detector-decoder pipeline based on MobileNetV2 + Feature Pyramid Network. BlazePose Full: 33 landmarks + 3D coordinates (depth z). BlazePose Lite: 33 landmarks (2D) without depth. Both models are available in MediaPipe Tasks Vision on Android, iOS, Python and Web. BlazePose Full processes 30–60 FPS on GPU, Lite — 60+ FPS.

3D Pose Estimation with BlazePose: the model predicts z-coordinate for each landmark, allowing depth estimation (limb approach/retreat) without a stereo camera. Z-coordinate is calculated in metric space (meters) relative to the camera. BlazePose 3D accuracy: 37 mm MPJPE (Mean Per Joint Position Error) on public benchmarks — sufficient for fitness and AR, insufficient for medical diagnostics. For ARKit/ARFoundation, BlazePose 3D provides natural depth.

// MediaPipe Pose Detection Tasks Vision
val options = PoseLandmarkerOptions.Builder()
    .setBaseOptions(BaseOptions.builder()
        .setModelAssetPath("pose_landmarker_full.task")
        .build())
    .setDelegate(Delegate.GPU)
    .build()

val landmaker = PoseLandmarker.createFromOptions(context, options)

val result = landmaker.detect(MPImage.fromBitmap(bitmap))
result.landmarks.forEach { pose ->
    pose.forEach { landmark ->
        drawLandmark(landmark.x, landmark.y, landmark.z)
    }
}

BlazePose vs ML Kit Pose Detection: BlazePose is faster (60+ FPS vs 30 FPS), supports 3D coordinates, works cross-platform via MediaPipe. ML Kit is easier to integrate (no MediaPipe SDK required), includes pre-trained models with high accuracy. For iOS-native projects — ML Kit Pose Detection (best ANE optimization). For cross-platform projects (Flutter, React Native) — MediaPipe BlazePose (unified model for all platforms). For accuracy in demanding scenes — both models are comparable.

MoveNet: Lightning and Thunder by TensorFlow

MoveNet — a family of Pose Detection models from TensorFlow, optimized for TFLite. Lightning (8 MB, INT8 — 4 MB): 17 COCO keypoints, 91% PCK, 8–15 ms latency, 30+ FPS. Thunder (13 MB, INT8 — 6 MB): 17 keypoints, 95% PCK, 25–40 ms latency, 20+ FPS. MoveNet uses CenterNet architecture + bilinear interpolation for high-resolution heatmap. MoveNet supports multi-pose detection (up to 6 people). Models are available on TensorFlow Hub.

MoveNet Lightning vs Thunder: Lightning — for real-time applications with high FPS (fitness, AR filters). Thunder — for accurate pose analysis (rehabilitation, sports analytics) on devices with GPU. Both models convert to Core ML via tf-coreml for iOS. MoveNet is also available through TFLite Task Vision PoseLandmarker API. Recommendation: start with Lightning, if accuracy is insufficient — switch to Thunder (only +15 ms latency overhead).

// MoveNet Inference with TFLite
Interpreter interpreter = new Interpreter(loadModel(context));
TensorImage image = TensorImage.fromBitmap(bitmap);
image.load(Bitmap.createScaledBitmap(bitmap, 192, 192, true));

float[][] output = new float[1][51];
interpreter.run(image.getTensorBuffer(), output);

float[] keypoints = output[0];
for (int i = 0; i < 17; i++) {
    float y = keypoints[i * 3];
    float x = keypoints[i * 3 + 1];
    float score = keypoints[i * 3 + 2];
    drawKeypoint(x, y, score);
}

MoveNet Multi-Pose: detection of up to 6 people in the frame. Instead of a standard heatmap approach, MoveNet uses person detection + pose refinement: first detects people (centroid-based), then estimates each pose. Multi-pose mode is 2–3x slower than single-pose: Lightning — 25–50 ms (6 people). For fitness apps with a single user, use single-pose. For group activities (yoga, crossfit) — multi-pose with frame rate limiting to save battery.

Pose Classification: From Points to Actions

Pose Classification — the stage after detection: converting landmarks into semantic actions (standing, sitting, raised hand, squatting). Approaches: Rule-based (manual rules — angles between bones), ML-based (logistic regression or Random Forest on landmark vector), DL-based (LSTM classifier of pose sequence across frames). Rule-based — 50–80% accuracy, simple and fast. ML-based — 85–95% accuracy with 200+ labeled examples. LSTM — 90–98% accuracy on time series (video).

Angles between bones: for each point, angles with neighboring points are calculated. Examples: right elbow angle (shoulder → elbow → wrist), right knee angle (hip → knee → ankle), torso angle (shoulders midpoint → hips midpoint). Angles are invariant to person scale and screen position. Normalizing angles to the [0, 1] range allows pose classification with a simple threshold rule: if elbowAngle < 30° — arm is bent, if > 150° — extended.

// Rule-based squat classifier
fun classifySquat(pose: Pose): SquatPhase {
    val kneeAngle = angle(
        pose.getLandmark(PoseLandmark.LEFT_HIP),
        pose.getLandmark(PoseLandmark.LEFT_KNEE),
        pose.getLandmark(PoseLandmark.LEFT_ANKLE)
    )
    return when {
        kneeAngle > 140f -> SquatPhase.STANDING
        kneeAngle < 90f  -> SquatPhase.SQUAT
        else           -> SquatPhase.TRANSITION
    }
}

MediaPipe Pose Classification — pre-trained classifiers included in MediaPipe Tasks: squats, push-ups, plank, leg raises. The classifier takes a list of landmarks and returns the action + confidence. Includes temporal smoothing (temporal filter): HED (Holt Exponential Double Smoothing) for smooth transitions between poses. For custom actions — train a classifier on 100–500 examples using MediaPipe Model Maker (TensorFlow Lite + metadata).

Pose Detection Applications in Fitness and Rehabilitation

Fitness trackers with technique correction — the app analyzes the user's pose during exercise and provides voice prompts: "lower your hips", "don't lean forward". ML Kit Pose Detection + rule-based classifier counts repetitions and evaluates quality. Squat: knee angle < 90° — correct squat, back angle < 45° — dangerous torso lean. Push-up: elbow angle < 90° at the bottom — full push-up. Pull-up: chin above the bar level.

Rehabilitation and physiotherapy — Pose Detection is used for remote monitoring of physical therapy exercises. A doctor sets a reference pose (e.g., shoulder abduction at 45°), the app measures the current angle and deviations. BlazePose 3D provides angle accuracy of ±3° — sufficient for clinical assessment. Frequency: 1 Frame per 3–5 seconds (battery saving). On-device — privacy of patient medical data. Post-stroke rehabilitation: tracking hand-to-shoulder, elbow-extension, hip-flexion.

AR filters and effects — Pose Detection powers AR filters in social networks (TikTok, Instagram, Snapchat). Landmarks of the head, shoulders and hands are used for overlaying masks, animations, decorative elements. MediaPipe BlazePose Full + Face Mesh (468 points) delivers 120+ FPS on flagship devices. ARKit/ARCore Face Tracking + Pose Detection — a combination for full-body AR. Requirements: FPS > 30, motion-to-photon latency < 20 ms.

// AR filter with pose landmarks
let request = VNDetectHumanBodyPoseRequest { req, _ in
    guard let observation = req.results?.first as? VNHumanBodyPoseObservation else { return }
    let keypoints = try observation.recognizedPoints(.all)
    let rightShoulder = keypoints[VNHumanBodyPoseObservation.JointName.rightShoulder]
    DispatchQueue.main.async {
        arView.placeFilter(at: rightShoulder?.location ?? .zero)
    }
}

Sports analytics — professional technique assessment: biomechanical running analysis (cadence, stride length, arm swing symmetry), swimming (body roll, elbow angle during stroke), tennis (shoulder rotation, wrist snap). MoveNet Thunder with post-processing provides sufficient accuracy for non-professional analytics. Professional sports require 3D Motion Capture (Vicon, OptiTrack), but phone-based Pose Detection is an affordable alternative for the mass market. Angle synchronization between frames is a key component.