Image Labeling: What It Is, Methods, and How It Works

What Is Image Labeling: Classification Basics

Image Labeling is a subtype of image classification where the model takes an image as input and returns probabilities for each class from the training set. Unlike object detection, Image Labeling does not determine the position of objects in the image — only their presence or absence. Unlike image segmentation, it does not outline object contours. Image Labeling answers the question “what is in the photo?”, not “where and what is in the photo?”.

Classifier architecture: a CNN backbone (MobileNet, ResNet, EfficientNet) extracts a feature map from the image, Global Average Pooling collapses spatial dimensions, and a fully connected layer (Dense) with Softmax activation outputs class probabilities. MobileNetV2 is the most popular backbone for mobile devices: 3.5M parameters, 0.5–2 ms inference on Pixel 6 via GPU. EfficientNet-Lite is an alternative with a better accuracy/parameters ratio.

Top-1 vs Top-5 accuracy: top-1 — the model correctly predicts the primary class (91% for ML Kit); top-5 — the correct class is among the five most probable (98% for ML Kit). For production applications, use top-5 and show users multiple label options. For content moderation, use top-1 with a confidence threshold >= 0.8 (reduces false positive rate by 40%).

On-device vs Cloud: on-device classification (ML Kit, Core ML) delivers 1–20 ms latency with full data privacy. Cloud API (Google Cloud Vision, AWS Rekognition) has 500–2000 ms latency plus cost per request, but is more accurate (97–99%) due to larger models (ViT, CLIP). For real-time applications (camera, AR), choose on-device. For complex scenarios (medical, expert analysis), use cloud with on-device fallback.

ML Kit Image Labeling on Android and iOS

ML Kit Image Labeling is a ready-to-use library from Google for on-device image classification. Available in two modes: on-device (free, 400+ labels, 10–20 ms) and cloud (paid, 10000+ labels, 500+ ms). The on-device version uses MobileNetV3 with INT8 quantization, occupying 4 MB of disk space. ML Kit automatically detects image orientation and optimizes size before classification.

ML Kit API: InputImage (from Bitmap, media.Image, filePath) → ImageLabeler → OnSuccessListener with a list of ImageLabel (label, confidence, index). The confidence threshold (default 0.5) is the minimum confidence to return a label. Raising the threshold to 0.7 reduces the number of labels per frame but increases each label’s precision. For content moderation, set the threshold to 0.8.

val labeler = ImageLabeling.getClient(
    ImageLabelerOptions.DEFAULT_OPTIONS
)

labeler.process(inputImage)
    .addOnSuccessListener { labels ->
        labels
            .filter { it.confidence >= 0.7f }
            .forEach { label ->
                log("Label: ${label.label}")
                log("Confidence: ${label.confidence}")
            }
    }
    .addOnFailureListener { error -> handleError(error) }

ML Kit on iOS: the API is similar to Android, using UIImage or CMSampleBuffer. ML Kit automatically utilizes Core ML + ANE on A12+ devices. For iOS 16+, ML Kit Image Labeling delivers 8–15 ms latency on iPhone 15 Pro. To minimize latency, pass the smallest possible image resolution (224x224 for MobileNet) — ML Kit scales it automatically, but converting large images adds 2–5 ms overhead.

Core ML and Vision Framework on iOS

Core ML is Apple’s framework for running ML models on-device. Paired with Vision Framework (VNCoreMLRequest), Core ML enables image classification with minimal code. Apple provides built-in models: SqueezeNet (5 MB, 80.5% top-1), ResNet50 (98 MB, 95.2%), MobileNetV2 (14 MB, 90.2%). Models in .mlmodel or .mlpackage format are converted via coremltools from TensorFlow or PyTorch.

VNCoreMLRequest is a Vision API that handles image preprocessing (scaling, crop-to-fill, color space conversion) and passes the result to the model. Vision returns VNClassificationObservation with identifier (class name) and confidence (0..1). MaxCandidates is the maximum number of returned labels (default 1). For auto-selection classification, use VNCoreMLRequest with imageCropAndScaleOption = .centerCrop.

guard let model = try? VNCoreMLModel(for: MobileNetV2().model) else { return }
let request = VNCoreMLRequest(model: model) { request, _ in
    guard let results = request.results as? [VNClassificationObservation] else { return }
    results.prefix(5).forEach { obs in
        print("Label: \(obs.identifier), Confidence: \(obs.confidence)")
    }
}
request.imageCropAndScaleOption = .centerCrop

let handler = VNImageRequestHandler(cgImage: cgImage, options: [:])
try handler.perform([request])

Core ML vs ML Kit on iOS: Core ML is native, requires no additional SDKs, and is better optimized for ANE (Apple Neural Engine). ML Kit is more accurate on complex scenes thanks to Google’s models. Core ML supports any model (converted via coremltools), while ML Kit only supports its own. For quick implementation, use ML Kit. For maximum model control, use Core ML. A practical approach: use both frameworks in one app — ML Kit for standard labels, Core ML for custom ones.

Training and Integrating Custom Models

Custom Image Labeling models involve training your own classifier for a specific task: recognizing fruit quality, detecting defects on a production line, classifying dog breeds. The process includes dataset collection (1000+ images per class), annotation, training via transfer learning on a pre-trained MobileNetV2 or EfficientNet, and conversion to TFLite / Core ML.

Transfer Learning is the primary method for training mobile classifiers. You take a model pre-trained on ImageNet, freeze the lower layers (CNN backbone), and retrain the upper layers (fine-tuning). This requires only 100–500 images per class and takes 1–2 hours on Google Colab (GPU). Libraries: TensorFlow Keras (Android), PyTorch + coremltools (iOS). Result: 95–98% accuracy on target classes.

// TFLite custom model inference
val interpreter = Interpreter(loadModelFile(context))
val inputImage = TensorImage.fromBitmap(bitmap)
    .apply { load(Bitmap.createScaledBitmap(bitmap, 224, 224, true)) }

val output = Array(1) { FloatArray(NUM_CLASSES) }
interpreter.run(inputImage.tensorBuffer, output)

val probabilities = TensorLabel.mapIndexToLabels(output[0])
val topClass = probabilities.maxByOrNull { it.value }

ML Kit Custom Model API is an alternative: no need to write TFLite Interpreter code — ML Kit loads the model and handles preprocessing automatically. The Custom Image Labeler accepts a TFLite model with input size 224x224x3 and output score float[NUM_CLASSES]. Simply provide the model file and get standard labels. ML Kit Custom Model API is recommended if the model matches the TFLite format. If you need finer control over inference (grippers, per-class thresholds), use the TFLite Interpreter directly.

TFLite and Core ML: Format Comparison

TFLite (TensorFlow Lite) is Google’s model format for mobile and edge devices. It supports quantization (FP16, INT8), which reduces model size by 4x and increases speed by 2–3x with a slight accuracy loss (1–2%). TFLite runs on Android via GPU Delegate (OpenGL/OpenCL) and on iOS via Core ML Delegate. TFLite Task API provides a unified interface for Image Classifier, Object Detector, and other tasks.

Core ML is Apple’s format (.mlpackage — new, .mlmodel — legacy). It supports quantization (FP16) and weight palettization (down to 1/2/4/6/8 bits), achieving up to 80% model compression. Core ML uses ANE (Neural Engine), GPU, and CPU — the system automatically selects the optimal engine. Conversion from PyTorch via torch.onnx.export + coremltools, from TensorFlow via tf-keras + coremltools. iOS 18+ supports on-device training via Core ML Training API.

ONNX as an intermediate format: convert models to ONNX (PyTorch → ONNX, TensorFlow → ONNX) and then to TFLite / Core ML via onnx2tf or onnx2coreml. ONNX is a unified format supported by 70+ frameworks. This simplifies the pipeline: one trained model → ONNX → native formats for both platforms. Training in PyTorch + conversion to ONNX → TFLite (Android) / Core ML (iOS) is the recommended pipeline for cross-platform projects.

Image Labeling Applications in Mobile Apps

Auto-tagging photos — gallery apps (Google Photos, Apple Photos) automatically assign labels to images: “beach”, “sunset”, “dog”, “food”. ML Kit Image Labeling processes each photo from the gallery in the background — 1–2 seconds for 100 photos (batch). Users can search by labels without manual sorting. Google Photos uses on-device classification with subsequent server verification for complex scenes.

Content moderation — automatic detection of inappropriate images in user-generated content (social networks, marketplaces, UGC platforms). ML Kit Custom Model detects NSFW content, violence, and propaganda with 92–96% accuracy. The trigger threshold is confidence 0.8. To reduce false positives (legitimate medical images), use a classifier committee: Image Labeling + Object Detection + Blur Detection. On-device moderation is faster and protects user privacy.

Visual product search — a user photographs a product (sneakers, bag, furniture), the app determines the label and finds similar products in the catalog. Image Labeling narrows the search to a category, then feature descriptors (feature vectors) find visually similar items. Pinterest Lens, Google Lens, and Amazon StyleSnap use this approach. On-device classification delivers results in 10–30 ms, cloud-based product search in 200–500 ms.

// Image Labeling with Core ML for visual search
let request = VNCoreMLRequest(model: productClassifier) { req, _ in
    guard let result = req.results?.first as? VNClassificationObservation,
          result.confidence > 0.6 else { return }
    SearchService.findSimilarProducts(
        category: result.identifier,
        image: capturedPhoto,
        limit: 10
    ) { products in
        DispatchQueue.main.async {
            self.showResults(products)
        }
    }
}

AR and educational apps — Image Labeling classifies objects in real time through the camera. A user points their phone at a plant — the app shows the name, care instructions, and watering schedule. Points at a mechanical part — it explains its purpose. Requirement: latency < 30 ms. EfficientNet-Lite on flagship devices delivers 15–25 ms. In low light, accuracy drops — use an adaptive confidence threshold (lower the threshold in low-light conditions to compensate for degraded input quality).