Vision: an iOS computer vision framework
Vision is a computer vision framework from Apple, first introduced in iOS 11 in 2017. Vision provides ready-to-use algorithms for face detection, text recognition (OCR), barcode detection, object tracking, image classification, and contour analysis — all performed on-device using the Neural Engine. According to Apple WWDC 2023, Vision is used in the standard Photos app for face recognition and in the Camera app for real-time text detection on iPhone and iPad.
Key Takeaways
- Vision is Apple’s on-device computer vision framework with a full analysis cycle on the device.
- Supports face detection, text recognition (OCR), barcodes, classification, and object tracking.
- Works through a unified VNRequest mechanism — requests to an image or video stream.
- Integrates with Core ML for custom models via VNCoreMLRequest.
- The framework is optimized for Neural Engine on A12+ chips for real-time performance.
What is Vision in iOS?
Vision is a high-level computer vision framework that abstracts complex machine learning algorithms behind a simple request API (VNRequest). Developers don’t need to understand convolutional neural networks — just create the appropriate request type, pass an image, and get the result.
The Vision architecture follows the Request → Handler → Result principle: VNImageRequestHandler accepts an image, executes VNRequest, and returns an array of VNObservation with results. This allows combining multiple requests on a single image — for example, simultaneously detecting faces and text in a photo.
Vision supports iOS 11+ and macOS 10.13+. Hardware acceleration via Neural Engine requires an A12 Bionic chip or newer. On A17 Pro and M-series devices, Vision processes up to 60 frames per second for streaming video.
Key advantage of Vision is complete privacy: all computation is performed on-device, images are never sent to a server. This makes the framework suitable for applications handling sensitive data, such as medical or banking apps.
Face Detection and Recognition
VNDetectFaceRectanglesRequest is the basic Vision request for detecting faces in an image. It returns the coordinates of rectangles bounding each face, without identifying a specific person.
For more detailed analysis, use VNDetectFaceLandmarksRequest, which identifies key facial points: eyes, eyebrows, nose, mouth, jaw contour. This data is used for applying masks, animating emojis, and real-time filters (as in FaceTime and Snapchat).
swift
import Vision
import UIKit
guard let image = UIImage(named: "photo") else { return }
let request = VNDetectFaceRectanglesRequest()
let handler = VNImageRequestHandler(cgImage: image.cgImage!, options: [:])
try handler.perform([request])
for observation in request.results ?? [] {
let bbox = observation.boundingBox
print("Face at x=\\(bbox.origin.x), y=\\(bbox.origin.y)")
}VNFaceObservation contains not only boundingBox but also confidence (0 to 1), and landmarks — an array of face points if the request was VNDetectFaceLandmarksRequest. Confidence below 0.5 usually indicates a false positive — such results should be discarded.
Vision also supports VNDetectFaceCaptureQualityRequest, which evaluates face image quality: lighting, sharpness, rotation angle. This is useful for selecting the best frame during user registration or creating an avatar.
Text Recognition (OCR) with Vision
VNRecognizeTextRequest is Apple’s built-in OCR engine, introduced in iOS 13. It recognizes printed text in images with support for 13 languages: English, Russian, Chinese, Japanese, Korean, Italian, German, Spanish, Portuguese, French, Arabic, Vietnamese, and Thai.
VNRecognizeTextRequest supports two accuracy levels: .fast (quick, for simple text on a contrasting background) and .accurate (precise, for complex scenes with various fonts and angles). .fast is 3–5 times faster but handles handwritten text and non-standard fonts less effectively.
swift
import Vision
let textRequest = VNRecognizeTextRequest { request, _ in
guard let observations = request.results as? [VNRecognizedTextObservation]
else { return }
for observation in observations {
let topCandidate = observation.topCandidates(1).first
print(topCandidate?.string ?? "")
}
}
textRequest.recognitionLevel = .accurate
textRequest.usesLanguageCorrection = trueThe usesLanguageCorrection property enables recognized text correction using a language model. This improves English accuracy by 10–15% but increases processing time. Russian language correction is also available when the appropriate recognition is specified.
According to Apple ML Research 2022, VNRecognizeTextRequest at .accurate level achieves 96% accuracy for clear document photos in English and about 88% for Russian. For text on packaging, signs, and screens, accuracy drops to 75–85%.
| Recognition Level | Speed | Accuracy (English) | Russian Support |
|---|---|---|---|
| .fast | 0.1–0.3 sec | ~85% | Yes |
| .accurate | 0.3–1.0 sec | ~96% | Yes |
Barcode and QR Code Detection
VNDetectBarcodesRequest is a Vision request for detecting and decoding barcodes and QR codes in images. All major formats are supported: EAN-8, EAN-13, UPC-A, UPC-E, Code 39, Code 93, Code 128, PDF417, Aztec, and QR.
Unlike AVFoundation (AVCaptureMetadataOutput), Vision works not only with video streams but also with static images — allowing code scanning from existing photos or screenshots. Vision also determines the code’s boundingBox with pixel precision, which is convenient for animating a frame around the detected code.
swift
import Vision
let barcodeRequest = VNDetectBarcodesRequest { request, _ in
guard let observations = request.results as? [VNBarcodeObservation]
else { return }
for observation in observations {
let payload = observation.payloadStringValue ?? ""
print("Barcode: \\(payload), Symbology: \\(observation.symbology.rawValue)"
}
}VNBarcodeObservation contains payloadStringValue (decoded content), symbology (code type), and confidence. For QR codes, the payload can be a URL, text, or JSON. For EAN/UPC, a numeric product code is returned, which can be used to look up the item in a database.
Vision can process multiple barcodes in a single image — the observations array contains one object for each detected code. This is useful for scanning batches of products or documents with multiple barcodes.
Object Tracking in Video Stream
VNDetectObjectAtPointRequest and VNSequenceRequestHandler are Vision tools for tracking objects in a video stream. The first identifies an object by the user’s touch point, the second tracks an object between video frames.
VNSequenceRequestHandler accepts a sequence of images (video frames) and returns the updated position of the tracked object for each frame. This is used in augmented reality apps, video editors, and surveillance systems.
swift
import Vision
let trackingRequest = VNTrackObjectRequest(detectedObjectObservation: initialObservation)
let sequenceHandler = VNSequenceRequestHandler()
for frame in videoFrames {
try sequenceHandler.perform([trackingRequest],
on: frame.cgImage!)
let newBBox = trackingRequest.results?.first?.boundingBox
// Update object position on screen
}VNTrackObjectRequest uses an optical flow-based tracking algorithm — it does not retrain the detector on every frame but calculates the object’s displacement from its previous position. This enables real-time performance even on devices without Neural Engine.
Limitation: tracking may lose the object during fast movement, occlusion, or sudden lighting changes. Apple recommends restarting detection (VNDetectObjectAtPointRequest) every 10–15 frames for tracking correction.
Image Classification and Contour Analysis
VNClassifyImageRequest is Vision’s built-in image classification model covering 1,000 categories (ResNet-50 model trained on ImageNet). The request returns an array of VNClassificationObservation with the category name and confidence.
VNDetectContoursRequest performs image contour analysis — extracts object boundaries, useful for segmentation, outlining, and preprocessing before recognition. The request returns an array of points describing each contour in the image.
swift
import Vision
// Image classification
let classificationRequest = VNClassifyImageRequest { request, _ in
guard let results = request.results as? [VNClassificationObservation]
else { return }
let topMatch = results.prefix(3).filter { $0.confidence > 0.3 }
for match in topMatch {
print("\\(match.identifier): \\(match.confidence)"
}
}VNClassifyImageRequest works well for general categories (cat, dog, car, food) but is not suitable for specific objects — for those, you need to train a custom Core ML model and use VNCoreMLRequest. Apple’s model is optimized for mobile devices and takes only 5 MB.
VNDetectContoursRequest returns contours as an array of points with the contour type (external, internal, hole). This request is used for preprocessing images before OCR (improving text contrast), for drawing effects (object outlining), and for shape analysis.
| Vision Request | Purpose | iOS Version |
|---|---|---|
| VNDetectFaceRectanglesRequest | Face detection in images | iOS 11 |
| VNRecognizeTextRequest | Text recognition (OCR) | iOS 13 |
| VNDetectBarcodesRequest | Barcode and QR detection | iOS 11 |
| VNClassifyImageRequest | Image classification | iOS 13 |
| VNDetectContoursRequest | Contour analysis | iOS 14 |
| VNTrackObjectRequest | Object tracking | iOS 11 |
Frequently Asked Questions
What is the difference between Vision and Core ML for computer vision?
Vision provides ready-to-use algorithms (face detection, OCR, barcodes) without model training. Core ML is the engine for executing custom models. Vision + VNCoreMLRequest allows running Core ML models within the Vision pipeline, getting the best of both worlds: Vision’s ready detectors + Core ML custom classification.
Does Vision work in real-time on video?
Yes, Vision supports real-time video stream processing through VNSequenceRequestHandler for tracking and AVCaptureVideoDataOutput for frame-by-frame processing. On devices with A12+ and Neural Engine, Vision processes up to 60 fps for face detection. For heavy tasks (OCR, classification), it is recommended to process every 5–10th frame.
What languages does VNRecognizeTextRequest support?
VNRecognizeTextRequest supports 13 languages: English, Russian, Chinese (simplified and traditional), Japanese, Korean, Italian, German, Spanish, Portuguese, French, Arabic, Vietnamese, and Thai. To recognize multiple languages in a single image, specify an array in the recognitionLanguages property.
Can Vision be used with a Core ML model?
Yes, through VNCoreMLRequest. You create a Core ML model wrapped in VNCoreMLModel and pass it to VNCoreMLRequest. Vision automatically handles image preprocessing (scaling, cropping) and feeds it to the Core ML model. The result is returned as VNCoreMLFeatureValueObservation with the model’s output data.
Does Vision send images to Apple’s server?
No, Vision performs all analysis entirely on-device. Images never leave the user’s device. This is Apple’s core architecture: all ML frameworks (Vision, NaturalLanguage, Core ML, Speech) work on-device to ensure privacy. No data is sent to Apple’s servers.
Summary
- Vision — Apple’s on-device computer vision framework with a VNRequest-based API, available since iOS 11 on all Apple devices.
- VNDetectFaceRectanglesRequest and VNDetectFaceLandmarksRequest detect faces and key points for filters, masks, and animation.
- VNRecognizeTextRequest — built-in OCR for 13 languages with .fast and .accurate levels, achieving up to 96% accuracy for documents.
- VNDetectBarcodesRequest decodes all popular barcode and QR code formats in both photos and video streams.
- VNTrackObjectRequest tracks objects between video frames via optical flow without retraining the detector.
- Core ML integration via VNCoreMLRequest enables running custom classification and detection models within the Vision pipeline.
- All computation is performed on-device using the Neural Engine — no image data is sent to servers, ensuring complete data privacy.
We will develop a mobile application turnkey
IT Sectr creates iOS and Android applications for startups and businesses since 2017. We will advise you and propose the best solution.
Read also