Vision: computer vision framework and how it works on iOS

What is Vision?

Vision is a high-level computer vision framework introduced by Apple at WWDC 2017. It provides developers with a unified interface for performing various image and video analysis tasks without needing to dive into computer vision mathematics or optimize for specific neural processors. Vision automatically uses the Apple Neural Engine on devices with A12+ chips.

Unlike low-level libraries like OpenCV, Vision abstracts the complexity: the developer creates a VNRequest object, configures parameters, and runs processing through VNImageRequestHandler. The framework decides which compute unit to use — CPU, GPU, or ANE — and returns structured results. According to Apple (WWDC 2025), Vision is used in 40% of App Store applications that work with images.

Core Capabilities

Vision includes APIs for face and landmark detection (up to 68 points), text recognition (OCR) with support for 30+ languages, QR and EAN barcode scanning, object tracking between frames, rectangle and contour detection, image classification via Core ML, motion estimation and optical flow, and human detection with segmentation. Each operation is represented by a separate VNRequest subclass.

Key Vision APIs

Vision is built on the Request → Handler → Results architecture, where each request is a specific task: find faces, recognize text, track an object. Let’s look at the most commonly used APIs.

VNRequest — the Foundation of All Operations

VNRequest is an abstract base class from which all concrete Vision requests inherit. Each request has a completionHandler, configuration parameters, and a regionOfInterest for processing part of an image. After creating a request, it is passed to VNImageRequestHandler (for static images) or VNSequenceRequestHandler (for video streams). Results are returned as an array of observations — subclasses of VNObservation.

Face and Landmark Detection

VNDetectFaceRectanglesRequest finds all faces in an image and returns their bounding boxes. VNDetectFaceLandmarksRequest additionally identifies 68 key landmarks: eye contours, eyebrows, nose, lips, and jaw. VNDetectFaceCaptureQualityRequest evaluates face capture quality — from 0 to 1 — useful for biometrics. According to Apple, face detection accuracy on devices with Neural Engine reaches 99% at frontal angles.

Text Recognition

VNRecognizeTextRequest is the API for optical character recognition (OCR) on images. It supports printed text in Latin, Cyrillic, Chinese, Japanese, Korean, and other languages. The result is an array of VNRecognizedTextObservation objects, each containing a text string, bounding box, and confidence level. Two modes are available: Fast for document scanning and Accurate for complex fonts.

• VNDetectFaceRectanglesRequest — face detection returning bounding boxes
• VNDetectFaceLandmarksRequest — 68 face landmark points
• VNRecognizeTextRequest — OCR with 30+ language support
• VNDetectBarcodesRequest — QR, EAN, PDF417 scanning
• VNCoreMLRequest — running custom Core ML models

Integrating Vision in iOS

To use Vision, simply import the framework, create a VNRequest object, and pass it to VNImageRequestHandler. Let’s look at an example of text recognition on an image.

Swift Code Example

The code creates a VNRecognizeTextRequest with accurate recognition mode, runs it on an image, and outputs the recognized text to the console. This simple example demonstrates minimal Vision integration in a Swift project using VNImageRequestHandler in just a few lines of code.

import Vision

// 1. Creating a text recognition request
let request = VNRecognizeTextRequest { request, error in
    guard let observations = request.results
        as? [VNRecognizedTextObservation]
    else { return }

    for observation in observations {
        let topCandidate = observation
            .topCandidates(1)
            .first
        print("Text: \(topCandidate?.string ?? "")")
    }
}

// 2. Enabling accurate mode
request.recognitionLevel = .accurate
request.usesLanguageCorrection = true

// 3. Running the handler
let handler = VNImageRequestHandler(
    url: imageURL, options: [:]
)
try? handler.perform([request])

The Swift code configures a VNRecognizeTextRequest with accurate recognition level and language correction enabled. VNImageRequestHandler loads the image from a URL and executes the request. The results contain recognized text strings with bounding boxes and confidence scores for each token. The VNRecognizedTextObservation array can be conveniently displayed on screen.

For real-time video processing, use VNSequenceRequestHandler instead of VNImageRequestHandler. It accepts an array of images (frames) and executes the same request sequentially, maintaining state between frames — essential for object tracking. VNSequenceRequestHandler automatically manages memory: old observations are removed when an object leaves the frame. Real-time performance depends on request complexity: face detection runs at 60 FPS, while text recognition runs at 15–30 FPS on devices with A16 chips.

Vision vs Core Image

Vision and Core Image are two Apple frameworks for working with images, but they solve fundamentally different tasks. Core Image is a framework for filtering and transforming images (applying filters, color correction, blurring). Vision is a framework for understanding image content: what is depicted in it.

Use Vision when you need to understand what is in an image: faces, text, QR codes, objects. Use Core Image when you need to modify an image: apply a filter, adjust colors, crop. Often these two frameworks are combined: first Core Image improves image quality, then Vision recognizes text on it.

In practice, Vision and Core Image are used together in document scanning applications. Core Image automatically increases contrast and removes shadows (CIFilter with CIPhotoEffectNoir), while Vision recognizes text on the enhanced image. According to Apple (WWDC 2025), OCR accuracy increases by 15–20% when preprocessing images through Core Image compared to raw camera frames.

Another important use case for combined usage is real-time face analysis for augmented reality applications. Vision detects the face and identifies 68 landmarks, while Core Image applies filters to the detected regions. ARKit uses Vision for face tracking and Core Image for rendering effects, resulting in a total latency of less than 16 ms on devices with A15+ chips.

When developing in SwiftUI for Vision integration, the Representable protocol is used, wrapping AVCaptureSession and VNImageRequestHandler in UIViewRepresentable. The camera is displayed through PreviewView, each frame is passed to VisionRequestHandler via AVCaptureVideoDataOutputSampleBufferDelegate. This architectural approach preserves SwiftUI reactivity and delivers Vision analysis results as @Published properties for immediate UI updates.

Vision Use Cases

Vision is used in a wide range of iOS applications: from document scanners to augmented reality apps. Let’s look at three typical scenarios.

Document Scanning

VNDetectDocumentSegmentationRequest (available since iOS 17+) automatically detects document boundaries in an image, corrects perspective, and returns a straightened image. Combined with VNRecognizeTextRequest, it creates a full-featured OCR scanner capable of recognizing invoices, business cards, and book pages. According to Apple, document segmentation takes 30–80 ms on a device with an A15 chip.

Object Tracking in Video

VNTrackObjectRequest tracks the movement of a selected object between video frames in real time. The developer specifies the object’s bounding box on the first frame, and Vision automatically calculates its new position in subsequent frames. Tracking is used in video analytics applications, AR games, and surveillance systems. Tracking accuracy on A16 chips is 95% at object movement speeds of up to 30 pixels per frame.

Rectangle and Contour Detection

VNDetectRectanglesRequest finds rectangular areas in an image: screens, sheets of paper, signs, documents. The API returns corner coordinates with perspective correction. Combining rectangles with VNDetectContoursRequest allows extracting precise object contours — useful for augmented reality applications and graphics editors. VNDetectContoursRequest returns an array of contour control points that can be converted to UIBezierPath for rendering.