SFSpeechRecognizer — what it is, API and iOS integration

What is SFSpeechRecognizer: Feature Overview

SFSpeechRecognizer is a class from Speech.framework that represents a speech recognizer for a specific language. It is initialized with a Locale (ru_RU, en_US, zh_CN). SFSpeechRecognizer manages a recognition request (SFSpeechRecognitionRequest) and returns a result (SFSpeechRecognitionResult) with transcriptions and metadata. It supports two types of requests: SFSpeechAudioBufferRecognitionRequest (live audio stream) and SFSpeechURLRecognitionRequest (audio file). Both return SFTranscription — an array of alternative recognition hypotheses.

SFSpeechRecognitionResult contains: bestTranscription (best hypothesis, SFTranscription), transcriptions (all alternatives), isFinal (final/intermediate). SFTranscription contains: formattedString (text), segments (array of SFTranscriptionSegment with timestamps, confidence, substringRange, alternativeSubstrings). Confidence for each segment (0..1) — allows filtering unreliable fragments. Segments are a key feature of Speech.framework: they provide per-word markup with confidence scores.

SFSpeechRecognizer Architecture: AVFoundation (audio capture) → Audio Buffer → SFSpeechAudioBufferRecognitionRequest → SFSpeechRecognizer (ASR engine) → SFSpeechRecognitionResult → SFSpeechRecognitionTask (control: cancel, finish, pause). Apple’s ASR engine uses a hybrid architecture: Conformer (encoder) + RNNT (decoder) for on-device, Transducer for cloud. Models are optimized for Apple Neural Engine (ANE). On A17 Pro, on-device ASR runs with RTF 0.3–0.6 (faster than real time).

Audio Requirements: SFSpeechRecognizer accepts LPCM (16 kHz, 16 bit, mono) or Opus (iOS 17+). AVFoundation can capture buffers in any format, the request converts automatically. Minimum length: 0.5 seconds (silence detection). Maximum: 1 minute (cloud) / 2 minutes (on-device) per request. For long transcription, split audio into 30–60 second chunks with 2–5 second overlap.

SFSpeechRecognizer Setup and Permissions

User Permissions: SFSpeechRecognizer requires two explicit permissions. NSMicrophoneUsageDescription (Privacy — Microphone Usage Description) — for microphone access. NSSpeechRecognitionUsageDescription (Privacy — Speech Recognition Usage Description) — for speech recognition access. Both are strings explaining the reason to the user. SFSpeechRecognizer.requestAuthorization — displays the permission dialog. Statuses: notDetermined, denied, restricted, authorized. Without authorized status, calling the recognizer returns error 216 (Speech framework error).

Availability Check: SFSpeechRecognizer.isAvailable — checks whether ASR is available for the current language. On iOS 16-, isAvailable = false without internet. On iOS 17+, isAvailable = true for on-device languages even offline. SFSpeechRecognizer.supportedLocales — a static method to get the list of languages supported by the device. It is recommended to check isAvailable before each launch (the user may disable Siri/Dictation in settings). Availability depends on region: Chinese — only in China, Arabic — in UAE.

// SFSpeechRecognizer setup
import Speech

SFSpeechRecognizer.requestAuthorization { status in
    guard status == .authorized else { return }
}

let recognizer = SFSpeechRecognizer(locale: Locale(identifier: "ru_RU"))!
guard recognizer.isAvailable else {
    print("ASR not available. Enable Siri & Dictation in Settings")
    return
}

let request = SFSpeechAudioBufferRecognitionRequest()
request.requiresOnDeviceRecognition = true

Error Handling: SFSpeechRecognizer is called with a completion handler that may return an Error. Main errors: 203 — no internet (cloud, iOS 16-); 216 — not authorized (no permission); 301 — audio error (microphone/format issue); 401 — service unavailable (service overloaded); 601 — language unavailable (language not supported on device). For on-device iOS 17+, main errors: 301 (audio), 601 (language). Always handle the completion handler — errors can occur at any point during recording.

Live Speech Recognition from Microphone

Live ASR Pipeline: AVAudioEngine (microphone capture) → installTap (audio buffer) → request.append(buffer) → SFSpeechRecognizer → recognitionTask → delegate/resultHandler. AVAudioEngine provides low latency (5–10 ms from mic to buffer). SFSpeechRecognitionDelegate: didHypothesizeTranscription (every 200–500 ms — partial text), didFinishRecognition (final result). Task.isFinishing — can cancel when recognition is complete. For continuous recognition (infinite dictation), create a new task after isFinal.

SFSpeechRecognitionTaskDelegate: optional methods. speechRecognitionDidDetectSpeech (speech start) — for UI indication. didHypothesizeTranscription (intermediate text) — for display while speaking. didFinishRecognition (final result) — for text finalization. didFinishSuccessfully (success/error) — for completion. didProcessAudio (audio buffer processed) — for RMS meter. It is recommended to implement didHypothesizeTranscription — the user sees text immediately without delay. Final transcription is for saving.

// Live speech recognition
let audioEngine = AVAudioEngine()
let request = SFSpeechAudioBufferRecognitionRequest()
request.shouldReportPartialResults = true

recognitionTask = recognizer.recognitionTask(with: request) { result, error in
    if let result = result {
        textView.text = result.bestTranscription.formattedString
    }
    if error != nil || result?.isFinal == true {
        audioEngine.stop()
        request.endAudio()
    }
}

let inputNode = audioEngine.inputNode
let recordingFormat = inputNode.outputFormat(forBus: 0)
inputNode.installTap(onBus: 0, bufferSize: 1024,
                     format: recordingFormat) { buffer, _ in
    request.append(buffer)
}
audioEngine.prepare()
try audioEngine.start()

Continuous Recognition: for “always listening” mode (voice input, assistant), you need to restart the task after isFinal. Create a loop: finishTask → request = SFSpeechAudioBufferRecognitionRequest() → recognitionTask = recognizer.recognitionTask(with: request). To avoid gaps, overlap the end of one task with the start of the next (start a new request 1–2 seconds before the previous one ends). AVFoundation AVAudioSession — configure .playAndRecord for simultaneous playback and recording. On iOS 17+, continuous ASR with on-device consumes 2–5% battery per hour (A15+).

Audio File and Recording Recognition

SFSpeechURLRecognitionRequest — a request for recognizing an audio file by URL. Supports formats: .wav (16 kHz, 16 bit, mono), .m4a (AAC), .mp4, .mov. Maximum length: ~1 minute for cloud, ~2 minutes for on-device. For longer files, split into chunks (AVAssetExportSession + trim). SFSpeechURLRecognitionRequest does not support streaming (no partial results) — only the final result. For partial results with a file, convert to an Audio Buffer Request.

Getting Audio File Transcription: SFSpeechRecognizer.recognitionTask(with: request) resultHandler. Extract bestTranscription.formattedString. For word-level timestamps — segments from bestTranscription.segments (segment.duration, segment.timestamp, segment.substring). Confidence per segment — ASR confidence for each word (use for filtering). Alternative hypotheses — transcription formatter alternatives for words with low confidence. For files with multiple speakers, SFSpeechRecognitionRequest may return multiple requests (one per speaker, iOS 17+).

// Audio file transcription
let audioURL = Bundle.main.url(forResource: "recording", withExtension: "m4a")!
let request = SFSpeechURLRecognitionRequest(url: audioURL)
request.requiresOnDeviceRecognition = true

recognizer.recognitionTask(with: request) { result, error in
    guard let result = result, error == nil else {
        print("Error: \(error?.localizedDescription ?? "unknown")")
        return
    }

    let transcription = result.bestTranscription
    let text = transcription.formattedString
    let words = transcription.segments.map { segment in
        Word(text: segment.substring,
              start: segment.timestamp,
              duration: segment.duration,
              confidence: segment.confidence)
    }
}

Splitting Long Audio Files: for files > 1 minute, split into 30–60 second chunks with 2–3 second overlap (stitching). AVAssetExportSession + CMTimeRange — export chunks. Alternative: UISelectionView (user selects a segment). Transcription stitching: concat texts, stitch by overlap (last 2–3 words of previous chunk compared with first 2–3 words of next chunk — remove duplicate). Vosk and Whisper support long audio natively, SFSpeechRecognizer does not. For long transcription, I recommend Whisper (Core ML) — no length limit.

On-device vs Cloud: Mode Comparison

On-device Mode (iOS 17+): request.requiresOnDeviceRecognition = true. ASR runs locally on Apple Neural Engine (ANE). Advantages: no internet required, privacy (audio never leaves the device), zero cost (free), low latency (RTF 0.3–0.6). Disadvantages: lower accuracy (WER 12–14% vs 7–12%), 2 minute limit per request, limited language set (not all 60 languages available on-device). The on-device model takes ~50–100 MB on the device and is downloaded on first request.

Cloud (iOS 16-): request.requiresOnDeviceRecognition = false (or parameter absent). ASR on Apple servers — more accurate (WER 7% EN, 12% RU), more languages, up to 1 minute per request. Requires internet (WiFi or cellular). Apple does not charge developers for cloud ASR (free). Limits: 1 request per minute per application (unspecified), but Apple may throttle at production scale. Cloud ASR provides best-effort quality without SLA. For enterprise apps, use Google Cloud ASR or Whisper (Core ML).

Combined Recognition (iOS 17+): request.requiresOnDeviceRecognition = false with internet (cloud), = true when offline (fallback). Apple automatically selects the mode. For accuracy-critical apps: check NetworkMonitor (NWPathMonitor) — use cloud with internet, on-device without. For privacy-critical apps: always on-device. For transcription: cloud (more accurate). For voice input: on-device (faster). Combined is recommended: 80% cloud, 20% on-device fallback.

SFSpeechRecognizer Use Cases in iOS Apps

Voice Input in Custom Keyboard — SFSpeechRecognizer integrated into keyboard extensions (iOS 10+). The user taps the microphone button, ASR transcribes speech into text and inserts it into the text field. Requirements: partial results (see text in real time), on-device (keyboard must work without internet). SFSpeechRecognizer + AVSpeechSynthesizer — voice input + voice output. For custom keyboards on iOS 17+, use on-device. Keyboard extension limitations: AVAudioEngine is available but with time restrictions (background execution limited).

Meeting and Lecture Transcription — apps for recording and transcribing audio (Otter.ai, Rev, Apple Voice Memos). SFSpeechURLRecognitionRequest processes .m4a files. For long recordings (30–60 minutes) — Whisper Core ML (no length limit). SFSpeechRecognizer segments with timestamps — for syncing text with audio (text highlighting during playback). Confidence per segment — for displaying unreliable fragments (yellow highlight). For speaker diarization (who spoke) — Apple does not provide an API, use pyannote-audio + Whisper on the server.

Voice Commands in iOS Apps — SFSpeechRecognizer + VGS framework (Voice Grammar Services) for executing commands: “open settings”, “send message”, “turn on the light”. Grammar-based recognition: SFSpeechRecognitionRequest contextualStrings = array of commands. This improves command recognition accuracy to 95% (vs 85% free-form). SFSpeechRecognitionTask.cancel — to interrupt after command execution. VGS + SFSpeechRecognizer + Shortcuts — custom voice commands without writing UI. For accessibility: Voice Control (built-in) + SFSpeechRecognizer (custom commands).

// Voice commands with contextual strings
let request = SFSpeechAudioBufferRecognitionRequest()
request.requiresOnDeviceRecognition = true
request.shouldReportPartialResults = true
request.contextualStrings = ["open contacts", "send message",
                             "show notifications", "turn on flashlight"]

recognitionTask = recognizer.recognitionTask(with: request) { result, _ in
    guard let text = result?.bestTranscription.formattedString.lowercased() else { return }
    voiceCommands.first { text.contains($0) }.map { command in
        DispatchQueue.main.async { self.executeCommand(command) }
        recognitionTask?.cancel()
    }
}

Dictation in Medical and Legal Apps — SFSpeechRecognizer for creating structured documents by voice. Domain Adaptation: contextualStrings with medical/legal terms (500+ phrases). SFSpeechRecognitionRequest.customLmProbability — contextualStrings impact (0.1–1.0). For medical dictation, use on-device (patient data privacy). Whisper fine-tuned on medical data — alternative with WER 5% instead of 12% for base SFSpeechRecognizer. HIPAA compliance: on-device mode (data never leaves the device). For appointment transcription — SFSpeechURLRecognitionRequest + segments with speaker timestamps.