Speech Recognition — technology, ASR principles and how it works

What Is Speech Recognition: ASR Principles

Automatic Speech Recognition (ASR) is a pipeline: audio → preprocessing (16 kHz mono, noise reduction, VAD — voice activity detection) → feature extraction (MFCC, LogMel FilterBank) → acoustic model (neural network: CTC / RNNT / Transducer) → language model (LM) → decoding (text). Modern end-to-end models (Whisper, Google USM, Conformer CTC) combine the acoustic + language model into a single Transformer, simplifying the pipeline and improving accuracy.

ASR Architectures for Mobile Devices: CTC (Connectionist Temporal Classification) — fast, does not require audio-text alignment, used in Vosk and Android native. RNNT (Recurrent Neural Network Transducer) — streaming ASR, low latency (Google USM). Conformer — a hybrid of CNN + Transformer, best accuracy but heavier: Whisper Medium (769M params) — 30–60x latency. For on-device, CTC is preferred: Vosk CTC models take 50–100 MB, latency 0.3–0.8x real-time.

ASR Metrics: WER (Word Error Rate) — percentage of substituted, deleted, and inserted words relative to the reference. Google Cloud ASR — 7% WER (EN), 12% (RU). Vosk — 13% (RU), 9% (EN). Whisper Small — 6% (EN), 10% (RU). CER (Character Error Rate) — similar, at the character level. Real-Time Factor (RTF) — processing time / audio duration. RTF < 1 — faster than real time. RTF < 0.3 — real-time ASR. Voice input requires RTF < 1, transcription requires RTF < 3.

VAD (Voice Activity Detection) — detection of voice activity, separating speech from silence and noise. WebRTC VAD is the standard for mobile ASR: 30 ms frames, three modes (0, 1, 2 — from conservative to aggressive). VAD reduces RTF by 1.5–3x (skips non-speech segments). Silero VAD (MIT) is a neural VAD, more accurate than WebRTC (94% vs 85% ROC AUC), 5–10 ms latency, TFLite and Core ML. For production ASR, use the VAD → ASR cascade: this reduces false positives and speeds up processing.

Android SpeechRecognizer API

Android SpeechRecognizer is a built-in Android SDK class for speech recognition. It works in two modes: cloud (Google server) — high accuracy, requires internet; on-device (since Android 10+) — GBoard embedded ASR model, 20+ languages, WER 15–18%. SpeechRecognizer returns intermediate results (partial results) during speech and the final text. Supports 60+ languages via RecognizerIntent.EXTRA_LANGUAGE. Recommended for voice input — quick integration, free.

SpeechRecognizer API: created via SpeechRecognizer.createSpeechRecognizer(context). Intent with RecognizerIntent.ACTION_RECOGNIZE_SPEECH with extras: LANGUAGE_MODEL_FREE_FORM (free text) or LANGUAGE_MODEL_WEB_SEARCH (search queries). Results via RecognitionListener: onReadyForSpeech → onBeginningOfSpeech → onRmsChanged → onPartialResults → onResults. For continuous recognition (always-listening mode) — restart the recognizer after onResults. For battery saving, use onDeviceOnly = true.

// Android SpeechRecognizer
val recognizer = SpeechRecognizer.createSpeechRecognizer(context)
recognizer.setRecognitionListener(object : RecognitionListener {
    override fun onResults(results: Bundle) {
        val text = results
            .getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)
            ?.firstOrNull() ?: ""
        showResult(text)
    }
    override fun onPartialResults(partialResults: Bundle) {
        val partial = partialResults
            .getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)
            ?.firstOrNull() ?: ""
        showPartial(partial)
    }
})

val intent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply {
    putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
        RecognizerIntent.LANGUAGE_MODEL_FREE_FORM)
    putExtra(RecognizerIntent.EXTRA_LANGUAGE, "ru-RU")
}
recognizer.startListening(intent)

On-device vs Cloud on Android: on-device works without internet (Android 10+, Pixel 4+, Samsung S20+). Cloud is more accurate (Google Neural Network ASM Model) but has higher latency (300–800 ms including network). For voice input, use a hybrid approach: cloud with fallback to on-device when no network is available. Android SpeechRecognizer automatically selects the mode based on RecognizerIntent.EXTRA_PREFER_OFFLINE. For maximum privacy — set onDeviceOnly = true (but accuracy is 15–18% WER for Russian).

iOS SFSpeechRecognizer and Speech Framework

SFSpeechRecognizer is Apple’s framework for speech recognition on iOS, macOS, and watchOS. Available since iOS 10. On-device mode — since iOS 17 (local model, no internet required). Supports 60+ languages. Accuracy: WER 7–12% (on-device — 12–15%). SFSpeechRecognizer is available via Speech.framework — no additional SDKs required. Permission request via SFSpeechRecognizer.requestAuthorization.

SFSpeechRecognizer API: SFSpeechRecognizer(locale: Locale(identifier: “ru_RU”)) → SFSpeechAudioBufferRecognitionRequest (live audio) or SFSpeechURLRecognitionRequest (audio file). Streaming via recognitionTask(with:delegate:) with SFSpeechRecognitionTaskDelegate (didHypothesizeTranscription — partial, didFinishRecognition — final). On-device mode: request.requiresOnDeviceRecognition = true. For iOS 15+: request.shouldReportPartialResults = true (streaming result with low latency).

let recognizer = SFSpeechRecognizer(locale: Locale(identifier: "ru_RU"))
guard recognizer.isAvailable else { return }

let request = SFSpeechAudioBufferRecognitionRequest()
request.requiresOnDeviceRecognition = true
request.shouldReportPartialResults = true

recognitionTask = recognizer.recognitionTask(with: request) { result, error in
    if let result = result {
        let text = result.bestTranscription.formattedString
        let isFinal = result.isFinal
        DispatchQueue.main.async {
            textView.text = text
        }
    }
}

audioEngine.inputNode.installTap(config: bufferConfig) { buffer, _ in
    request.append(buffer)
}
audioEngine.prepare()
try audioEngine.start()

SFSpeechRecognizer vs Android SpeechRecognizer: SFSpeechRecognizer has native streaming without restart (Android requires repeated startListening). On-device on iOS is available since iOS 17 (Android since Android 10). Both require user permission (NSMicrophoneUsageDescription + NSSpeechRecognitionUsageDescription for iOS, RECORD_AUDIO for Android). SFSpeechRecognizer is more accurate for English (on-device), Android SpeechRecognizer is more accurate for Russian (cloud). For iOS before 17, on-device is unavailable — internet required. For iOS 17+, on-device gives WER 12–14% for Russian.

Vosk: Offline Recognition in 20+ Languages

Vosk is an open-source ASR library by Alpha Cephei for offline speech recognition. Based on the Kaldi ASR toolkit + CTC models. Supports 20+ languages: Russian, English, German, French, Spanish, Chinese, Arabic. Models from 50 MB (small — 50 MB, ru) to 1.2 GB (large — 1.2 GB, en). Vosk works on Android (JNI), iOS (C++ wrapper), Linux, Windows, Raspberry Pi. RTF: 0.3–0.8 on flagship devices. Vosk is the best choice for full offline ASR.

Vosk API: VoskWaveformModelLoader (model loading) → VoskWaveformRecognizer (recognizer creation) → recognizer.createGrammar(list) or recognizer.getResult() / recognizer.getPartialResult(). Grammar support (finite command set) — GrammarRecogniser — gives RTF 0.1–0.3 (3x faster). For voice input, use free recognition (getResult). For voice commands — GrammarRecogniser (99% accuracy on commands). Vosk also supports Speaker Identification (voice vector analysis).

// Vosk offline ASR
val model = VoskModel("model/vosk-model-small-ru-0.22")
model.load()

val recognizer = VoskRecognizer(model, 16000.0f)
recognizer.setWords(true)

audioRecord.startRecording()
while (isListening) {
    val buffer = ByteArray(3200) // 100ms audio
    audioRecord.read(buffer, 0, buffer.size)
    if (recognizer.acceptWaveform(buffer)) {
        val result = recognizer.result // JSON
        text += JSONObject(result).getString("text")
    }
}

Vosk vs Cloud ASR: Vosk is fully offline — privacy, zero latency, free. Cloud (Google, Yandex) is more accurate in complex scenarios (noise, accent) — WER 3–5% lower. Vosk is ideal for: voice input without internet, accessibility apps, call transcription (privacy). Cloud ASR is for voice assistants where accuracy matters more than privacy. Recommendation: Vosk for offline, SFSpeechRecognizer (iOS) / SpeechRecognizer (Android) for online — combine approaches for different scenarios.

Whisper OpenAI on Mobile Devices

Whisper is an OpenAI model for speech recognition, trained on 680,000 hours of audio (multilingual, 99 languages). Available in sizes: tiny (39M, WER 10% EN, 15% RU), small (244M, WER 6% EN, 10% RU), medium (769M, WER 4.5% EN, 8% RU). Whisper runs on mobile devices via Core ML (iOS, ANE) and TFLite (Android, GPU). Whisper tiny: RTF 4–10 on CPU, RTF 0.5–2 on GPU (Android). Whisper small: RTF 2–6 on GPU. Whisper medium — RTF 8–15, only for non-real-time.

Whisper on iOS (Core ML): Apple MLX Whisper — an implementation of Whisper for Core ML and MLX (Apple Neural Engine). Whisper tiny Core ML on iPhone 15 Pro: RTF 0.3–0.8 (faster than real-time!). Whisper small Core ML: RTF 1–3. Whisper Core ML uses ANE on A17 Pro (Neural Engine 35 TOPS). Hugging Face Transformers → Export to Core ML via coremltools-whisper. For streaming, use WhisperStitcher (chunking + stitching). Whisper on iOS is the most accurate on-device ASR (WER 6% EN, 10% RU).

// Whisper Core ML on iOS
import MLXWhisper

let model = try WhisperModel(from: "whisper-tiny")
let audio = try AudioUtils.loadAudio(url: audioURL)
let segments = try model.transcribe(audio: Audio, options: TranscribeOptions(
    language: "ru",
    wordTimestamps: true,
    temperature: 0.0
))

for segment in segments {
    print(segment.text) // final text with timestamps
}

Whisper vs Vosk: Whisper is more accurate (6% vs 9% WER EN), supports 99 languages, includes language detection, punctuation, and emotion recognition. Whisper is heavier (39M–769M vs 50M Vosk), slower in real-time (RTF 0.5–6 vs 0.3–0.8). Whisper tiny is comparable to Vosk in speed (RTF 0.5–2 GPU). Vosk is lighter, faster, better for streaming real-time. Whisper is for one-shot transcription where accuracy is prioritized over speed. Vosk is for real-time voice input. Use Whisper for final text, Vosk for interactive use.

Speech Recognition Use Cases in Mobile Applications

Voice Text Input — the most widespread use of ASR: dictating messages, search queries, notes. Requirement: RTF < 1 (real time), partial results (see text as spoken). Android Gboard and iOS Keyboard have built-in ASR (on-device, WER 12–18%). For custom implementation, use Android SpeechRecognizer / SFSpeechRecognizer. For maximum accuracy — Cloud ASR + Vosk fallback. For voice input with 98% accuracy in Russian — combine Vosk (offline) + Yandex SpeechKit (online).

Call and Meeting Transcription — ASR for automatic transcript generation. Requirements: no latency requirement, WER < 10%, speaker diarization (who is speaking). Whisper Small (offline) + pyannote-audio (diarization) is the standard stack for transcription. On iOS — Whisper Core ML (RTF 1–3, WER 6–10%). On Android — Whisper TFLite (RTF 2–6, WER 8–12%) or Google Cloud ASR + diarization. For privacy (calls do not go to the server) — Whisper on-device. Diarization accuracy: 80–90% DER.

Voice Assistants — Siri (SFSpeechRecognizer), Google Assistant (Android SpeechRecognizer), Alexa (on-device ASR). Custom assistant: ASR → NLU (Natural Language Understanding) → Action → TTS. ASR is the first stage — it determines the accuracy of the entire chain. For NLU, use RASA, Snips NLU (on-device), or Cloud NLU (Dialogflow, Amazon Lex). For TTS: Android TTS / iOS AVSpeechSynthesizer. An on-device voice assistant (Vosk + RASA + TTS) is fully private, works without internet, latency < 2 seconds per request.

// Voice assistant with Android SpeechRecognizer + TTS
recognizer.setRecognitionListener(object : RecognitionListener {
    override fun onResults(results: Bundle) {
        val query = results.getStringArrayList(
            SpeechRecognizer.RESULTS_RECOGNITION
        )?.firstOrNull() ?: return

        val intentResult = nluService.classify(query)
        textToSpeech.speak(intentResult.response)
        executeAction(intentResult.action)
    }
})

Accessibility for People with Motor Impairments — Speech Recognition allows controlling the app by voice: open contacts, send a message, dial a number. Android Voice Access and iOS Switch Control + VoiceOver are built-in solutions. For custom implementation: GrammarRecogniser (Vosk) with a fixed command set (50–100 phrases) — RTF 0.1–0.3, 99% accuracy. Vosk Grammar allows defining grammar in BNF format. Speed is critical for accessibility — Vosk Grammar is 5x faster than free recognition and consumes less battery.