Text Recognition in Apps — What It Is, OCR and Vision

What is Text Recognition: OCR Principles

Text Recognition (OCR) is a computer vision process that converts text images into machine-readable characters. OCR consists of stages: image preprocessing (binarization, deskew, skew correction), segmentation (splitting into lines, words, characters), recognition (classifying each character), and post-processing (dictionary check, error correction). Modern OCR systems (ML Kit, Vision) use end-to-end neural networks that combine all stages.

Types of OCR: printed text — recognition of typewritten text from documents, signs, screens — accuracy 95–99%; handwriting — recognition of handwritten input — accuracy 80–90% on Latin, 70–80% on Cyrillic; scene text — text in natural scenes: house numbers, road signs, store names — the most challenging due to perspective distortions and glare.

CRNN + CTC Loss — the architecture used in modern OCR models. Convolutional Recurrent Neural Network (CNN + LSTM) extracts image features, while Connectionist Temporal Classification decodes the character sequence without requiring prior segmentation. CRNN works with texts of any length and is robust to skew and distortions. According to arXiv (2025), CRNN models achieve 97.5% accuracy on the ICDAR 2019 benchmark.

CRNN for mobile devices — ML Kit uses a MobileNetV2 + CRNN model with INT8 quantization. The model takes 2–5 MB (latent) and runs on GPU/NPU via TFLite Delegate. Unlike Tesseract (classic pipeline), neural network OCR does not require separate character segmentation and is more robust to noise. Downside: requires GPU or NPU for real-time — on pure CPU, speed drops to 5–10 FPS.

ML Kit Text Recognition on Android and iOS

ML Kit Text Recognition is the primary OCR tool for mobile applications. It is available in two versions: V2 (Latin-based — optimized for Latin, Cyrillic, digits) and V1 (Latin + CJK — Japanese, Chinese, Korean, but slower). V2 uses an end-to-end CRNN model, V1 uses an older CNN + LSTM. Google recommends V2 for all cases except when CJK recognition is needed.

ML Kit result structure: Text → TextBlock → Line → Element (Word/Symbol). Each level has a bounding box, confidence (0..1), and recognized text. Text is the root object with fullText (all recognized text in one line). TextBlock is a logical text block (paragraph, column). Line is a text line. Element is a word or symbol. Use confidence to filter inaccurate recognitions.

// ML Kit Text Recognition V2
val recognizer = TextRecognition.getClient(
    TextRecognizerOptions.DEFAULT_OPTIONS
)

recognizer.process(inputImage)
    .addOnSuccessListener { text ->
        val fullText = text.text
        text.textBlocks.forEach { block ->
            val blockText = block.text
            val blockRect = block.boundingBox
            block.lines.forEach { line ->
                line.elements.forEach { element ->
                    val word = element.text
                    val confidence = element.confidence
                }
            }
        }
    }
    .addOnFailureListener { error -> /* error */ }

Text Recognition on iOS via ML Kit: the API is similar to Android but uses UIImage/CVPixelBuffer instead of InputImage. ML Kit automatically uses Apple Neural Engine on A12+ via Core ML Delegate. For iOS, ML Kit Text Recognition V2 achieves 15–30 ms speed on iPhone 15 Pro. For maximum performance, convert UIImage to CVPixelBuffer before passing — this reduces conversion overhead.

Apple Vision Framework: VNRecognizeTextRequest

Apple Vision Framework provides native OCR through VNRecognizeTextRequest (iOS 13+). Vision OCR uses Apple Neural Engine (ANE) and does not require additional SDKs. Supports 13 languages (EN, FR, IT, DE, ES, PT, ZH, JP, KO, RU, AR, TH, VI). Vision automatically detects the text language (language correction) and supports multiple languages in a single frame. Speed — 10–40 ms on A16+.

Vision OCR features: recognitionLevel (fast vs accurate), automaticLanguageDetection, customWords (adding specific terms), supportsTop candidates (multiple recognition variants for blurry text). Fast mode gives 90% accuracy at 10–20 ms, accurate mode gives 95% at 30–50 ms. For production, use accurate with confidence filtering >= 0.5.

import Vision

let request = VNRecognizeTextRequest { request, error in
    guard let observations = request.results as? [VNRecognizedTextObservation] else { return }
    for observation in observations {
        let topCandidate = observation.topCandidates(1).first
        let text = topCandidate?.string ?? ""
        let confidence = topCandidate?.confidence ?? 0
        let boundingBox = observation.boundingBox
    }
}
request.recognitionLevel = .accurate
request.usesLanguageCorrection = true

let handler = VNImageRequestHandler(cgImage: cgImage, orientation: .up)
try handler.perform([request])

Vision vs ML Kit on iOS: Vision is faster on older devices (iPhone X–13) due to the built-in ANE. ML Kit is more accurate on complex scenes (distortions, skew, glare) thanks to Google's model trained on millions of scene text images. Vision does not support confidence for individual characters (only for words), which limits filtering. For documents — Vision (faster), for scene text — ML Kit (more accurate).

Tesseract OCR: Open-Source Alternative

Tesseract OCR is an open-source text recognition engine developed by HP (1985–1998) and maintained by Google (2006+). Tesseract 5 (LSTM-based) uses the CRNN neural network architecture, providing a significant accuracy boost over Tesseract 4 (classic + LSTM). Tesseract supports 100+ languages, including Cyrillic, Arabic, Hebrew, and CJK. For Android — tess-two (fork), for iOS — swift-tesseract.

Tesseract drawbacks: speed 50–200 ms per frame (CPU-only, no GPU acceleration), requires image preprocessing (binarization, deskew, orientation detection) before passing to the engine, no built-in text detection — you need to pass the image region (often paired with OpenCV Text Detection or EAST). Tesseract achieves 90% accuracy on clean documents and ~70% on scene text.

When to choose Tesseract: if the app runs on devices without Google Play (Huawei), needs support for rare languages (Sanskrit, Hebrew), or requires full OCR pipeline customization (training on custom fonts). For all other cases, ML Kit or Vision are faster, more accurate, and require less code. Tesseract is only justified when third-party SDKs are restricted.

// Tesseract OCR via tess-two
val tess = TessBaseAPI()
tess.init(dataPath, "rus+eng")
tess.pageSegMode = TessBaseAPI.PageSegMode.PSM_AUTO

val bitmap = BitmapFactory.decodeResource(resources, R.drawable.text)
val gray = Bitmap.createBitmap(bitmap.width, bitmap.height, Bitmap.Config.ARGB_8888)
OpenCV.process(bitmap, gray) // Binarization + deskew

tess.setImage(gray)
val result = tess.utF8Text
tess.recycle()

Preprocessing for Tesseract is mandatory: convert to grayscale, binarization (Otsu or Adaptive), deskew, noise removal (median blur). Without preprocessing, Tesseract accuracy drops to 50–60%. Use OpenCV for preprocessing: Imgproc.cvtColor → Imgproc.threshold → Imgproc.erode/dilate → Core.rotate (deskew). For documents with a uniform background, binarization is sufficient; for scene text, a full pipeline is needed.

Image Preprocessing for OCR

Image quality is the main factor for OCR accuracy. ML Kit and Vision have built-in preprocessing, but for difficult cases (dark photos, blur, overexposure), additional processing improves accuracy by 10–15%. Key techniques: contrast enhancement (CLAHE), sharpening (unsharp mask), perspective correction (perspective transform), shadow and glare removal.

CLAHE (Contrast Limited Adaptive Histogram Equalization) — adaptive histogram equalization that improves contrast in local regions. CLAHE is effective for photos of documents with uneven lighting (partly shadowed, partly lit). OpenCV Core.createCLAHE with clipLimit=2.0 and tileGridSize=(8,8) gives optimal results for OCR. After CLAHE, ML Kit accuracy on dark documents improves from 70% to 88%.

Perspective correction — mandatory for photos of documents taken at an angle. Use OpenCV findHomography + warpPerspective to align the document. For automatic document boundary detection, use Canny edge detection + findContours + approxPolyDP. After perspective correction, OCR accuracy improves by 20–30% for shots at angles >30°.

// CLAHE + OpenCV preprocessing
val mat = Mat()
Utils.bitmapToMat(bitmap, mat)
Imgproc.cvtColor(mat, mat, Imgproc.COLOR_RGB2GRAY)

val clahe = Imgproc.createCLAHE(2.0, Size(8.0, 8.0))
clahe.apply(mat, mat)

Imgproc.GaussianBlur(mat, mat, Size(3.0, 3.0), 0.0)
Imgproc.threshold(mat, mat, 0.0, 255.0, Imgproc.THRESH_BINARY or Imgproc.THRESH_OTSU)

val processedBitmap = Bitmap.createBitmap(mat.cols(), mat.rows(), Bitmap.Config.ARGB_8888)
Utils.matToBitmap(mat, processedBitmap)

Text detection before OCR — for scene text, use EAST (Efficient Accurate Scene Text Detector) or CRAFT (Character Region Awareness for Text Detection) before passing to OCR. EAST detects text bounding boxes in a scene in 5–15 ms. CRAFT is more accurate (97%) but slower (50–100 ms). Text detection allows filtering out areas without text and passing only relevant regions to OCR, speeding up overall processing.

OCR Applications in Mobile Apps

Document scanning — the most widespread OCR application. Scanning apps (CamScanner, Adobe Scan, Google Drive) use ML Kit or Vision to recognize text from document photos. Typical pipeline: document boundary detection → perspective correction → CLAHE processing → OCR → formatting (PDF, DOCX). ML Kit Text Recognition V2 handles it in 100–200 ms per A4 page.

Real-time text translation — a combination of OCR and machine translation. Google Translate, Microsoft Translator, Yandex Translate use ML Kit or Vision to recognize text from the camera and overlay the translation on top of the original text (AR translation). Requirement: latency < 200 ms for comfortable UX. ML Kit V2 + NNAPI delivers 30–80 ms per frame, leaving headroom for translation and rendering.

Automatic data entry — OCR for extracting data from business cards, bank cards, IDs. ML Kit recognizes the text, then post-processing parses the structure: name, phone, email (business cards) or card number, expiry, CVV (payment cards). For business cards, use regular expressions after OCR. For bank cards — specialized ML models (paid, ~99% accuracy).

// OCR + AR translation (simplified)
let request = VNRecognizeTextRequest { req, _ in
    guard let results = req.results as? [VNRecognizedTextObservation] else { return }
    for observation in results {
        let text = observation.topCandidates(1).first?.string ?? ""
        let rect = observation.boundingBox
        // Translation and AR rendering over rect
        DispatchQueue.main.async {
            overlayView.showTranslation(text, at: rect)
        }
    }
}
request.recognitionLevel = .fast
request.usesLanguageCorrection = false

OCR for visually impaired users — Assistive Technology: apps read text from packages, menus, signs aloud. They use ML Kit OCR + Text-to-Speech (Android TTS / iOS AVSpeechSynthesizer). Key requirement — real-time with low latency (< 100 ms). Seeing AI (Microsoft) and Envision AI use this approach. For accessibility apps, use fast recognition mode and do not filter by confidence — any text is valuable to the user.