TF Lite Delegate: what it is, GPU and NNAPI delegates

What is TF Lite Delegate: architecture and how it works

TF Lite Delegate is a software adapter between TensorFlow Lite Runtime and the device's hardware accelerator. The delegate intercepts model graph operations (convolutions, pooling, matrix multiplications) and executes them on a specialized compute unit instead of the CPU. If the delegate does not support a particular operation, it runs on the CPU — this is called partial delegation.

TF Lite Delegate Architecture is built around the SimpleDelegate API interface and the TfLiteDelegate structure in C++. Each delegate implements methods for delegating graph nodes, memory allocation, and execution on the target device. The developer connects the delegate via Interpreter::ModifyGraphWithDelegate before calling Invoke. According to the TensorFlow Guide, the delegate is applied to all supported model operations automatically.

Compatibility check is a mandatory step. Not all operations are supported by every delegate. TensorFlow Lite provides the Delegation Compatibility Check tool: run the model with the delegate and check how many operations (ops) are delegated. If less than 80% are delegated, consider choosing another delegate or sticking with CPU. According to TensorFlow (2025), GPU Delegate supports 45 operations, NNAPI — 60+.

At IT Sectr projects, we use GPU delegates for iOS and XNNPACK for Android with built-in compatibility verification: the model is tested on all delegates, selecting the fastest one with at least 90% delegation completeness.

// GPU Delegate connection on Android
val gpuDelegate = GpuDelegate()
val options = Interpreter.Options().apply {
    addDelegate(gpuDelegate)
    setNumThreads(4)
}
val interpreter = Interpreter(modelBuffer, options)
interpreter.run(input, output)
gpuDelegate.close()

Checking delegated operations — compatibility control via Interpreter. By default, the delegate accepts all operations it supports. For debugging, use logging of the delegated node count. If the model contains custom ops, the delegate does not accelerate them but does not break them either — they run on the CPU.

// Check delegated operations count
val delegateCount = interpreter.getDelegateOpsCount(gpuDelegate)
val totalNodes = interpreter.getNodesCount()
Log.d("TFLite", "Delegated: $delegateCount / $totalNodes")
if (delegateCount < totalNodes * 0.8) {
    // 80% threshold — choose another delegate
}

GPU Delegate: acceleration on the graphics processor

GPU Delegate is a TensorFlow Lite delegate for running models on the GPU via OpenGL ES 3.1+ (Android) or Metal (iOS). Graphics processors are optimized for parallel matrix operations — Core ML and convolutional neural networks (CNN) benefit the most. GPU Delegate reduces inference latency by 4–8x for models like MobileNet, EfficientNet, Inception.

GPU Delegate limitations: does not support all operations (only 45 out of ~120 in TF Lite), requires OpenGL ES 3.1 or Metal, consumes more power than CPU. GPU is inefficient for batch size > 1 in mobile scenarios. According to TensorFlow benchmarks (2025), on Pixel 8 with Adreno 740 GPU, MobileNetV2 runs in 4.2 ms on GPU vs 18.7 ms on CPU — a 4.4x acceleration.

GPU Delegate configuration includes precision selection: float16 reduces accuracy but speeds up inference by 30–40% without noticeable quality loss (for most tasks). Enable allow_precision_loss=true for maximum GPU performance. For high-precision tasks (medical, finance), use float32.

// GPU Delegate with precision optimization
val gpuOptions = GpuDelegateFactory.Options().apply {
    isPrecisionLossAllowed = true
    inferencePreference = GpuDelegateFactory.Options.InferencePreference.SUSTAINED_SPEED
}
val delegate = GpuDelegateFactory.create(gpuOptions)

GPU Delegate on iOS uses Metal Performance Shaders via Metal Delegate. On iOS, the delegate works through Core ML delegate for Apple Neural Engine or Metal directly. Apple A17 Pro runs MobileNetV2 in 1.3 ms — 6x faster than CPU. Metal delegate is available since iOS 12 and supports all devices with A7+ chips.

NNAPI Delegate: neural networks on Android via Neural Networks API

NNAPI Delegate (Android Neural Networks API) is a TF Lite delegate that uses hardware acceleration through Android NN HAL (Hardware Abstraction Layer). NNAPI redirects model operations to NPU, DSP, or GPU depending on the available device drivers. Supported on Android 8.1+ (API 27+) and is the recommended default delegate for Android.

NNAPI advantage — automatic accelerator selection. If the device has an NPU (Qualcomm Hexagon, MediaTek APU, Samsung NPU), NNAPI delegates operations to it. If no NPU — to GPU via OpenCL. If GPU is also unsupported — to CPU with DSP optimizations. The developer does not specify a particular accelerator — NNAPI selects the optimal one. According to Google I/O 2024, the NNAPI delegate on Snapdragon 8 Gen 3 accelerates LLM models by 12x.

NNAPI limitations: uneven support across devices. Older devices (Android 8.1) have a limited set of drivers. Huawei with Kirin does not support NNAPI through Google Services — use GPU Delegate. For guaranteed compatibility, Google recommends NNAPI Delegate as the first choice, with a fallback to GPU or XNNPACK.

// NNAPI Delegate with CPU fallback
val nnApiOptions = NnApiDelegate.Options().apply {
    acceleratorName = null // null = auto-select
    allowFp16 = true
    executionPreference = NnApiDelegate.ExecutionPreference.SUSTAINED_SPEED
}
val delegate = NnApiDelegate(nnApiOptions)
val interpreter = Interpreter(model, Interpreter.Options().apply {
    addDelegate(delegate)
    setNumThreads(4)
})

NNAPI Accelerator Name — a parameter for explicit accelerator selection. If null is passed, NNAPI selects automatically. For testing, specify a specific one: "qti", "google-edgetpu", "mediatek". The list of available accelerators is output via NnApiDelegate.getAvailableAccelerators(). In production, use auto-selection with a compatibility threshold.

XNNPACK Delegate: CPU optimization via SIMD

XNNPACK Delegate is a TensorFlow Lite delegate for optimized CPU execution via SIMD instructions (ARM NEON, SSE, AVX). XNNPACK does not require GPU or NPU — it is a pure CPU delegate, but with 2–4x acceleration through SIMD, operator fusion, memory pre-fetching, and quantized inference (INT8). Ideal for devices without a dedicated NPU or when guaranteed compatibility is needed.

XNNPACK was developed by Google as the default TF Lite delegate for CPU (included in TFLite Runtime by default). It supports 90+ operations — more than GPU or NNAPI. XNNPACK is especially effective for quantized models (INT8, INT16): operations run 2–3x faster than the float32 version. According to Google benchmarks (2025), XNNPACK accelerates INT8 MobileNetV2 by 2.8x relative to baseline CPU.

XNNPACK vs GPU: on devices without NPU, XNNPACK is often faster than GPU Delegate for small batch (1–4) — GPU has overhead on data transfer between CPU and GPU. XNNPACK operates in the same CPU address space — no memory copying. For models up to 5MB, XNNPACK can be faster than GPU. For large CNN models, GPU wins due to parallelism.

// XNNPACK Delegate enabled by default in TFLite 2.18+
// Explicit usage via XnnpackDelegate
val xnnpackOptions = XnnpackDelegate.Options().apply {
    numThreads = 4
    flags = XnnpackDelegate.Flags.USE_XNNPACK
}
val delegate = XnnpackDelegate(xnnpackOptions)
val interpreter = Interpreter(modelBuffer, Interpreter.Options().apply {
    addDelegate(delegate)
})

XNNPACK Flags: USE_XNNPACK — forcibly enables the delegate, FLUSH_TO_ZERO — handles denormalized numbers for acceleration, ENABLE_XNNPACK_SHAPES — dynamic input sizes. For maximum compatibility, use default options. If errors occur on older devices, disable XNNPACK — the model still runs on CPU but slower.

Core ML and Metal Delegate: acceleration on iOS

Core ML Delegate is a TensorFlow Lite delegate for iOS that uses Apple Core ML Framework. Core ML converts the TF Lite model to .mlmodel format and executes it on Apple Neural Engine (ANE), GPU (Metal), or CPU. Apple A17 Pro and M3 have a dedicated Neural Engine that Core ML uses automatically. Core ML Delegate accelerates inference by 5–10x compared to CPU on modern iOS devices.

Core ML on iOS supports flex delegate (dynamic delegation of operations available to Core ML) and full model conversion (converting the entire model to .mlmodel). Apple recommends flex delegate — it is faster as it works at runtime without prior conversion. Core ML Delegate supports float32, float16, and quantized models (INT8).

Metal Delegate is a lower-level delegate that works directly with Metal Performance Shaders. Use Metal when Core ML does not support certain operations. Metal Delegate provides maximum GPU control but requires more code. According to Apple (WWDC 2024), Metal Delegate for native Swift models can be 15–20% faster than Core ML due to no conversion overhead.

// Core ML Delegate on iOS via TFLite Swift API
import TensorFlowLite

let coreMLDelegate = CoreMLDelegate()
var options = InterpreterOptions()
options.addDelegate(coreMLDelegate)

let interpreter = try Interpreter(modelPath: modelPath, options: options)
try interpreter.invoke(at: 0)

Choosing between Core ML and Metal: Core ML for production, Metal for edge cases. Core ML automatically manages NE memory, supports CPU fallback, and does not require manual conversion. Metal provides buffer control and is suitable for custom operations. At IT Sectr projects, we use Core ML Delegate for all iOS models and Metal only for real-time video analytics tasks.

How to choose a delegate for your project

Choosing a TF Lite Delegate depends on the target platform, model, and latency requirements. There is no universal best delegate — each has strengths and weaknesses. The optimal strategy: test all available delegates on the target device and choose the minimally fast one — not the fastest, but minimally sufficient.

Recommendations by choice: for Android with NPU (Snapdragon 8 Gen 2+, Dimensity 9000+) — NNAPI Delegate. For Android without NPU — XNNPACK Delegate (default). For iOS — Core ML Delegate. For cross-platform projects, use the strategy: NNAPI on Android, Core ML on iOS, XNNPACK as fallback. GPU Delegate — when NNAPI is unavailable and XNNPACK is not fast enough.

Latency testing is a mandatory selection step. Measure inference at minimum temperature (cold device) and after 5 minutes of continuous operation (thermal throttling). GPU and NNAPI may degrade performance when heated. XNNPACK (CPU) is less affected. Use TensorFlow Lite Benchmark Tool for automatic testing of all delegates on your device.

// Delegate selection strategy
fun selectDelegate(): TfLiteDelegate {
    return when {
        NnApiDelegate.getAvailableAccelerators()?.isNotEmpty == true ->
            NnApiDelegate()
        GpuDelegateFactory.isGpuDelegateAvailable() ->
            GpuDelegate()
        else -> XnnpackDelegate()
    }
}

Fallback strategy — load the model without exceptions: if the delegate is not supported, run on CPU. TensorFlow Lite throws IllegalArgumentException on delegate creation error. Wrap delegate creation in try-catch and run the model without delegate on error. Slow but working AI is better than an error for the user.