Decoding in Mobile Apps: What It Is, Basic Concepts, and How It Works

What Is Decoding?

Decoding is the process of converting compressed digital data back into its original uncompressed format. In the context of media, decoding restores video frames from a compressed bitstream created by an encoder. Without decoding, users cannot see video or hear audio, as all modern media formats use compression to save bandwidth and disk space.

A typical video stream in H.264 format at 5 Mbps takes up 100 times less space than an uncompressed RGB stream at the same resolution. The decoding algorithm must restore each frame back to its original resolution and color space, following the codec specification in reverse order relative to encoding. To do this, the decoder processes intra-frame (I-frame) and inter-frame (P-frame, B-frame) data.

On mobile devices, decoding can occur either on the CPU or on dedicated hardware blocks. Modern SoCs from Apple (A series), Qualcomm (Snapdragon), and MediaTek (Dimensity) contain built-in decoders for all popular formats. The video processor handles the heavy work of inverse discrete cosine transform and motion compensation, freeing the CPU for other tasks.

How Does Decoding Work?

The decoding process consists of several sequential stages that invert the encoding steps. First, headers and compression parameters — profile, level, resolution, color space — are extracted from the bitstream. Then the decoder sequentially processes compressed macroblocks, applying inverse transforms to them.

Video Stream Decoding Stages

The first stage is entropy code extraction. Entropy decoding uses CABAC or CAVLC algorithms to restore the discrete cosine transform coefficients. This stage does not depend on video resolution — it processes a stream of bits, not pixels, and its complexity is determined by bitrate, not frame dimensions.

The second stage is inverse quantization and inverse DCT. The decoder multiplies the quantized coefficients by the quantization step, restoring approximate DCT coefficient values, and then applies the inverse DCT transform. Inverse DCT restores spatial data from the frequency domain, forming a macroblock of pixels. The transform is performed independently for chroma and luma.

The third stage is motion compensation. For P- and B-frames, the decoder uses motion vectors extracted from the bitstream and references previously decoded reference frames. Motion compensation creates a predictor for the current macroblock, to which the residual signal after inverse DCT is added. The result is a fully restored frame ready for output.

// Basic video frame decoding pseudocode
struct DecodedFrame {
    uint8_t* y_plane;
    uint8_t* u_plane;
    uint8_t* v_plane;
    int width, height;
};

class Decoder {
public:
    bool decodeNALUnit(const uint8_t* nalUnit, size_t size) {
        if (!parseNALUHeader(nalUnit, size))
            return false;
        
        int sliceType = parseSliceType(nalUnit);
        entropyDecode(nalUnit);
        inverseQuantize();
        inverseDCT();
        
        if (sliceType != I_SLICE)
            motionCompensation();
        
        return true;
    }
};

The example above shows the basic structure of an H.264 decoder. The decodeNALUnit function takes a NAL unit — the basic block of the compressed H.264 stream. The decoder sequentially parses the header, extracts the slice type, applies entropy decoding, inverse quantization, and inverse DCT. For P- and B-slices, motion compensation is additionally performed using reference frames from the DPB buffer.

Compression Formats and Codecs

Modern video codecs differ in compression algorithms, efficiency, and computational resource requirements. Format choice directly affects file size, image quality, and power consumption during decoding on a mobile device.

H.264 is the most widespread video codec, supported by all mobile devices. Its main advantage is universality: any Android smartphone and iPhone can decode H.264 in hardware. However, at the same bitrate, H.264 loses in quality to more modern codecs like H.265 and AV1, requiring 30–50% more bitrate for similar visual quality.

H.265 provides twice the compression compared to H.264 at the same quality. H.265 decoding requires a more powerful hardware block: VideoToolbox on iOS supports H.265 starting from iPhone 6 (A8), and Android devices from Snapdragon 805 and above. When choosing H.265 for a mobile app, keep in mind that older devices may lack hardware support and will decode this format in software, sharply increasing power consumption.

AV1 is an open codec from the Alliance for Open Media, providing the best compression among all modern formats. AV1 is 30% more efficient than H.265 and 50% more efficient than H.264 at the same visual quality. Hardware AV1 decoding appeared only in SoCs from 2023+: Apple A17 Pro, Qualcomm Snapdragon 8 Gen 2 and newer. For older devices, AV1 decoding is only possible via software through the dav1d library, creating a significant CPU load.

Software vs Hardware Decoding

The choice between software and hardware decoding is a key architectural decision when developing a mobile media player. Each approach has its advantages and limitations that must be considered when designing the application.

Performance and Power Consumption

Hardware decoding is performed on specialized video processing blocks that consume significantly less energy than the CPU when performing the same task. According to Qualcomm, a hardware H.265 decoder consumes 5–10 times less energy than software decoding on a Snapdragon 8 Gen 1 CPU when playing 4K video. This is critical for mobile devices where every milliwatt affects battery life.

Software decoding, on the other hand, provides maximum flexibility. FFmpeg with the libavcodec library supports dozens of codecs and containers, including rare and obsolete formats that lack hardware support. The developer can modify the decoding pipeline, add post-processing and filters on the fly, which is impossible when using closed hardware blocks.

When to Choose Software Decoding

Software decoding is justified in several scenarios: when playing rare formats (ProRes, DNxHD, Motion JPEG), when precise control over each stage of frame processing is required, and when decoding AV1 on devices without hardware support. libavcodec from FFmpeg can decode virtually any known format, making it the de facto standard for universal media players.

The limitation of software decoding is heat dissipation. Continuous 4K video decoding on the CPU can heat the device to 45–50 degrees within 10–15 minutes, leading to throttling and frame rate reduction. On devices without active cooling (tablets, phones), this is especially noticeable. CPU power consumption during software decoding can reach 3–5 W versus 0.3–0.5 W during hardware decoding of the same stream.

When to Choose Hardware Decoding

Hardware decoding is the default choice for any production media player. It provides stable 60 fps for 4K video at minimal power consumption. VideoToolbox on iOS and MediaCodec on Android provide native APIs for hardware decoding that automatically select the optimal processing block depending on the codec and resolution.

Platform APIs handle frame buffer management (surface pool on Android, CVPixelBufferPool on iOS), display synchronization, and memory optimization. The developer simply needs to open a decoder with the required parameters and receive ready frames. Hardware decoding supports a low-latency end-to-end pipeline: from receiving the bitstream to displaying on screen takes 5–15 ms versus 30–80 ms for software decoding.

Decoding Code Examples

Let us look at a practical implementation of decoding on both mobile platforms. On iOS, hardware decoding is done through VideoToolbox, and software decoding through FFmpeg. On Android, MediaCodec is used for hardware decoding.

Hardware Decoding on iOS with VideoToolbox

@interface VideoDecoder ()
@property (nonatomic) VTDecompressionSessionRef session;
@end

@implementation VideoDecoder

- (void)setupDecoder {
    CMVideoFormatDescriptionRef formatDesc;
    CMVideoCodecType codecType = kCMVideoCodecType_H264;
    
    OSStatus status = CMVideoFormatDescriptionCreate(
        NULL, codecType, 1920, 1080, NULL, &formatDesc
    );
    
    VTDecompressionOutputCallbackRecord callback;
    callback.decompressionOutputCallback = &decodingCallback;
    
    VTDecompressionSessionCreate(NULL, formatDesc, NULL,
        NULL, &callback, &_session);
}

- (void)decodeFrame: (uint8_t*)nalData length:(size_t)size {
    CMBlockBufferRef blockBuffer;
    CMBlockBufferCreateWithMemoryBlock(NULL, nalData,
        size, NULL, NULL, 0, size, 0, &blockBuffer);
    
    CMSampleBufferRef sampleBuffer;
    CMSampleBufferCreate(NULL, blockBuffer, true, NULL,
        NULL, NULL, 1, 0, NULL, 0, NULL, &sampleBuffer);
    
    VTDecompressionSessionDecodeFrame(_session,
        sampleBuffer, 0, NULL, 0);
}

@end

The code demonstrates initialization of an H.264 hardware decoder on iOS. VTDecompressionSessionCreate creates a decoding session, and VTCreate calls a callback when a ready frame appears. The session automatically uses the hardware block if available for the specified codec. To obtain decoded frames in CVPixelBuffer format, a callback is used that passes each ready frame with minimal delay.

Hardware Decoding on Android with MediaCodec

MediaCodec decoder = MediaCodec.createDecoderByType("video/avc");
MediaFormat format = MediaFormat.createVideoFormat(
    "video/avc", 1920, 1080
);
format.setInteger(MediaFormat.KEY_FRAME_RATE, 30);

decoder.configure(format, surface, null, 0);
decoder.start();

ByteBuffer[] inputBuffers = decoder.getInputBuffers();
int inputIndex = decoder.dequeueInputBuffer(10000);

if (inputIndex >= 0) {
    ByteBuffer buffer = inputBuffers[inputIndex];
    buffer.clear();
    buffer.put(nalData);
    decoder.queueInputBuffer(inputIndex, 0, nalData.length, pts, 0);
}

On Android, MediaCodec uses a Surface for output rather than a pixel buffer, minimizing data copying between GPU and CPU. The decoder automatically selects the hardware block (OMX component) based on the codec type. For H.264, OMX.google.h264.decoder is used, which can be either hardware or software depending on the manufacturer's implementation.

How to Choose a Decoder for a Mobile App

Choosing a decoding strategy depends on the application's target audience, supported formats, and performance requirements. The optimal solution often involves a hybrid approach: hardware decoding for mainstream formats (H.264, H.265) with a software fallback for rare codecs.

Priority-Based Selection Strategy

If the application is aimed at maximum compatibility — use H.264, which is guaranteed to decode in hardware on any device. For video streaming services, H.265 with hardware support on devices after 2016 is justified. AV1 is the choice for services where bandwidth savings are important: YouTube, Netflix, and other major platforms are actively adopting AV1 to reduce CDN costs while maintaining quality.

A critical parameter is the decoder's buffer size. Hardware decoders have a fixed buffer pool (typically 4–16 frames). When playing a high-bitrate stream, buffers can overflow, leading to frame drops. MediaCodec provides the getOutputFrameRate method to determine the actual decoder performance on a specific device, while VideoToolbox allows controlling realtime priority via kVTDecodeFrame_EnableAsynchronousDecompression.

Thermal throttling is another factor. Even hardware decoding can heat up the device during prolonged 4K HDR video playback. It is recommended to monitor temperature through ProcessInfo on iOS and BatteryManager on Android, lowering quality or stream resolution when overheating occurs. This is especially critical for games and streaming applications with long viewing sessions.