AVFoundation — What It Is, Audio and Video API in iOS

What is AVFoundation?

AVFoundation is Apple's foundational multimedia framework, introduced back in iOS 4 (2010) as a replacement for the outdated UIWebView for audio and video playback. Since then, AVFoundation has evolved into a full-featured set of tools for working with all types of media: from simple MP3 file playback to professional video capture with a multi-channel microphone and sound transformation through real-time effects.

The AVFoundation architecture is built around several core abstractions: AVAsset (media data), AVPlayer (playback), AVCaptureSession (capture), and AVAudioEngine (processing). Each component operates asynchronously, which is critical for maintaining UI smoothness during loading or processing of large files. According to Apple (WWDC 2025), AVFoundation handles up to 8K video on devices with M-series chips without performance loss.

Key Capabilities

AVFoundation provides video capture from the camera with support for 4K ProRes, HDR, and up to 240 FPS in slow-motion, audio recording and mixing from built-in and external microphones, media playback with support for HLS, MPEG-4, QuickTime, MP3, AAC, and FLAC, file export and conversion via AVAssetExportSession, real-time audio processing through AVAudioEngine with reverb, equalizer, and pitch-shifting effects, as well as streaming via AirPlay and network protocols.

Key Components of AVFoundation

AVFoundation consists of several modules, each responsible for a specific area of media work. The choice of component depends on the task: capture, playback, export, or real-time processing.

AVCaptureSession — Camera Capture

AVCaptureSession is the central object for video and audio capture. It manages incoming data streams from physical devices (camera, microphone), configures their settings (resolution, FPS, format), and directs them to output: preview (AVCaptureVideoPreviewLayer), file recording (AVCaptureMovieFileOutput), or frame-by-frame processing (AVCaptureVideoDataOutput).

AVPlayer — Media Playback

AVPlayer is a universal AVFoundation player capable of playing media content both locally and over the network. AVPlayer supports HLS streaming, MPEG-4, QuickTime Movie, and most audio formats. Playback control is handled asynchronously through AVPlayerItem. AVPlayerViewController provides a ready-made UI with controls: play/pause, scrubbing, AirPlay, and PiP.

Capturing Media with AVFoundation

Video and audio capture is one of the most in-demand features of AVFoundation. Setting up a capture session involves several steps: creating AVCaptureSession, adding inputs (camera, microphone), configuring outputs, and starting the session.

Setting up AVCaptureSession

Let's walk through a step-by-step setup of AVCaptureSession for capturing video from the iPhone rear camera while simultaneously recording sound from the microphone. The process consists of four steps in Swift.

import AVFoundation

// 1. Creating and configuring the session
let session = AVCaptureSession()
session.sessionPreset = .hd1920x1080

// 2. Adding camera input
guard let camera = AVCaptureDevice.default(
    for: .video
) else { return }
let cameraInput = try? AVCaptureDeviceInput(device: camera)
guard let input = cameraInput,
      session.canAddInput(input)
else { return }
session.addInput(input)

// 3. Adding output for recording
let output = AVCaptureMovieFileOutput()
guard session.canAddOutput(output) else { return }
session.addOutput(output)

// 4. Starting the session
session.startRunning()

The Swift code creates an AVCaptureSession with a Full HD preset, connects the rear camera as a video source, and adds an output for file recording. After session.startRunning() is called, the camera begins transmitting frames to the preview and output file. Recording is stopped via output.stopRecording() with the option to save to the gallery.

For audio-only capture (without video), only the microphone is added to AVCaptureSession via AVCaptureDevice.default(.audio). The output is AVAudioFileOutput or AVCaptureAudioDataOutput for real-time sound processing. AVFoundation supports multi-channel capture: up to 8 audio channels on iPad Pro and up to 2 on iPhone, allowing stereo recording from external microphones via Lightning, USB-C, or Bluetooth. Input source selection is done through AVAudioSession.sharedInstance().setPreferredInput, which is critical for applications working with external headsets and Bluetooth microphones.

Playing Media

AVFoundation provides two primary ways to play media: AVPlayer for video and AVAudioPlayer for audio. Let's look at an example of playing a remote HLS stream using AVPlayer.

AVPlayer with HLS Example

The code creates an AVPlayer, loads an HLS stream from a URL, and starts playback at a reduced volume level. AVPlayerItem manages the loading and buffering state, and KVO observation of the status allows handling errors.

import AVFoundation
import AVKit

// 1. Creating AVPlayer with HLS stream
let url = URL(
    string: "https://example.com/stream.m3u8"
) !
let player = AVPlayer(url: url)
player.volume = 0.7

// 2. Playback
player.play()

// 3. Displaying the player
let playerVC = AVPlayerViewController()
playerVC.player = player
present(playerVC, animated: true)

// 4. Monitoring status
player.currentItem?.addObserver(
    self,
    forKeyPath: "status",
    options: .new,
    context: nil
)

The code creates an AVPlayer with an HLS URL, sets the volume to 70%, and starts playback. AVPlayerViewController provides a built-in UI with controls, AirPlay, and picture-in-picture mode. The player status is monitored via KVO to handle cases when the HLS stream is temporarily unavailable.

For audio playback without a video interface, AVAudioPlayer is used — a lightweight player that does not require AVPlayerItem. AVAudioPlayer supports audio formats MP3, AAC, ALAC, FLAC, WAV, and AIFF, provides play(), pause(), and stop() methods, as well as the AVAudioPlayerDelegate for tracking playback completion and decoding errors. AVAudioPlayer is ideal for background music playback, podcasts, and sound effects in games on iOS and macOS.

Additional Code Examples

Let's look at two more typical scenarios for working with AVFoundation: audio processing via AVAudioEngine and video export to another format.

Audio Processing via AVAudioEngine

AVAudioEngine enables real-time audio processing by connecting nodes in a processing graph: input node (microphone) → effects → output node (speaker). The example below adds reverb to the voice.

let engine = AVAudioEngine()
let reverb = AVAudioUnitReverb()

// Setting up reverb
reverb.loadFactoryPreset(.largeHall)
reverb.wetDryMix = 40

// Connecting: microphone -> reverb -> speaker
engine.attach(reverb)
engine.connect(
    engine.inputNode,
    to: reverb,
    format: nil
)
engine.connect(
    reverb,
    to: engine.mainMixerNode,
    format: nil
)

try? engine.start()

AVAudioEngine creates a graph of three nodes: input (microphone), effect (reverb with a large hall preset), and output (speaker). After engine.start(), the microphone audio is processed by the effect in real time and output to the device's headphones or speaker. Processing latency is under 10 ms on all modern Apple devices.

An additional capability of AVAudioEngine is sound analysis via AVAudioUnitEQ and AVAudioUnitMixer. The developer can add an equalizer, compressor, and limiter to any audio stream. For music applications, AVFoundation provides AVAudioSequencer for MIDI file playback and AVSpeechSynthesizer for speech synthesis. All components work in a single audio graph, providing latency between capture and processing of under 5 ms on Apple devices.

When working with AVFoundation, it is important to consider system permissions: accessing the camera requires NSCameraUsageDescription in Info.plist, and the microphone requires NSMicrophoneUsageDescription. AVFoundation automatically shows the system permission dialog on the first attempt to access a device. User denial is handled via AVCaptureDevice.authorizationStatus(), allowing a proper message about camera unavailability with a suggestion to go to Settings.

AVFoundation supports background audio playback through the Audio Background Mode in the project Capabilities. To activate it, simply add the audio entry in UIBackgroundModes Info.plist and configure AVAudioSession with the .playback category. Video does not play in the background by default, but you can use AVPlayer with picture-in-picture mode to continue video playback in a minimized app on iPad. The PiP mode icon is configured via AVPictureInPictureController along with the protocol delegate.