Request Deduplication: what it is, methods and working mechanisms

What is Request Deduplication?

Request Deduplication is a technique that prevents executing multiple identical requests to a single data source within the same time window. Instead of sending 10 identical HTTP requests, the system sends one, while the other 9 wait for its result.

The problem of duplicate requests is especially acute in mobile applications with state-based architecture (MVVM, MVI, Redux). When multiple observers subscribe to the same data within a short period, each triggers its own request, creating redundant load. According to Uber Engineering (2024), up to 18% of all requests in Uber mobile clients are duplicates, and client-side deduplication reduced their number by 4 times.

Deduplication is not the same as caching. Cache stores the request result after execution. Deduplication prevents redundant requests before and during their execution. After the request completes, caching takes over.

class DeduplicatorT(
    private val source: suspend () -> T
) {
    private val inFlight = ConcurrentHashMap<String, Deferred<T>>()

    suspend fun get(key: String): T = inFlight.getOrPut(key) {
        async {
            source().also { inFlight.remove(key) }
        }
    }.await()
}

This Kotlin class guarantees that only one coroutine runs per key. All concurrent calls with the same key await a single Deferred. After completion, the key is removed, and the next request executes normally.

Why Deduplication Is Needed in Mobile Applications

Reduced server load is the first and most obvious reason. Each duplicate request consumes server resources: CPU, memory, database connections. At the scale of millions of devices, even 10–15% duplicate requests create significant load, requiring additional servers.

Reduced battery and data usage — each HTTP request on a mobile device consumes radio module energy. According to Google I/O (2025), a single failed or duplicate request can consume up to 15% of one network session's energy. Deduplication reduces the number of radio module activations, extending device battery life.

Avoiding data conflicts — if two duplicate requests write data to local storage, race conditions may occur: the second request could overwrite the first result with stale data. Deduplication guarantees that the write to local storage happens only once, eliminating races.

Improved UX — the user does not see multiple loading indicators for the same data. UI state (loading / success / error) is managed by a single source of truth rather than multiple competing requests.

Memoization — In-Memory Caching

Memoization is caching a function's result during its execution. If a function is already running with the same arguments, a new call does not start a second process but receives the result of the first. This is the simplest form of deduplication for in-process scenarios.

A typical implementation in mobile applications is a HashMap of keys to Deferred or Promise. The key is usually the request URL string or a concatenation of parameters. The entry lifetime is from the first request until the response completes. According to Dropbox Engineering (2024), memoization in the Dropbox mobile client reduced duplicate API requests by 40%.

Flawed deduplication — a dangerous mistake: if the key is not removed after an error, all subsequent requests will forever return the same error. A correct implementation must handle Error and Failure, clearing the cache and allowing retry.

class MemoizedLoaderT(
    private val loader: suspend () -> T
) {
    private var cachedResult: Result<T>? = null

    suspend fun get(): T = cachedResult ?.getOrThrow() ?: run {
        loader().let {
            Result.success(it)
        }.also { cachedResult = it }
    }.await()
}

MemoizedLoader uses Result<T> for correct error handling: on success — caches, on error — allows retry. This approach ensures that a temporary network failure does not block subsequent requests.

Request Merging — Batch Combining

Request Merging is a technique where multiple different requests to the same source are collected into a group and sent as a single batch request. Unlike deduplication, the requests here are not identical — they differ in parameters but address the same resource.

A typical scenario: 5 application screens request profiles of different users. Instead of 5 individual requests to /api/users/1, /api/users/2, etc., the system waits 20 ms, collects all IDs, and sends one request /api/users?ids=1,2,3,4,5. Window timeout is the key parameter: too long a window hurts UX, too short — fails to collect enough requests.

According to Netflix Engineering (2023), in the GraphQL aggregator BFF (Backend for Frontend), request merging reduced the number of HTTP calls between layers by 65% and average response time by 120 ms by eliminating extra RTTs. Async window (debounce) is the standard implementation via coroutines or RxJava.

class BatchMergerT {
    private val pending = ConcurrentLinkedQueue<Pair<String, CompletableDeferred<T>>>()

    suspend fun get(id: String): T = suspendCoroutine { cont ->
        pending.add(Pair(id, cont))
        scheduleFlush()
    }
}

This mixin uses suspendCoroutine to suspend each request and a 30 ms window to collect the group. After the timer expires, all collected IDs are sent in a single batch request, and each coroutine receives its result.

Server-Side Deduplication via DataLoader

DataLoader is a library (originally for JavaScript/GraphQL) that implements batching and memoization on the server side. It groups all requests to the same data source within a single event loop tick and executes them with one call. DataLoader is widely used with GraphQL but can be applied in any REST application.

How it works: all loader.load(id) calls within a single microtask are collected into an array of IDs and passed to the batch function. After receiving results, each ID gets its array element. Caching in DataLoader works only within a single HTTP request — on the next request the cache is cleared, ensuring data freshness.

According to Meta Engineering (2024), implementing DataLoader in Facebook's GraphQL layer eliminated the N+1 problem, reducing database queries from 200 to 10 per typical page. Batch scheduling — DataLoader's key innovation — uses process.nextTick (Node.js) or DispatchQueue.main (iOS) to optimize grouping.

Which Deduplication Strategy to Choose

Memoization is optimal for a single process (mobile app, microservice). Simple to implement and effective for identical parallel calls. The downside is it does not work across processes or devices.

Request Merging is suitable for the BFF layer or aggregator service. Requires batch endpoint support on the server. Best choice when the frontend makes many small requests for different data of the same type.

DataLoader is the standard for GraphQL servers. It automatically solves the N+1 problem and does not require manual cache configuration. Recommended for any server with a GraphQL layer.

HTTP cache with deduplication — at the OkHttp (Android) or URLSession (iOS) level, deduplication can be configured via Interceptor or delegate. OkHttp CacheInterceptor is a custom interceptor that checks if a request with the same URL is already in progress and merges them. This method operates below the business logic level and covers all application requests without changing feature code.