Protobuf — what it is, binary data format and how it works

What are Protocol Buffers?

Protocol Buffers is a structured data serialization mechanism developed by Google, similar to JSON and XML, but with a fundamental difference: data is encoded in binary format. This means that a Protobuf message cannot be read with the naked eye, but it takes significantly less space and processes faster than text-based counterparts.

Protobuf was created inside Google to solve performance issues in data exchange between services. In 2008, the technology became an open-source project with support for many languages: C++, Java, Python, Go, JavaScript, Kotlin, Swift, Dart, and others. Version proto3, released in 2016, simplified the syntax and added support for more languages.

The key difference between Protobuf and JSON is the need to define a schema before data exchange begins. The schema file (.proto) describes the message structure: fields, types, and unique field numbers. From this schema, the protoc compiler generates classes in the target language for serialization and deserialization.

Protobuf Ecosystem

• gRPC — high-performance RPC framework that uses Protobuf by default
• protoc — schema compiler from .proto to code in various programming languages
• Google API — most public Google APIs use Protobuf
• Envoy Proxy — uses Protobuf for configuration and data exchange

Protobuf Architecture

Protobuf Architecture includes three key components: the schema definition language (.proto), the protoc compiler, and runtime libraries for specific languages. The developer describes the data structure in a .proto file, runs the compilation, and receives ready-made classes for working with this data.

Each field in a Protobuf message has a unique number (field number) — this is not a sequential number, but a tag used in the binary format to identify the field. Numbers 1 to 15 are encoded in one byte, numbers 16 to 2047 — in two bytes. Therefore, important fields with high usage frequency should be numbered 1 to 15 to optimize size.

Protobuf Data Types

Protobuf supports a wide range of types: scalar (int32, int64, float, double, bool, string, bytes), enums (enum), composite (message), and special types (oneof, map). Each type has a specific binary representation optimized for the corresponding use case.

Advantages of Binary Format

Protobuf provides three key advantages over text-based formats: message size, serialization speed, and strong typing. In high-load systems and mobile applications with limited traffic, these advantages become critical.

Protobuf binary representation uses variable-length encoding (Varint) for numbers: small numbers take 1 byte, large numbers — up to 10 bytes. This allows efficient encoding of identifiers, flags, and counters that would take tens of bytes as text strings in JSON or XML. For example, the number 150 in Protobuf takes 2 bytes, in JSON — 3 bytes (as text “150”), in XML — 3 bytes + tags.

An additional advantage is backward compatibility. Adding a new field to the schema does not break old clients: they simply ignore unknown fields. Deleting a field only requires reserving its number to avoid collisions in the future.

Defining .proto Schema

The .proto schema describes the data structure in a special language. The file starts with specifying the syntax (proto3), package, and import dependencies. Each message is defined with the keyword message with fields, where each field has a type, name, and unique number.

Good schema practices: field numbers 1 to 15 for frequently used fields, meaningful names, grouping related fields into separate message, using oneof for fields that can only be one of several options.

syntax = "proto3";

package mobileapp;

message User {
  string user_id = 1;
  string name = 2;
  string email = 3;
  int32 age = 4;
  UserRole role = 5;
  repeated string tags = 6;
  map<string, string> metadata = 7;
}

enum UserRole {
  USER_ROLE_UNSPECIFIED = 0;
  USER_ROLE_USER = 1;
  USER_ROLE_ADMIN = 2;
  USER_ROLE_MODERATOR = 3;
}

Nested Messages

Messages can be nested: a profile field of type Profile will contain user data inside. Nesting support allows describing complex hierarchical structures without duplicating definitions.

syntax = "proto3";

package mobileapp;

message Order {
  string order_id = 1;
  repeated OrderItem items = 2;
  double total_price = 3;
  PaymentInfo payment = 4;
}

message OrderItem {
  string product_id = 1;
  string title = 2;
  int32 quantity = 3;
  double price = 4;
}

message PaymentInfo {
  string method = 1;
  string transaction_id = 2;
  double amount = 3;
}

Code Generation from .proto

The protoc compiler converts .proto files into source code in the target language. For Kotlin/Java, the --java_out parameter is used, for Swift --swift_out (via Apple Swift Protobuf plugin), for Dart --dart_out. The generated classes contain builder methods for constructing messages and serialization/deserialization methods.

On Android, the com.google.protobuf plugin version 0.9+ in Gradle is used for working with Protobuf. After adding the plugin and specifying the .proto files, the build automatically generates Kotlin classes ready for use in the application.

import com.google.protobuf.kotlin.toByteString
import com.example.mobileapp.UserOuterClass.User

fun createUser(): User {
    return User.newBuilder()
        .setUserId("usr_001")
        .setName("IT Sectr")
        .setEmail("team@itsectr.com")
        .setAge(5)
        .setRole(UserOuterClass.UserRole.USER_ROLE_ADMIN)
        .addTags("mobile")
        .addTags("backend")
        .build()
}

fun serializeAndDeserialize(user: User): User {
    // Serialization to binary format
    val bytes = user.toByteArray()

    // Deserialization from binary format
    return User.parseFrom(bytes)
}

Protobuf Usage Examples

Protobuf is especially effective in microservice architecture and mobile applications. Consider a typical scenario: a mobile application receives a list of products from the server via gRPC. The Protobuf message contains product information including identifier, name, price, and category. The binary format reduces response size by 5-8 times compared to JSON.

gRPC Service in Kotlin

Example of a gRPC service with Protobuf on the server side. The service declares the GetProducts RPC method, which accepts a request with pagination parameters and returns a list of products. The Kotlin implementation uses generated classes for working with requests and responses.

import com.example.mobileapp.ProductServiceGrpcKt
import com.example.mobileapp.ProductOuterClass.Product
import com.example.mobileapp.ProductOuterClass.GetProductsRequest
import com.example.mobileapp.ProductOuterClass.GetProductsResponse

class ProductService : ProductServiceGrpcKt.ProductServiceCoroutineImplBase() {

    override suspend fun getProducts(
        request: GetProductsRequest
    ): GetProductsResponse {
        val products = fetchProductsFromDb(
            page = request.page,
            limit = request.limit
        )

        return GetProductsResponse.newBuilder()
            .addAllProducts(products)
            .setTotalCount(products.size)
            .build()
    }
}

Setting up Protobuf in Android Project

Setting up Protobuf in an Android project is done through the com.google.protobuf Gradle plugin. The plugin automatically runs the protoc compiler during build and generates Kotlin classes from .proto files. To work, you need to add the plugin to the project-level build.gradle and apply it in the application module.

After configuring Gradle, .proto files are placed in the src/main/proto directory. The protoc compiler processes them during each build, generating Kotlin classes that can be used in the application code. It is important to correctly specify the protobuf and protoc versions to avoid dependency conflicts with other project libraries.

For iOS projects, Protobuf is connected via CocoaPods or Swift Package Manager. The Swift Protobuf plugin automatically generates Swift structures conforming to the Codable protocol. In Dart projects for Flutter, the protobuf package is used, and compilation is done via dart run protoc_plugin.