It Works on Prod: What It Is, Why It Happens and Why It’s Dangerous

What Does “It Works on Prod” Mean

“It works on prod” is a set phrase among developers that describes a situation where code functions on the production server but refuses to work in a test environment or on a colleague’s local machine. On the surface it sounds like “there is no problem,” although in reality the problem does exist — it just cannot be reproduced in the production environment. The root of the discrepancy lies in differences in configurations, versions, and data between environments.

The phrase was born as the antithesis of another well-known excuse — “It works on my machine.” If a developer says “it works locally,” the bug only exists for others. But if “it works on prod,” the bug only appears on staging or a test environment, while production is clean. The irony of fate: in both cases the problem is real, it just doesn’t manifest for the person looking at it. According to the DevOps Research and Assessment (DORA) 2023 study, teams with a high level of deployment automation encounter such discrepancies 3 times less often.

From a business perspective, the “it works on prod” situation is more dangerous than it seems. If a bug exists on staging but not on prod, a developer might ignore it — and then with the next deployment the error will reach production. Temporary relief turns into a future problem that will have to be fixed under user pressure.

Why Developers Say “It Works on Prod”

The psychological reason behind the phrase’s persistence is a defense reflex. A developer who sees a bug on staging but not on production may subconsciously downplay the problem: “everything is fine in production, so it’s not urgent.” A classic cognitive bias — survivorship bias, where the visible success of production outweighs the potential threat of a future failure.

The second reason is blurred responsibility. If production works but staging doesn’t, the environment is blamed, not the code. The developer shifts responsibility for the bug onto a DevOps engineer or admin. According to the Atlassian State of DevOps 2022, in teams without a unified deployment environment (Docker, Kubernetes), such blame-shifting occurs 60% more often.

The third reason is fear of a zero-downtime release. If a developer fixes the bug on staging and rolls out the fix, it will require another code review, testing, and deployment. The phrase “it works on prod” allows postponing the fix until the next release, reducing the current workload. Deferred fixes are one of the main causes of technical debt accumulation in teams.

Difference Between Development and Production Environments

Production and staging are never completely identical — this is technically impossible due to differences in scale, load, and data. However, key parameters must match: the operating system version, compiler, interpreter, database, web server, and all project dependencies. If even one parameter differs, the code’s behavior can change.

The main differences between environments include:

• Hardware — processor, RAM size, disk type (SSD vs HDD) can affect timings and multithreading performance
• Network environment — firewall, DNS, proxy, load balancers are only present in production
• Database data — staging usually has test data, while real user records have unexpected patterns
• Dependency versions — even a minor library update can change code behavior
• Environment variables — API keys, tokens, feature flags can differ between environments

Containerization solves most of these problems. The same Docker image built for production should be used on staging as well. The only differences are environment variables and volume mounts. According to the Docker State of Application Development 2023, teams using a single image across all environments reduce discrepancies by 74%.

Typical Causes of Production Behavior Mismatch

The first and most common cause is different dependency versions. A developer installs a package locally with the --save flag but forgets to update package.json or the lock file. During production deployment, a different version is installed that behaves differently. For the npm ecosystem, the lock file completely solves the problem; for other package managers, analogous mechanisms exist (Gemfile.lock, Podfile.lock, pubspec.lock).

The second cause is missing or extra environment variables. A developer uses a .env file on their local machine but does not add the corresponding variables to the CI/CD pipeline or the server. As a result, the code crashes with an API or database connection error. According to the GitLab DevSecOps Survey 2023, 27% of production incidents are related to incorrect environment variables.

The third cause is database state. On staging, the database may contain records that do not exist in production, or conversely — migrations may be missing. A typical scenario: a developer writes code that works with a new table field, but the migration has not yet been applied to production. A migration strategy with backward compatibility is the only way to avoid such situations.

The fourth cause is regional and language settings. Date formatting, decimal separators, text encoding — all of these can differ between a developer’s local machine and the server. This is especially relevant for projects with internationalization. The solution is to explicitly specify the locale in the application configuration and not rely on system settings.

How to Diagnose the “It Works on Prod” Problem

The first step is to compare logs from both environments. A difference in logging level often hides the cause: production may have INFO enabled while staging has DEBUG. Configure the same logging level and ensure both environments write in a format that allows machine comparison. Use centralized log collection systems — Sentry, Datadog, ELK Stack.

The second step is to check dependency versions. Compare lock files, list installed packages on both environments. A difference in a minor or patch version is the most likely cause of the discrepancy. Tools like npm ls, pip freeze, mvn dependency:tree help quickly identify mismatches.

The third step is to reproduce the production environment locally. Use Docker Compose or similar tools to spin up an exact copy of the production infrastructure. If the bug reproduces in a local container, the problem is in the code, not the environment. If it does not reproduce, look for a configuration difference.

The fourth step is to check feature flags and A/B tests. Perhaps the code runs in a different mode in production because the wrong flag is enabled. According to the LaunchDarkly State of Feature Management 2023, up to 40% of unexpected behavior in production is related to incorrect feature flag values. A unified flag manifest for all environments solves this problem.

Preventing Environment Discrepancies in a Project

The main prevention tool is Infrastructure as Code (IaC). All environments should be described in code: Dockerfile, docker-compose.yml, Terraform scripts, or Ansible playbooks. Manual changes on the server are prohibited — any configuration change goes through the repository and code review. This ensures that all environments have the same configuration.

The second most important tool is a unified CI/CD pipeline. The same build, test, and deployment script should be used for all environments. The only difference is in target variables (URLs, keys). If the pipeline for staging and production differs in steps, discrepancies are inevitable.

The third tool is automatic data synchronization. Regularly (daily or on a schedule) update staging with an anonymized copy of the production database. This allows testing code on real data rather than synthetic fixtures. Tools: pg_dump/pg_restore for PostgreSQL, mysqldump for MySQL, specialized services like DataGrip.

The fourth tool is discrepancy monitoring. Set up alerts when differences between staging and production are detected. A simple script comparing hashes of configuration files or versions of installed packages will save hours of debugging. Prevention is always cheaper than diagnosis: preventing environment discrepancies requires less effort than finding the cause of the “it works on prod” bug.