Case Study: The $1,172.80 OCR Incident

This case study is intentionally direct. Its purpose is not to blame someone for choosing the wrong AWS feature. Its purpose is to show how quickly a manageable technical mistake becomes an expensive operational incident when ownership, verification, and humility disappear at the same time.

Incident summary

A Project Technical DRI created a personal AWS account to avoid waiting for client-provided credentials, despite the Technical Delivery Lead (TDL) explicitly saying not to proceed that way.

The project was deliberately migrating only seven registered end clients. Files belonging to unregistered clients were expected to fail with a retryable client not found error in Restate until each client was intentionally registered. That failure was not a bug; it was the rollout boundary.

Agent-generated code changed that behavior. When a file arrived for an unknown client, the system created the client automatically. The DRI did not identify the scope expansion during review, and hundreds of end clients entered the processing path. Approximately 7,000 documents, totaling 23,456 pages, were sent to Amazon Textract before a monthly bill of $1,172.80 exposed the problem.

When the TDL had asked whether the implementation used the cheapest OCR option, the DRI answered yes. Investigation later showed that requests enabled the FORMS feature. FORMS was the most effective capability for the requirement; choosing it was not the mistake. The concern was that the DRI had not verified the request, understood its cost, or known the code well enough to explain why that capability was in use.

Intended system behavior

The phased rollout existed to constrain uncertainty:

• Only seven end clients were registered and eligible for processing.
• Files for unknown clients returned a retryable client not found error.
• Restate retained and retried that work until the client was deliberately registered.
• Issues could therefore be discovered and fixed within a controlled seven-client blast radius.

Automatic client creation silently removed every one of those safeguards. It converted an explicit migration decision into an open-ended ingestion system.

The retryable error was part of the control plane. Treating it as an inconvenience to eliminate changed the business behavior, not merely the error handling.

Timeline

1. A decision boundary was ignored. The DRI proposed creating an AWS account instead of waiting for the client’s credentials. The TDL said no. The DRI proceeded anyway.
2. The rollout invariant was removed. Agent-generated code created new client records when unknown-client files arrived.
3. The change passed without adequate human review. The DRI did not trace how an incoming file could create durable data, expand processing scope, and trigger a metered external service.
4. The deployed behavior was not understood. The DRI said the cheapest Textract option was in use without verifying the deployed request. FORMS was enabled and was appropriate for the requirement, but its cost and operational controls were not understood.
5. No cost containment caught the expansion. There was no effective cache, spend guardrail, or early alert capable of stopping repeated or unexpectedly broad OCR processing.
6. The invoice became the monitoring system. About one month later, a $1,172.80 AWS bill revealed that approximately 7,000 documents and 23,456 pages had been processed.

Impact

• Hundreds of end clients entered a rollout intentionally limited to seven.
• Approximately 7,000 documents and 23,456 pages incurred OCR charges.
• The AWS bill reached $1,172.80.
• The team had to investigate behavior that should have been understood before deployment.
• Trust was damaged because a direct instruction was ignored and a cost-related answer was given without evidence.

The financial impact is the most visible consequence, but it is not the most important one. The system stopped reflecting the agreed migration strategy, and nobody noticed until an external invoice arrived.

AWS billing detail for the incident: 23,456 Textract Forms pages produced a $1,172.80 charge.

Cost comparison and true counterfactual

For the same 23,456 pages, plain text OCR at $0.005 per page would have cost:

23,456 pages × $0.005 = $117.28

The $1,055.52 difference is the price of comparing two different capabilities. It is not automatically avoidable spend: plain text OCR did not satisfy the requirement as effectively as FORMS.

The correct counterfactual is FORMS processing for only the seven intentionally registered end clients, with one paid extraction per unique document, caching across workflow retries, and cost thresholds capable of stopping an unexpected expansion. That amount should have been substantially lower than $1,172.80.

The precise avoidable amount cannot be calculated from the invoice alone. It requires identifying pages from clients outside the approved rollout, duplicate extractions that a cache should have prevented, and processing that should have stopped after an alert or budget threshold.

FORMS itself was not the incident. It was the right capability. The failure was running it across uncontrolled scope without sufficient caching, spend limits, or observability—and not knowing the deployed code and cost model well enough to recognize those risks.

Root cause

The root cause was a failure of technical ownership: the DRI made and deployed decisions without maintaining an accurate understanding of the system’s behavior, rollout boundaries, or cost model.

Several failures compounded:

Governance failure

An explicit “do not create the account” decision was ignored. Speed does not authorize a DRI to cross an account, credential, security, billing, or architectural boundary.

Review failure

Agent-generated code was treated as implementation rather than a proposal requiring engineering judgment. The reviewer did not identify that the code created durable client records and expanded the production workload.

Domain failure

The expected client not found retry was misunderstood as a defect rather than the mechanism enforcing phased migration.

Cost-awareness failure

The DRI did not inspect the actual Textract request, calculate expected spend, or explain why the selected capability justified its price before answering the TDL. The correct feature still requires a cost model and operational guardrails.

Resilience failure

An expensive operation had no sufficient cache or idempotency protection. Retryable workflows and file synchronization can amplify paid calls; a costly operation must not be repeated merely because the surrounding workflow retries.

Observability failure

The project lacked an effective budget alarm, usage threshold, and processing-volume alert. A monthly bill should never be the first signal of runaway metered work.

The core values applied

Fair

Fairness means not transferring the cost of an avoidable, unauthorized decision to the client or silently asking the company to absorb it. It also means reporting what happened accurately, including which part of the spend was expected, which part was avoidable, and who controlled the decisions that produced it.

Effective

Effectiveness means protecting the intended seven-client rollout, choosing the capability that best satisfies the requirement, and then controlling its cost through caching, limits, and observability. Shipping quickly while multiplying scope and paid work is not effective delivery.

Humble

Humility means accepting that agent output can be wrong, that your first interpretation may be incomplete, and that an answer about deployed behavior requires evidence. A humble engineer opens the request payload, checks the pricing page, traces the retry path, adds the cache, and says “I need to verify” instead of guessing.

Core-value exam question

A DRI ignored an explicit instruction not to create an AWS account, deployed unreviewed behavior that expanded a seven-client rollout to hundreds, could not accurately explain the deployed OCR mode or its cost controls, and incurred a $1,172.80 bill. What is the fair thing to do with the bill?

A strong answer starts with ownership, not negotiation. The client should not pay for avoidable spend caused by an unauthorized account and an uncontrolled rollout. The DRI should disclose the incident immediately, provide an honest cost breakdown, and proactively offer to take responsibility for the avoidable cost rather than presume that the company will reimburse it.

The exact financial resolution must follow the applicable contract, employment law, and a leadership decision; it must not be improvised through an unlawful wage deduction. Fairness is demonstrated first by the person responsible volunteering the complete truth and accepting consequences—not by minimizing the mistake, blaming the agent, or treating the bill as a routine project expense.

Required engineering behavior

Before enabling any metered external service:

1. Use the client-owned or company-approved account and credentials. Do not create a substitute account after approval has been denied.
2. Document the unit price, expected volume, average units per item, and projected monthly cost.
3. Inspect the exact deployed request and confirm that every paid feature is required.
4. Set budget alarms and usage alerts low enough to catch mistakes during the rollout, not after the monthly invoice.
5. Enforce the rollout population explicitly. Prefer an allowlist or validated state transition over relying only on a downstream failure.
6. Cache costly results using a stable input or content hash, and make paid operations idempotent across retries.
7. Alert on unexpected changes in client count, document volume, pages processed, retry volume, and external-service spend.
8. Test that an unregistered client remains unregistered and cannot reach paid processing.

Review questions for agent-generated code

The human reviewer must be able to answer these questions before merging:

• Can this change create durable records?
• Can it expand the number of clients, users, files, jobs, or billable requests?
• Does it turn an expected failure into automatic success?
• Can retries repeat a paid or destructive operation?
• Is the result cached or idempotent?
• Which external-service features are enabled in the actual request?
• What does this cost at expected volume, and what does it cost if volume is 10× higher?
• Which alert will fire before the spend becomes material?

If the DRI cannot answer from the code and current documentation, the change is not ready to merge.

Final lesson

The incident was not caused by one surprising AWS bill. It began when an explicit boundary was treated as optional. It grew when an expected rollout guardrail was removed. It became expensive when code was accepted without understanding its business behavior, request options, retry semantics, or cost.

AI can produce code. It cannot accept accountability. The Project Technical DRI must know what the system does, prove that it matches the approved scope, and protect the client’s budget before production teaches the lesson at full price.