Cause of Error

A method for root-cause analysis

A worked example

This site asks you to produce a single document with seven sections, and until now it never showed you one.

The incident below is invented. The company, the systems and the people in it do not exist, and the numbers were chosen to be consistent with each other rather than to be true.An invented example is the honest choice here. A real one is somebody's bad evening, and the details that make it teachable are the details that make it theirs. Everything else about it is meant to be real: the register, the level of detail, and the things it admits.

It is shown at its second issue, because that is the normal shape of a CoE. The first went out the morning after. This one went out three weeks later, when the analysis had moved on and most of the actions were closed — and when the timeline had grown backwards, which is the part people find surprising.


CoE 2026-014 — Signed-in account pages served from a shared cache

Issued 3 March 2026 · Revision 1.1, 27 March 2026 · Status: open, 3 actions outstanding

1. Summary

For 38 minutes on the evening of Monday 2 March 2026, some customers who signed in and opened their account summary page were shown a page belonging to another customer.

A change to our content delivery configuration, made that afternoon, allowed a page built for one person to be stored in a cache that everybody shares. 396 page views were served this way. They disclosed the details of 41 customers to 214 other customers.

We were told by a customer, not by our own monitoring. We rolled the change back 38 minutes after it took effect, notified the Information Commissioner within 72 hours, and wrote to all 255 customers involved.

2. Customer impact

A customer who opened the account summary page during the incident saw somebody else's name, postal address, current balance and up to ten recent transactions, with the last four digits of that person's account number.

Nobody could act on the account they were shown. The summary page is read only, and every action reached from it re-checks the session against the signed-in customer, so no customer was able to move money, change an address or see anything beyond that one page.

Four customers contacted us while it was happening. The first did so at 19:51, nine minutes in, and is the only reason we knew at all. Two of the four described it as a display fault and were, at first, agreed with.

No customer lost money. The harm here is the disclosure itself, and the reasonable worry of anybody who saw a stranger's address on our site and had to wonder who had seen theirs.

3. Security impact

Confidentiality was breached. Integrity and availability were not.

Disclosed, per affected customer: full name, postal address, current balance, up to ten recent transactions, and the last four digits of one account number. Not disclosed: any credential, full account number, card number, date of birth or contact details.

We treated this as a personal data breach under Article 33 and notified the Information Commissioner's Office at 11:20 on 3 March, within 72 hours of becoming aware. We wrote to all 255 customers on 5 March.

We checked whether a request with no session could be served one of the cached pages, which would have made this a disclosure to the open internet rather than to other customers. It could not: a separate edge rule redirects requests without a session cookie before the cache is consulted.

That was luck, not design. Nothing in the change we made depended on that rule being there, nobody checked for it beforehand, and it protects this path only because it was written for an unrelated reason in 2024. It appears in Lessons learned for that reason and not as reassurance.

4. Timeline

All times GMT, Monday 2 March 2026.

15:10 - Change CDN-4471 reviewed and approved. Two approvers, both
        checking the rule syntax and the cache-hit target
19:42 - CDN-4471 reaches the last edge location. Caching begins
19:51 - First customer contact. Agent records it as a display fault
20:03 - Second and third contacts within four minutes. Contact centre
        lead escalates
20:07 - Incident opened. Priya begins investigating
20:09 - Ravi comes online and takes communications
20:11 - Fourth customer contact
20:14 - Reproduced internally: two test accounts on the same edge
        location are served the same page
20:16 - CDN-4471 identified as the only change to that path that day
20:18 - Rollback of CDN-4471 begun
20:20 - Rollback complete at all edge locations. Caching stops
20:24 - Cache purged for /account/*
20:31 - Two test accounts confirm pages are no longer shared
20:44 - Incident closed
21:30 - Log extraction begun to establish who was served what

Added at revision 1.1, once the analysis had run:

2024-11-06 - The edge rule that redirects sessionless requests is
             written, for an unrelated reason
2026-02-14 - The caching rule for marketing pages is written. It is
             correct for those pages, which are identical for everyone
2026-03-02 - CDN-4471 is raised, copying that rule onto /account/summary

The last entry is the one that matters and it was not in the first issue, because on the night we were describing an outage and by revision 1.1 we were describing how a correct rule became an incorrect one.

5. Five whys

Why did a customer see another customer's details? The account summary page was served from a cache shared by all customers.

Why was a personalised page in a shared cache? Change CDN-4471 applied a caching rule to /account/summary, and the cache key for that rule was the URL alone.

Why did the cache key not include the session? The rule was copied from the rule that caches our marketing pages, where the URL is the whole of the identity of the response.

Why did copying that rule pass review? Both approvers checked what the change was — valid syntax, sensible time to live, a plausible cache-hit target. Nothing in the review asks what the response contains.

Why does the review not ask that? Because we cannot answer it. There is no list of which routes return data belonging to one customer, so a reviewer has no way to check the claim short of reading the application code for the path.

Why is there no such list? Our route inventory was built to configure load balancing, where the only attributes that mattered were the host, the path and the health check. Personalisation was not an attribute anybody needed until caching moved to the edge in 2025, and the inventory was never revisited when it did.

Why did we not detect it ourselves? Every check we run signs in as one user. Nothing we run compares what two different users are served, so a page correctly built for the wrong person looks exactly like a page correctly built for the right one.

Root cause. We moved caching to the edge without recording which responses are specific to a customer. That left a review process asking a question the data does not exist to answer, and a test suite that cannot see the class of fault it creates. The cache key was the mistake; the missing classification is the reason the mistake had nothing to stop it.

6. Lessons learned

Our monitoring did not detect this and would not have. Four customers did, and the first was recorded as a display fault — the fastest signal we had was sitting in the contact centre for twelve minutes before anybody treated it as an incident.

A change review that checks whether a change is correct will not catch a change that is correct somewhere else. The rule that was copied is still right for the pages it was written for.

We were lucky about unauthenticated access, and we should say so plainly rather than let a paragraph about a control we did not know we had read as though we had planned it.

We started gathering evidence 46 minutes after closing the incident. The 72-hour clock had already been running for nearly two hours. Nothing was lost, but nothing guaranteed that.

The rollback itself was quick and the communications were good. One person spoke to the outside and everybody else worked, which is why the timeline above is as detailed as it is.

7. Next actions

Next actions as at revision 1.1

IDActionOwnerDueStatus
NA-1Purge and re-verify every cache rule that matches a path under /accountPriya S2026-03-03Done
NA-2Cache rules for authenticated paths require a key that includes the sessionPriya S2026-03-09Done
NA-3Add a test that requests each page as two signed-in users and fails if the responses matchDan O2026-03-20Done
NA-4Classify every route as personalised or shared, and publish the listAma T2026-04-17In progress
NA-5Change review for edge configuration asks whether the response varies by userAma T2026-04-17In progress
NA-6Alert on a rise in cache-hit ratio for any route classified as personalisedDan O2026-05-01Not started
NA-7Begin evidence gathering when an incident opens, not when it closesRavi B2026-03-13Done

Three remain open at revision 1.1. NA-4 is the root cause and the slowest, and this CoE will be reissued when it closes.


Read the seven sections for what each one is for, starting at 1. Summary. What good and bad look like in practice is under Implementing.

ver 0.2 rev 2026-08-02 status live