Most of the operations a dataset supports are forgiving. An insert can be updated. An update can be corrected. Even a bad publish can be superseded by the next one. Delete is different. A permanent removal reaches backward and forward at the same time: it changes what the dataset says about the world, and it changes what everyone who already read the data should now believe.
I spent three years as the technical lead of the team maintaining NaPTAN, the national record of public transport access points in Great Britain. The work that taught me the most was not the high-volume path. It was the low-volume, high-risk one: permanently removing a stop.
Deletion is not a statement; it is a workflow. The datasets that treat it as one are the ones you never read about in post-mortems.
Why removal is not an update
A normal update is idempotent by nature: replay it and you converge on the same state. A removal is a one-way door. The stop may be referenced by active routes in other datasets — in our case the bus open data services BODS and TNDS — and by downstream consumers who have already built journeys, maps, and accessibility information on the assumption that the stop exists. An invalid removal cannot be treated as a normal correction, because the published artifact has already been believed.
Volume makes this worse, not better. Removals are rare, which means the path never gets the battle-testing that ingestion gets daily, and the temptation is to handle them manually: a support ticket, a database statement, a quiet fix. That works until the day it produces an erroneous removal in a national dataset.
The rule: removal is a workflow, not a statement
The design that survived production treats permanent removal as a managed workflow with guards at every step, and no single action capable of causing damage on its own.
- Restricted initiation. Only an authorized data producer can request a removal, through a role-restricted workflow. The request is recorded, not executed.
- Validation before availability. The stop and its removal state are validated before any irreversible action becomes available.
- Cross-dataset guards. The workflow checks active route usage in BODS and TNDS. A stop that still serves published routes cannot be removed quietly — the check runs before the door opens, not after.
- Explicit human confirmation. A person with the right role must confirm, knowing the checks have run. The system informs; the human decides.
- Removal with a trail. The decision is auditable end to end. State is persisted so a failure mid-workflow can be retried or resumed safely, and compensation is used where reversal is still possible.
- Publishing safeguards. A removed stop must not quietly reappear in published files on the next ingestion cycle.
None of these steps is individually exotic. The discipline is in refusing to collapse them into one.
What the guards bought
The managed-removal capability has run in production for at least twelve months. Processing time and manual support effort dropped by at least 75 percent. The number I care about most is the quietest one: zero erroneous removals.
An honest account has to say what this does not establish. This is an anonymized personal account of one engagement; it does not prove the approach transfers unchanged to another dataset, another regulator, or another team. What I believe does transfer is the shape of the answer: when an operation is irreversible, rare, and cross-referenced, it earns a workflow — with authorization, validation, external checks, human confirmation, and an audit trail — before it earns automation.
The lesson worth keeping
Shared datasets are judged by their quietest operations, not their busiest ones. Ingestion gets the architecture diagrams because it is daily and measurable. Removal gets a support ticket because it is rare. But the rare, irreversible path is exactly where a national dataset — or any shared data product — either deserves the trust placed in it or does not.
If your dataset has a managed path for adding rows and an unmanaged one for removing them, the second one is the incident you have not had yet.