Personal account

Leading safer change in a national transport dataset

A personal account of technical leadership across event-driven NaPTAN ingestion and controlled permanent stop removal.

Evidence boundaryThis is an anonymized account of my work as Tech Lead / Tech Principal on NaPTAN from January 2023 to January 2026. Client identity and sensitive implementation details are omitted. Public links describe service context and do not independently verify my personal contribution.

Problem

I technically led a multidisciplinary team of roughly 7-12 people maintaining and modernising NaPTAN. The existing upload path accepted one file per submission, with timeout, failure-coupling, and concurrency constraints. Permanent stop removal depended on manual intervention and lacked automated checks against active route usage.

Constraints

  • NaPTAN is a national public transport dataset used by data producers and downstream consumers, so changes must preserve publishing integrity.
  • File ingestion needed on-demand concurrency without coupling every file to one long-running request.
  • Permanent removal is low-volume but high-risk: an invalid decision can affect published transport services and cannot simply be treated as a normal update.
  • The work crossed engineering, data operations, policy, transport-operator, and external-consumer boundaries.
  • Client identity, internal system names, exact stop volumes, raw operational data, security configuration, and sensitive implementation details must remain private.

Architecture

Multi-file ingestion, asynchronous

  1. Authorized submissionAccept up to 10 files, replacing a one-file-per-submission path, and assign per-file processing state.
  2. Pub/Sub event routingCreate independent events and a retry boundary so files can progress without one long-running synchronous request.
  3. Parallel
    Cloud FunctionsRun lightweight event triggers.
    Cloud RunRun heavier processing and custom dependencies.
  4. Reliability controlsUse retries and dead-letter handling, idempotency and duplicate-event protection, per-file status, and partial-failure isolation.
  5. Per-file outcomeExpose observable status and continue each successful file through the publishing path.
Simplified public reconstruction from this personal account. Internal names and security-sensitive details are omitted.

Managed removal, synchronous and guarded

  1. Authorized producerRequest permanent stop removal through a role-restricted workflow.
  2. NaPTAN validationValidate the stop and removal state before an irreversible action is available.
  3. Parallel
    BODS checkCheck active route usage in BODS.
    TNDS checkCheck active route usage in TNDS.
  4. Human confirmationRequire role-based authorization and explicit human confirmation.
  5. Remove, audit, and republishPersist state for safe retry or resumption, use compensation where reversal remains possible, retain an auditable decision, and prevent removed stops from returning in published files.
Simplified public reconstruction from this personal account. Internal names and security-sensitive details are omitted.
  • 10xincrease in per-submission file capacity, from one file to ten
  • >=75%reduction in manual support effort and processing time for managed removal
  • 12+ monthsproduction operation with zero erroneous removals

Decisions

Decouple multi-file ingestion

ChoiceUse Pub/Sub with Cloud Functions and Cloud Run workers, plus idempotency, retry and dead-letter handling, per-file status, and partial-failure isolation.

Trade-offThis introduced asynchronous state and operational controls, but avoided the timeout and coupling risks of a larger synchronous request and the latency of scheduled batching. The production result was a tenfold increase in per-submission file capacity.

Keep permanent removal synchronous and guarded

ChoiceComplete cross-system validation, present a clear decision, and require explicit operator confirmation before removal.

Trade-offThe workflow accepts tighter runtime coupling instead of an asynchronous saga because removal is low-volume but high-risk and operators need immediate validation before an irreversible action.

Design partial-failure and governance controls together

ChoicePersist workflow state for safe retry or resumption, use compensating actions where reversal remains possible, enforce role-based authorization, and retain auditable records.

Trade-offThe additional state and controls increase implementation and operational effort, but make failure handling and accountability explicit. The production capability reduced both manual support effort and processing time by at least 75%, and operated for at least 12 months with zero erroneous removals.

Treat alignment as an engineering mechanism

ChoiceUse architecture decision records, design reviews, stakeholder workshops, roadmap planning, demonstrations, governance and risk forums, and operational-readiness reviews.

Trade-offThese mechanisms consume delivery time, but keep policy, operations, data consumers, and implementation decisions from diverging across a multi-year engagement.

Public context

Review the public service context separately from this personal account. It describes the service behavior and does not independently verify my personal contribution.

Limitations

  • This is a personal account of my contribution; client identity and sensitive implementation details are omitted.
  • The public service context describes managed-removal behavior and does not independently verify my personal contribution.
  • Exact stop volumes remain private, and no raw operational data or security configuration is disclosed.