Engineering & Architecture Brief Published: 2026-11-15
When the traffic micro-simulation stalls, a sensor injection bridge set to hold its last sample keeps feeding the perception stack. It restamps that old object list with the current co-simulation clock, so to the stack it looks like fresh data. The ego vehicle keeps driving through a world that has stopped moving, the scenario passes, and the dashboard stays green. Nobody catches it in review, because the change that causes it reads like a resilience fix.
Why the diff gets approved The pull request is small. Nightly scenario runs were flaking because the micro-simulation sometimes paused on a large road network during route recomputation or a memory compaction, so someone raised the bridge timeout and set the fallback from abort to hold last value. To a reviewer that looks like graceful degradation, which is normally what we want. The trouble is in the restamping. The bridge copies the master clock into the header of the held frame, so every downstream consumer sees a monotonic, well-formed stream, and the freshness checks that inspect only the header have nothing to complain about. Actor positions in sensor space freeze, while ego dynamics keep integrating at full rate.
What should keep working when one part fails In a driving twin, the failure of one federate should be loud at its boundary and quiet nowhere else. Hold-last-value is a legitimate model for a camera or radar dropout, but only when the scenario scheduled it as a fault and the frame keeps its original source time. When the traffic simulator stalls, that is an infrastructure fault, and an infrastructure fault has to invalidate the run, because the planner now faces an easier problem than the one specified: static obstacles in place of a merging truck or a pedestrian stepping off the kerb. Safety evidence frameworks for automated driving, such as ISO 26262 and ISO 21448, make simulation results count only when you can show the toolchain did what it claimed. A run that quietly swapped dynamic traffic for a still image is worse than no run, since it improves the KPIs and pushes the real regression further into the future. The design rule that keeps the system honest is a separate source timestamp, written by the producer and never touched by any bridge.
A check you can run against tonight's logs Take any recorded injection log or bag from a completed batch and pull three fields from every injected object list: the header stamp, the simulation time the traffic federate reported when it produced the list, and a digest of the serialized actor states. Compute frame age as the header stamp minus the source time, and flag every frame whose age exceeds one master step, which is 10 ms in a 100 Hz lockstep configuration. Then walk the frames in order and flag any run of identical digests that lasts more than one step while the ego odometry shows the vehicle moving, because live traffic almost never produces two identical lists in a row. If your producer does not write a source time at all, that gap is your first finding, and the digest check alone will still expose the stall. By rough estimate, a team running a few thousand scenarios a night can add this as a post-run gate in well under a day, and it costs far less than reopening a perception regression ticket that turns out to be a frozen federate.
The decision rule If any frame's age exceeds one master step and the scenario did not schedule a sensor dropout at that moment, the run is reported as invalid instead of passed or failed, and that third state is what stops the same incident from being triaged again every quarter as a planner or perception bug.
Technical FAQ: Why not just restore abort on timeout and accept the flaky runs? Abort is safe, but it treats every stall the same, so teams end up raising the timeout again within weeks. Keeping a fallback for scheduled sensor faults, enforcing the staleness gate on producer source time, and reporting infrastructure stalls as invalid keeps the flake rate visible without letting stale frames reach the evidence set. What's left to decide is how many invalid runs per batch you accept before you treat the micro-simulation's own scaling as the incident.