Skip to content

Generation Barriers — Lockstep Worker Synchronization

See also: Documentation Index, PBT Core Components, Workload Orchestrator, Performance Evaluation, ADR-003

Why barriers exist

When N workers evaluate in parallel on one host, their critical paths inevitably diverge: one finishes pg_reload_conf() quickly while another spins through a slow restart; one warms up its buffer cache from disk while another already has data resident. If the workers' measurement windows don't overlap, the late one measures under lighter contention than the early one — and the score difference reflects scheduling artefacts, not configuration quality.

The GenerationBarrier enforces lockstep evaluation: every worker waits at every sub-step until all workers have arrived. This guarantees that the measurement window (B9) of every worker overlaps the measurement window of every other worker in the same generation. The score difference between workers within a generation then reflects only their knob configurations.

This is the single piece of machinery that makes "8 workers in parallel on one host" methodologically defensible. Without it the parallel speedup would come at the cost of measurement bias.

Table of Contents

  1. Barrier table (B1–B17)
  2. API
  3. Graceful degradation
  4. Why no timeout
  5. Sequential mode
  6. Integration with the orchestrator
  7. Design decisions
  8. Related documentation

Barrier table (B1–B17)

Every call into WorkloadOrchestrator.evaluate_worker() passes through 17 ordered sub-steps. Each one ends with barriers.wait(name, worker_id) that blocks until all N workers in the generation have arrived.

# Name (BARRIER_NAMES) Sub-step What just completed
B1 connected Connect TCP/psycopg2 connection established.
B2 config_applied Apply config ALTER SYSTEM SET … + pg_reload_conf() for sighup knobs.
B3 restarted Restart (or skip) Restart triggered by should_restart finished. Workers that didn't need a restart pass through immediately and wait here for those that did.
B4 reconnected Reconnect Post-restart re-connection succeeded.
B5 config_verified Verify KnobApplicator.verify() read-back; quantised values merged into worker.knob_config.
B6 pre_stats_captured Pre-stats snapshot pg_stat_database / pg_stat_bgwriter / pg_stat_user_tables snapshot taken.
B7 benchmark_ready Benchmark prepare Schema validated; data loaded if first run.
B8 warmup_done Warmup Executor's warmup phase complete.
B9 measurement_done Measurement Timed measurement window complete. This is the only window whose metrics enter the score.
B10 post_stats_captured Post-stats snapshot Second pg_stat_* snapshot.
B11 io_computed I/O delta I/O delta + buffer stats derived from pre/post snapshots.
B12 system_metrics_collected System metrics Memory + cache + scan metrics via metric_instrumentation.
B13 memory_pressure_computed Memory pressure Derived memory-pressure signal computed.
B14 reliability_gated Reliability gate Failure / degradation classification applied to the gate G ∈ [0, 1].
B15 vacuum_done Vacuum analyze Post-DML VACUUM ANALYZE (bounded by vacuum_analyze_timeout_seconds).
B16 score_computed Score CompositeScorer.score() → ScoreBreakdown.
B17 disconnected Disconnect Connection closed.

The canonical list is the BARRIER_NAMES constant. Any code that adds or removes a sub-step must update this list and the orchestrator's call sites together.

API

Location: src/tuners/engine/barriers.py

class GenerationBarrier:
    def __init__(self, num_workers: int, enabled: bool = True) -> None: ...

    @property
    def enabled(self) -> bool: ...
    @property
    def broken(self) -> bool: ...

    def wait(self, name: str, worker_id: int) -> None: ...
    def drain_remaining(self, start_from: str, worker_id: int) -> None: ...
    def abort(self) -> None: ...
    def reset(self) -> None: ...
    def next_barrier_name(self, current: str) -> Optional[str]: ...

wait(name, worker_id)

Blocks until all num_workers workers have called wait() with the same name. Logs the wait under the per-worker WorkerBarrier logger context. Raises ValueError if name is not in BARRIER_NAMES. Returns immediately if the barrier set is disabled or broken.

drain_remaining(start_from, worker_id)

Releases all barriers from start_from onward (inclusive). Called by a worker's exception handler: when worker k fails at sub-step j, it calls drain_remaining(name_at_j, k) so it still "arrives" at every barrier from j onward, letting peers proceed instead of deadlocking at the next barrier.

Internally this is a loop of barrier.wait() calls — it does not break the barriers, only contributes this worker's missing arrival.

abort()

Instantly breaks every barrier (raises BrokenBarrierError on all current and future waiters) and marks subsequent wait() calls as no-ops. In the PBT path it is called from exactly one place: the except clause around future.result() in Population.evaluate_generation() — i.e. when a worker thread's evaluation raises. Unlike drain_remaining, abort() does not try to honour the protocol — it tears down the whole synchronization mechanism for the generation.

reset()

Recreates the barriers for the next generation. Called once per generation by Population.train_generation().

next_barrier_name(current)

Returns the name of the barrier immediately after current, or None if current is the last barrier. Used by orchestrator error paths to construct the correct start_from argument for drain_remaining().

Graceful degradation

A live worker thread that crashes mid-evaluation must not deadlock the other threads at the next barrier. Two exception-driven safeguards make this work; a worker that hangs without raising is the one case they cannot reach.

                  ┌────────────────────────────────────┐
                  │    Worker thread runs evaluation    │
                  └──────────────────┬─────────────────┘
                                     │
     ┌──────────────┬────────────────┼─────────────────┬────────────────────┐
     │              │                │                 │                    │
 succeeded    raises, caught    raises, reaches    hangs without raising
     │        in the worker     the population     (no timeout fires)
     │              │           future.result()          │
     ▼              ▼                │                    ▼
 wait() through drain_remaining      ▼              nothing can trip
 B1..B17        (next barrier   population calls     abort(); peers
 normally       after failure), barriers.abort()     block at the next
                then re-raises  → breaks every        barrier forever
     │              │           barrier                     │
     └──────────────┴────────────────┘              deliberately accepted
                    ▼                                residual — see
     Peers proceed past every barrier                "Why no timeout"
     (real arrivals, or drained / aborted).

Path 1 — clean completion

Every worker calls wait("connected", id) … wait("disconnected", id) in order. The orchestrator's normal control flow handles this.

Path 2 — caught exception (drain_remaining)

The orchestrator wraps each sub-step in a try/except. If, say, sub-step B7 raises:

except Exception:
    barriers.drain_remaining("benchmark_ready", worker_id)
    raise

Now this worker has "arrived" at B7, B8, B9, …, B17 from the barrier's point of view. Peers can finish their generation; the failing worker propagates the exception up to the population layer, which logs it and lets rescue_dead_workers() handle the recovery on the next generation boundary.

Path 3 — raised exception reaches the population (abort)

When a worker's evaluation raises and the exception propagates out of evaluate_worker to the population's future.result(), the population calls barriers.abort(). Every waiter immediately receives BrokenBarrierError, the orchestrator catches it, every per-worker thread exits its evaluation, and the generation ends with a rescue_dead_workers() call.

Exception-driven by design — and what that leaves uncovered. Both escape paths fire only on a raised exception, and that is deliberate. The layers around the barrier already convert the failures that matter into exceptions: connection attempts are bounded by connect_timeout, the external benchmarks bound their own work (TPC-H sets a failsafe statement_timeout; sysbench runs under a subprocess communicate(timeout=…)), B15's VACUUM ANALYZE is bounded by vacuum_analyze_timeout_seconds, and the environment lifecycle operations (verify_instances, _wait_until_connectable, recover_instance / rebuild_worker_instance, Docker's SDK operation timeouts) either return or raise rather than block forever. A crashed, refused, dropped, or unresponsive instance therefore surfaces as an exception that trips drain_remaining (Path 2) or abort() (Path 3). Liveness is not polled on a background thread — there is no health-check thread and no DatabaseEnvironment.is_alive(), and neither has ever existed in the codebase.

The one case neither path reaches is a worker that hangs without raising: a synchronous call on its main thread that neither returns nor raises and is bounded by no timeout — for example a query on an already-established connection to a server wedged at the socket-read level, where connect_timeout bounds only the initial handshake and no statement_timeout is set (the internal WorkloadExecutor sets none on its measurement queries). Nothing outside that thread can trip the abort, so its peers block at the next barrier indefinitely. This residual is accepted on purpose: the only ways to bound it are a per-barrier timeout (rejected — see Why no timeout) or a genuine out-of-band liveness thread (not implemented). It is a bounded, understood trade-off, not an unnoticed hole.

Why no timeout

threading.Barrier accepts a per-wait timeout argument. We deliberately don't use it.

The problem. A timeout that's short enough to detect a true hang (e.g. 60 s) will fire spuriously on legitimately slow operations:

  • A TPC-H Q21 on a small-RAM container can take 5–10 minutes.
  • A first-time dbgen data load can take many minutes.
  • A restart_instance() after a postmaster knob change can take 30–60 s on its own; coupled with a slow disk, the B3 barrier might legitimately need to wait several minutes.

Any timeout small enough to be useful for hangs is small enough to false-positive on these. False positives convert "everything is fine, just slow" into a broken-barrier event that triggers rescue_dead_workers() for all workers — losing the generation.

The solution. Wait indefinitely and let liveness surface as an exception from the layers around the barrier. Connection attempts (connect_timeout), the external benchmarks (TPC-H's failsafe statement_timeout, sysbench's subprocess communicate(timeout=…)), the bounded B15 VACUUM ANALYZE, and the environment lifecycle operations (verify_instances, _wait_until_connectable, recover_instance / rebuild_worker_instance, Docker's SDK operation timeouts) each bound their own work and raise on a dead or unreachable instance — telling "PostgreSQL is busy" (a slow call that eventually returns) apart from "PostgreSQL is dead" (a call that errors). A raised exception then trips drain_remaining or abort(); the barriers themselves never time out.

These probes are synchronous, not a background poller: they surface a failure only at a call that actually returns or raises. A worker wedged in a call that does neither — the residual described under Graceful degradation — is not bounded by them. Accepting that narrow residual is the price of never false-positiving on a legitimately slow generation.

Sequential mode

GenerationBarrier(num_workers=N, enabled=False) produces a no-op barrier object. wait() returns immediately; drain_remaining() and abort() are also no-ops. This is used in:

  • single-worker debugging (--population 1 plus --no-sync),
  • unit tests that exercise the orchestrator's body without setting up real concurrency,
  • the BO baseline's parallel path, which manages its own ask-tell synchronization and doesn't need lockstep barriers.

Code that consumes the barrier object never has to branch on the enabled flag — the no-op API is identical.

Integration with the orchestrator

Population.train_generation()
  │
  ├─► barriers.reset()                                # fresh barriers for this generation
  │
  ├─► ThreadPoolExecutor(max_workers=N)
  │     │
  │     │ for each worker:
  │     │   orchestrator.evaluate_worker(worker)
  │     │     │
  │     │     ├─► connect()
  │     │     ├─► barriers.wait("connected", worker_id)       # B1
  │     │     ├─► apply_configuration()
  │     │     ├─► barriers.wait("config_applied", worker_id)  # B2
  │     │     ├─► maybe_restart()
  │     │     ├─► barriers.wait("restarted", worker_id)       # B3
  │     │     │   ... (B4–B17) ...
  │     │     └─► return (metrics, score_breakdown)
  │     │
  │     │ on exception in orchestrator:
  │     │   barriers.drain_remaining(failed_sub_step, worker_id)
  │     │   re-raise
  │     │
  │     │ population, on exception from future.result():
  │     │   barriers.abort()          # the only abort call site
  │     │   re-raise
  │
  ├─► rescue_dead_workers()
  ├─► update_metric_ranges_if_needed() / _finalize_scores()
  ├─► record_generation()
  └─► execute_exploit_explore() / env.clone_instances() / should_stop()

The orchestrator owns the try/except + drain_remaining plumbing; the population owns reset and abort. The barrier object itself owns no state beyond the threading.Barrier instances and a single _broken flag.

Design decisions

1. One barrier per sub-step, not one per phase

We could coarsen to 3–4 barriers (e.g. apply_done, measurement_done, score_done). We don't, because the measurement-window overlap guarantee is what matters scientifically, and that requires sub-step-level alignment around B7–B10. The B1–B17 granularity is a cheap-to-pay generalisation of that need to the whole evaluation.

2. Fixed barrier names, not dynamic

BARRIER_NAMES is a hard-coded list. Adding a barrier means editing both the list and the call sites. This makes the protocol auditable: the doc table above is enforced by the type system, not a convention. A dynamic registry would risk silent name drift between orchestrator and barrier object.

3. No timeout

See Why no timeout. The cost of a false positive (a lost generation) is high and certain; the cost of the uncovered case (a worker that hangs without ever raising) is a rare, deliberately accepted residual, because every failure that raises is already caught by the exception-driven drain_remaining / abort() paths.

4. Graceful degradation has two paths, not one

drain_remaining and abort are distinct because they describe different failures. drain_remaining is for "this worker is failing cleanly and propagating an exception" — peers still trust the protocol. abort is for "the protocol is broken; tear it down" — peers no longer trust the protocol. Coalescing them would lose that distinction and prevent the orchestrator from cleanly logging which workers failed for which reasons.

5. Barrier object owns minimal state

The barrier object only knows about num_workers, BARRIER_NAMES, and a _broken flag. It does not know about generations, workers, or scoring. Population owns the lifecycle (reset, abort); orchestrator owns the call sites (wait, drain_remaining). This keeps the file small and testable.

File locations