Skip to content

ADR-006: Unified src/tuners Package and BaseTuner Lifecycle

  • Status: Accepted — copy-not-refactor invariant later superseded (see 2026-07-17 addendum); hook surface finalized (see Addendum 2026-10-26)
  • Date: 2026-06-19
  • Relates to: src/tuner (PBT) and src/scripts/bo_baseline (BO) — both since deleted, and the new src/tuners package (LHS-design + shared core).

Context

The project ships two configuration-tuning strategies today:

  1. PBT — Population-Based Training, in src/tuner/main.py (PBTTuner).
  2. BO — a Bayesian Optimization baseline, in src/scripts/bo_baseline/runner.py (BOBaselineRunner).

(Both paths are historical — see the 2026-07-17 and 2026-09-14 addenda. PBT now lives at src/tuners/pbt/ and BO at src/tuners/bo/.)

Both classes independently re-implement an almost identical lifecycle scaffold around their genuinely different optimizer cores:

  • resolve per-worker hardware resources (manual override vs. auto-detect);
  • build a workload executor + extract workload features (sysbench / tpch / custom template branch);
  • bring up PostgreSQL instances and prune knobs absent from the runtime pg_settings;
  • run a sequence of generations / iterations under lockstep barriers;
  • tear down instances (and optionally clean up data);
  • serialize a tuning_session JSON envelope with a shared header block plus best_configuration and worker_resources.

The only parts that genuinely differ between PBT and BO are how the next configuration(s) are proposed and when to stop. Everything else is duplicated, with the two copies already drifting (e.g. PBT degrades gracefully when pg_settings is unreachable, BO re-raises).

We want to add a third strategy — a Latin Hypercube Sampling (LHS) importance-design tuner — without tripling the duplication. The research framing (see docs/guides/scalpel-rollout.md) is that SCALPEL applied to an LHS design over the knob space yields DBA-competitive tiers, whereas applied to PBT trajectory variance the signal is too narrow. The LHS tuner must run in parallel like PBT (barriers, per-worker resources) and emit a schema-compatible session JSON.

Decision

Introduce a new top-level package, src/tuners, that holds:

  • BaseTuner (src/tuners/base.py) — an ABC encoding the invariant lifecycle as a concrete run() template method (Template Method pattern). It owns timing instrumentation, the generation loop, the teardown guard, and result assembly, and delegates the strategy-specific decisions to abstract hooks: setup, propose_initial_configs, step, should_stop, collect_best, build_session_payload, teardown. (Superseded: the hook list is now the one in the 2026-10-26 addendum.)
  • src/tuners/utils/ — the shared scaffolding extracted from PBT and BO:
  • types.py — TuningStrategy enum (pbt / bo / lhs), GenerationOutcome, TunerLifecycleConfig.
  • output_paths.py — resolve_tuner_output_root, unifying PBT's _build_output_dir and BO's resolve_bo_output_root into one strategy-parameterized resolver (including the @scalpel-v1 data-driven tier suffix).
  • resources.py — resolve_worker_resources, the manual-vs-auto dispatch.
  • executors.py — build_workload_bundle, the sysbench/tpch/custom branch.
  • knob_filter.py — runtime pg_settings knob pruning, as DB-free testable helpers.
  • calibration.py — the global score-recalibration utilities (relocated here from the now-deleted src/utils/rescoring.py; see the 2026-06-22 addendum). (Later moved again to src/utils/calibration.py during the 2026-07 migration — see the 2026-07-17 addendum.)
  • session_writer.py — convert_numpy_types, build_session_header, and the session/best-config write helpers.

Extraction is by copy, not refactor

src/tuner (PBT) and src/scripts/bo_baseline (BO) are not modified by this change. The shared utilities are lifted into src/tuners/utils as the canonical implementation that new strategies build on, but the incumbents keep their own inline copies for now.

Rationale:

  • Risk isolation. PBT and BO are the two strategies the paper's headline numbers depend on. Retrofitting them onto a new ABC in the same change that introduces the ABC would put those numbers at risk for no immediate benefit.
  • Reviewability. The diff for "add a third strategy" stays additive: new files, no behavioral change to existing runs.
  • Reversibility. If the BaseTuner abstraction turns out to fit LHS but chafe against PBT/BO, we can iterate on it against LHS alone before committing to a migration.

A follow-up ADR will decide whether (and how) to migrate PBT and BO onto BaseTuner once the abstraction has proven itself on LHS.

tuning_strategy is orthogonal to benchmark_name

The session JSON keeps benchmark_name as the workload driver ("sysbench" / "tpch" / custom) and adds tuning_strategy ("pbt" / "bo" / "lhs") as a separate discriminator (see the prior tuning_strategy field migration). A session has exactly one of each. Loaders in src/analysis/data_loader.py and src/evaluation/loader.py read the explicit field and fall back to a path heuristic (/pbt_runs/, /bo_runs/, /lhs_runs/) for legacy files.

Consequences

Positive

  • A third strategy can be added as a single BaseTuner subclass plus a CLI, reusing the shared lifecycle, output layout, and serialization.
  • The shared helpers are individually unit-tested without a database (tests/unit/tuners/), closing the PBT/BO drift gaps (e.g. graceful pg_settings degradation is now the documented default).
  • Output paths, numpy serialization, and the session header are guaranteed consistent across strategies because they flow through one implementation.

Negative / accepted costs

  • Temporary duplication. Until the follow-up migration, the lifecycle scaffold exists in three places (PBT inline, BO inline, src/tuners/utils). This is a deliberate, time-boxed cost taken to de-risk the paper's incumbents.
  • The BaseTuner API is provisional: it is validated only against LHS in this change, so its hook surface may shift before PBT/BO adopt it. (Resolved: the provisional phase ends with the 2026-10-26 addendum.)

Alternatives considered

  1. Refactor PBT and BO onto BaseTuner now. Rejected: couples a risky refactor of the headline strategies to the introduction of a new one.
  2. Put LHS inside src/tuner alongside PBT. Rejected: src/tuner is PBT-specific (population, workers, evolution, barriers); LHS is not a population method, and overloading the package would muddy both.
  3. No shared base; copy the whole PBT scaffold into the LHS tuner. Rejected: that is exactly the duplication this ADR exists to bound, and it would immediately drift like PBT/BO already have.

Addendum (2026-06-19): LHSDesignTuner — the first concrete strategy

src/tuners/lhs_design/tuner.py is the first concrete BaseTuner. It evaluates a fixed Latin Hypercube Sampling design over the knob space — no evolution, no exploit/explore, no perturbation:

  1. One design, drawn once. KnobSpace.sample_diverse_configs(design_size) produces a space-filling sample; slot 0 is anchored to the PostgreSQL default config (mirroring PBT/BO's pilot-seed convention).
  2. Swept in parallel batches. The design is sliced into batches of num_parallel_workers. Each batch is evaluated concurrently under the same GenerationBarrier lockstep PBT uses, so every configuration's measurement window experiences identical contention. max_generations is therefore ceil(design_size / num_parallel_workers) — each "generation" in the BaseTuner loop is one batch.
  3. No feedback. Because there is no optimizer reacting to scores, every knob varies independently of performance. That independence is exactly what SCALPEL needs: applied to this design the per-knob importance signal is wide enough to tier; applied to PBT's optimization trajectory the variance collapses (see docs/guides/scalpel-rollout.md).

The tuner composes PBT's environment, orchestrator, and Worker machinery for the actual apply→run→measure step, but drives them through the strategy-agnostic BaseTuner lifecycle rather than PBT's Population evolution loop. PBT and BO remain unmodified.

A run is launched via either entry point:

python -m src.tuners.lhs_design --tier core --benchmark sysbench \
    --sysbench-workload oltp_read_write --design-size 64 --parallel-workers 4
# equivalently:
python -m src.tuners --tier core --benchmark sysbench --design-size 64

Output lands at results/sessions/{workload}/lhs/{tier}/traces/trace_*.json, carrying tuning_strategy: "lhs" and a design_records array (one entry per design point with its config fractions, metrics, and score breakdown) that the SCALPEL pipeline consumes.

Addendum (2026-06-22): shared CLI, profiles, and PBT/BO parity

After LHSDesignTuner proved the BaseTuner abstraction, the strategy grew the remaining surface needed to be a first-class peer of PBT/BO. None of it touched src/tuner/ or src/scripts/bo_baseline/, so the copy-not-refactor invariant still holds.

Strategy-agnostic CLI + profile registry

src/tuners/cli.py now owns the full strategy-agnostic flag surface (Tuning Configuration, Workload Settings, Instance Management, Per-Worker Resources, Scoring & Normalization, Output & Logging). A new strategy entry point shrinks to just its own knobs (for LHS, only --design-size) plus a call to add_common_groups.

The profile system mirrors PBT's two-layer model. PROFILES in src/tuners/utils/profiles.py maps --config to a TunerProfile carrying the default worker count and a matched BenchmarkConfig; individual flags then override the profile under the "None means keep the profile default" convention. The profiles are rapid / standard / thorough / research (worker counts 2 / 4 / 8 / 12) — PBT's extreme is intentionally omitted, as it is population-scale specific and LHS is not a population method. Each strategy additionally owns a small {profile: scalar} map for the one hyperparameter it adds; LHS uses LHS_DESIGN_SIZE_BY_PROFILE (8 / 32 / 512 / 1024).

Per-batch baseline-snapshot restoration

Each LHS design batch now restores the shared read-only baseline snapshot on the per-profile cadence (snapshot_restore_interval: rapid=10 / standard=5 / thorough=1 / research=1, numerically identical to PBT's restart cadence), so a fresh batch no longer inherits the drifted DB state left by the previous one. The CLI/profile choice is combined with the workload bundle's own decision via a logical AND in src/tuners/base.py: a forced read-only / TPC-H auto-disable still wins, but otherwise the operator's --enable-snapshots / --disable-snapshots / --snapshot-restore-interval selection is honored.

HTML-log parity

The CLI attaches add_html_file_logging and writes a timestamped lhs_design_<ts>.html under the resolved output root, matching the HTML logs PBT and BO already produce.

Probe-disk diagnostics

--probe-disk (default on) calibrates the per-worker disk I/O budget with a short fio probe. src/utils/hardware_info.py now emits a WARNING when probing was requested but fio is absent, instead of silently falling back to the heuristic budget. That file is shared with PBT/BO but sits outside the copy-not-refactor boundary (it is not under src/tuner/ or src/scripts/bo_baseline/), so the additive diagnostic improves all three strategies.

Rescoring relocation

The global score-recalibration utilities moved from the now-deleted src/utils/rescoring.py into src/tuners/utils/calibration.py, with all consumers repointed. PBT and BO remain unmodified. (During the 2026-07 migration the post-hoc recalibration pathway was removed from the tuner lifecycle entirely and this leaf relocated once more to src/utils/calibration.py — see the 2026-07-17 addendum.)

Addendum (2026-07-17): copy-not-refactor invariant superseded — PBT migrated, src/tuner/ deleted

The original decision above deliberately extracted the shared lifecycle by copy, leaving src/tuner/ (PBT) and src/scripts/bo_baseline/ (BO) unmodified so the paper's headline strategies carried no migration risk. Once LHSDesignTuner and the shared surface proved stable, that invariant was retired on purpose: PBT was migrated onto BaseTuner and the legacy package was removed. This addendum records the reversal; the historical body above is left intact as the original decision.

What changed:

  • PBT is now a BaseTuner strategy. PBTTuner(BaseTuner) lives at src/tuners/pbt/tuner.py. The PBT-specific core relocated under src/tuners/pbt/ — population.py, evolution.py, config.py (was tuner_config.py), and worker.py (PBTWorker).
  • Worker was split. The generic evaluation vehicle BaseWorker lives at src/tuners/engine/worker.py; PBT evolution mechanics (is_ready/clone_from/perturb) moved onto PBTWorker(BaseWorker) in src/tuners/pbt/worker.py.
  • The engine layer is shared. orchestrator.py, barriers.py, restart_policy.py, and worker.py sit under src/tuners/engine/, with the former evaluate_worker monolith decomposed into activation.py, maintenance.py, feature_refinement.py, worker_metrics.py, and reliability_gate.py.
  • Foundation modules moved out of the PBT tree (Phase 1): knob_space.py and knob_loader.py → src/knobs/; workload.py → src/benchmarks/.
  • The legacy src/tuner/ package is deleted. Its entry point is gone; invoke PBT via the unified router python -m src.tuners pbt ... (or the direct door python -m src.tuners.pbt ...). References to python -m src.tuner.main are dead.
  • BO has not yet migrated — it remains at src/scripts/bo_baseline/ and is the next arc, adopting the same BaseTuner + subpackage + CLI-router pattern.

Addendum (2026-09-14): BO migrated onto BaseTuner

The "next arc" above is done. BO now lives at src/tuners/bo/ as BOTuner(BaseTuner) and is invoked through the unified router python -m src.tuners bo ... (direct door python -m src.tuners.bo ...); the src/scripts/bo_baseline/ package is gone. Session serialisation is the shared write_bo_results in session_writer.py. Per src/tuners/utils/types.py, PBT, LHS-design, and BO have all been migrated onto this framework. The historical body above records the point-in-time state before this migration and is left intact.

Addendum (2026-10-26): BaseTuner hook surface finalized

The original decision called the BaseTuner API provisional: it had only been validated against LHS, so "its hook surface may shift before PBT/BO adopt it". All three strategies now run on it, and the epic #193 closes the provisional phase. This addendum records what was fixed and why. The historical body above is left intact. Its hook list (which named setup and teardown as abstract hooks) is superseded by the contract below. setup() is a concrete phase sequence, and teardown() is a concrete default that strategies may extend.

The living reference for the contract is the module docstring of src/tuners/base.py. Tests in tests/unit/tuners/test_base.py parse that docstring and check it against each strategy's overrides, so the reference cannot drift from the code silently.

Mandatory hooks (#194)

Eleven members are abstract, so a strategy that omits one fails at construction instead of inheriting a placeholder: propose_initial_configs, step, should_stop, collect_best, build_session_payload, config_summary_lines, total_evaluations, converged, and the properties max_rounds, seeded_config_count and round_label. The first five were abstract from the start. #194 made the other six abstract: all three strategies already overrode them, so their base defaults were dead code that would only have hidden an omission.

best_config_fractions went the other way. All three strategies overrode it with the same body, so it became one concrete shared method that encodes against full_knob_space (a maintainer decision during the review of #194).

Persisted history keys (#195)

The per-record history keys (round index, elapsed seconds, overhead seconds) used to be derived from the display noun round_label, so renaming a label silently changed the on-disk schema. They are now declared explicitly per strategy in HISTORY_KEYS in src/tuners/utils/session_schema.py, the same module that holds the session readers. The reader alias tuples are derived from that registry, so every key a writer emits is one the reader recognises. A strategy with no registered keys fails at construction. round_label is now display-only.

Shared resource-adoption seam (#196)

Solo BO used to re-implement the tail of _resolve_resources (store the profile, resolve hardware ranges, apply the tuning-mode filter). That tail is now BaseTuner._adopt_worker_resources, and every _resolve_resources path ends in it. The seam is setup-phase only: it refuses to run once the orchestrator exists, because a later caller would also have to update the environment and orchestrator built from the old profile.

Distributed mode stays a PBT-only override

Distributed bring-up remains a set of PBT overrides of the setup phases _resolve_resources, _create_environment and _bring_up_instances, plus a PBT teardown that shuts the fleet down. It was deliberately not lifted into BaseTuner (a maintainer decision, recorded as out of scope in #193).

Rationale:

  • Only PBT runs across a device fleet. PBT is the strategy whose population maps one worker to one device. BO is sequential, and LHS is the SCALPEL design sweep; neither has a fleet run to support.
  • BO reproduces one fleet device instead. For the matched PBT-vs-BO comparison, solo BO (--no-cotenant) adopts the worker_resources that a distributed PBT session serialized, through the shared _adopt_worker_resources seam. This gives BO the same per-device envelope without BO running distributed itself.
  • A base-level abstraction would be speculative. With one implementer, a generic distributed hook would only move the PBT code into the base and widen the contract every strategy must reason about.

One point marks where the shared seam ends. After fleet bring-up, PBT replaces the coordinator's provisional resources with the device resources the agents report. That promotion runs after the environment and orchestrator exist, so it updates the knob spaces and the orchestrator's memory budget in place and deliberately does not call _adopt_worker_resources. The fleet mechanics (bootstrap, handshake, skipped knob prune) are described in Distributed tuning. Outside distributed mode, each of these distributed overrides delegates to super().

Closing tickets

  • #193 — Finalize the BaseTuner hook surface (epic)
  • #194 — Make always-overridden hooks abstract
  • #195 — Decouple persisted history keys from the round label
  • #196 — Move BO's resource adoption onto a shared base seam
  • #197 — Rewrite the base docstring and hook contract to match the code
  • #198 — This addendum