Skip to content

Evaluation Suite

See also: Documentation Index, Evaluation Reproducibility Runbook, Feature-Driven Scoring, Workload Orchestrator, Statistical Analysis

Overview

The post-hoc evaluation suite at src/evaluation/ compares a tuned configuration produced by a tuning run (PBT or BO) against the PostgreSQL default configuration under controlled conditions, and emits a JSON report containing significance tests, confidence intervals, and effect sizes.

The suite is invoked via python -m src.evaluation and is the canonical way to answer the question "is the tuned configuration statistically better than the default, and by how much?" It is independent of the tuning loop — it ingests a saved tuning-session JSON, sets up a fresh database environment, and runs paired evaluations of default and tuned configurations.

The runbook commands and reproducibility checklist live in EVALUATION_RUNBOOK.md. This document covers the architecture and statistical methodology.

Table of Contents

  1. Where it sits
  2. Module layout
  3. ComparisonConfig
  4. ComparisonRunner flow
  5. Session loader and version compatibility
  6. Statistical analysis
  7. Output JSON schema
  8. Multi-arm comparisons
  9. Design decisions
  10. Related documentation

Where it sits

                       ┌─────────────────────────────────┐
                       │  Tuning session JSON            │
                       │  results/sessions/{wl}/{strat}/ │
                       │  {tier}/traces/trace_{ts}.json  │
                       └────────────────┬────────────────┘
                                        │
                                        ▼
                       ┌─────────────────────────────────┐
                       │       loader.py                 │
                       │  load_tuning_session(path)      │
                       │   → TuningSessionData           │
                       └────────────────┬────────────────┘
                                        │
                                        ▼
                       ┌─────────────────────────────────┐
                       │      ComparisonRunner            │
                       │                                  │
                       │  1. Build benchmark executor     │
                       │  2. Build environment            │
                       │  3. For repetition i:            │
                       │      run default config          │
                       │      run tuned config            │
                       │  4. Compute statistics           │
                       │  5. Save JSON + HTML log         │
                       └────────────────┬────────────────┘
                                        │
                                        ▼
                       ┌─────────────────────────────────┐
                       │  Comparison JSON + HTML log     │
                       │  results/.../comparisons/.../   │
                       │  comparison_*.json              │
                       └─────────────────────────────────┘

The runner reuses three components from the tuning side:

  • WorkloadOrchestrator to drive each evaluation.
  • DatabaseEnvironment (preferentially Docker) to isolate runs.
  • The scoring engine to produce comparable scalar scores under a chosen scoring policy.

Module layout

File Responsibility
__main__.py CLI entry point. Argparse, runbook-friendly defaults, dispatch.
runner.py ComparisonRunner — orchestrates default-vs-tuned evaluations, multi-arm comparisons, JSON serialisation.
statistics.py Wilcoxon signed-rank, paired bootstrap CIs, paired Cohen's d, Holm correction for the secondary endpoint family.
loader.py load_tuning_session(path) — parses session JSON across schema versions, normalises tuning config, extracts scoring metadata.
types.py Dataclasses (none are frozen): ComparisonConfig, TuningSessionData, RunResult, MetricComparison, ComparisonResult, PairwiseResult, MultiArmComparisonResult.
exceptions.py Domain-specific exception hierarchy: EvaluationError base, plus TuningSessionLoadError, ScoringMetadataSchemaError, DockerEnvironmentError, BenchmarkExecutionError, KnobApplicationError.

__init__.py is a WIP placeholder — it re-exports nothing (__all__: list[str] = []). Consumers import directly from the submodules: from src.evaluation.runner import ComparisonRunner, from src.evaluation.loader import load_tuning_session, from src.evaluation.types import ComparisonConfig.

ComparisonConfig

@dataclass
class ComparisonConfig:
    tuning_session_path: Path
    benchmark: Optional[str] = None
    repetitions: int = 5
    scale_factor: Optional[float] = None
    sysbench_duration: Optional[int] = None
    sysbench_tables: Optional[int] = None
    sysbench_table_size: Optional[int] = None
    sysbench_workload: Optional[str] = None
    sysbench_warmup_seconds: Optional[int] = None
    tpch_warmup_passes: Optional[int] = None
    pair_seed: int = 50_000
    use_docker: bool = True
    docker_image: str = "pbt-eval"
    output_dir: Optional[Path] = None
    scoring_policy: Optional[str] = None
    scoring_policy_version: Optional[str] = None
    metric_reference_version: Optional[str] = None
    bo_session_path: Optional[Path] = None
    # ... plus data_dir, colocate_output, force_recreate_baseline — see source

The CLI in __main__.py builds this dataclass; the runner consumes it. ComparisonConfig is a plain (non-frozen) dataclass; the runner treats it as read-only after construction, which keeps the reproducibility metadata in the output JSON honest.

Runtime overrides (Sysbench tables / table size / duration / warmup, TPC-H scale factor / warmup passes) are flat Optional fields directly on ComparisonConfig so the override field set is auditable; the runner falls back to session-recorded values when an override is unset.

ComparisonRunner flow

ComparisonRunner.run() — paired comparison
─────────────────────────────────────────────
1. session = load_tuning_session(config.session_path)
2. tuned_knobs = _resolve_tuned_knobs(session)
3. _validate_session_compatibility(session)            # benchmark family, workload type
4. _validate_docker_prerequisites() if use_docker
5. executor = _create_executor()                       # SysbenchExecutor / TPCHExecutor / WorkloadExecutor
6. env = _build_environment(executor)                  # Docker (preferred) or bare-metal
7. orchestrator = WorkloadOrchestrator(...)
8. _run_paired_comparisons(orchestrator, env):
       for i in range(repetitions):
           seed_i = base_seed + i - 1                  # identical for default and tuned
           run_default_i = _run_single(default_knobs, seed_i)
           run_tuned_i   = _run_single(tuned_knobs,   seed_i)
9. statistics = compute_comparison_statistics(default_runs, tuned_runs)
10. result = ComparisonResult(... runs, statistics, metadata, reproducibility)
11. _save_result(result) → comparison_*.json + logs/evaluation_*.html

Notable choices:

  • Identical paired seeds. Repetition i uses base_seed + i - 1 for both the default and the tuned run. The two configurations face the same workload sequence, so paired statistical tests (Wilcoxon, paired bootstrap CI, paired Cohen's d) are valid.
  • Default knobs come from KnobSpace.get_default_config(). They are not the cluster's current knobs; they are the PostgreSQL defaults captured by the same KnobSpace that produced the tuned config. This is the only way the paired test is fair on knobs the tuner explored.
  • Tuned knobs come from session.best_configuration.knobs. When the session JSON stores fractional values (hardware-relative knobs), _resolve_tuned_knobs calls KnobSpace.fractions_to_config(...) against the evaluation host's resources, not the original tuning host's. This is what makes "tune on a 16-GB host, evaluate on a 32-GB host" sound — the fractional encoding is the transfer medium.
  • Output partitioning by tier and workload. _resolve_output_dir writes to results/comparisons/{workload_kind}/{tier}/ derived from session.tuning_session.knob_tier, with a fallback to the tier segment following the session path's {strategy}/ (or legacy {strategy}_runs/) directory. This keeps the results tree navigable even when the session metadata is partial.

Session loader and version compatibility

Location: src/evaluation/loader.py

load_tuning_session(path) parses both PBT and BO session JSONs into a single TuningSessionData shape. The loader is deliberately permissive about historical schema variations:

  • Scoring policy default. Sessions without a scoring_policy field fall back to the default constants in src/utils/scoring/constants.py — feature_driven_v2, policy version 2.0, metric reference version v2. Runs from before scoring-v2 still load and compare correctly.
  • Tuning-config normalisation. _normalize_tuning_config() coerces numeric fields (population_size, total_generations, sysbench_table_size, tpch_scale_factor, etc.) from strings or floats into the right types, since older runs sometimes wrote durations as strings.
  • Benchmark / workload inference. When a session JSON omits benchmark_name or workload_type, _infer_benchmark_and_workload() derives them from the session path (results/oltp/oltp_read_write/... → sysbench OLTP; results/olap/... → TPC-H OLAP).
  • Version compatibility check. _check_version_compatibility warns (does not block) on metric-reference-version mismatches between sessions in a multi-arm comparison. The user can override the active scoring policy via --scoring-policy to force re-evaluation under newer weights — at which point the comparison JSON records both the original session policies and the active comparison policy.
  • Workload features are not taken from any session. The loader still reads each session's persisted workload_features (they are recorded per arm in the output and logged as divergence against the comparison rubric), but they never feed the score. The runner resolves one vector of its own via feature_policy.py and scores every arm with it — see ADR-007.

The runbook EVALUATION_RUNBOOK.md lists the metadata fields the loader populates so a reviewer can audit them.

Statistical analysis

Location: src/evaluation/statistics.py

The statistical layer is paired-design throughout. Given default_runs and tuned_runs of equal length, where pair i shares its seed:

def compute_comparison_statistics(
    default_runs: list[RunResult],
    tuned_runs: list[RunResult],
    benchmark: str,
    alpha: float = 0.05,
) -> ComparisonStatistics: ...
# primary endpoint ('score'), the 8-endpoint secondary family, the 10,000
# resamples and the RNG seed (42) are module constants / hard-coded, not params.

Endpoints

  • Primary endpoint: score — the composite score from the active scoring policy. Tested at α = 0.05 with no family correction.
  • Secondary endpoint family (4 endpoints) — throughput, latency_p99, memory_pressure, scan_efficiency (_SECONDARY_ENDPOINTS). The family is fixed and does not vary by benchmark; it is deliberately curated to metrics that are genuinely measured and are not monotone restatements of one another.
  • Secondary p-values are corrected with Holm's step-down procedure (see _holm_adjusted_pvalues).
  • Reported endpoints (5, never hypothesis-tested) — latency_p95, latency_p50, error_rate, tail_amplification, latency_variance (_REPORTED_ENDPOINTS). These appear in the summary table and the JSON with endpoint_role="reported", carrying no p-value of record: they are near-constant on healthy runs, highly correlated with a tested endpoint, or niche restatements.

The asymmetry — primary endpoint uncorrected, secondary family corrected — is intentional: the primary endpoint is what the optimisation actually targets; the secondary family is for understanding the direction of the win, not for additional confirmatory testing.

Tests applied per endpoint

For each endpoint, the statistics module produces a MetricComparison with:

Field Computation
default / tuned StatSummary (mean, std, median, iqr_lower, iqr_upper, values)
improvement_pct, improvement_ci Median-based percent improvement and its bootstrap 95% CI (p_value_corrected and endpoint_role also present).
p_value Wilcoxon signed-rank test on per-pair differences. Falls back to 1.0 when all differences are exactly zero (degenerate case for short reps).
improvement_ci Plain percentile bootstrap CI (10,000 resamples) on the median difference. RNG seed pinned to 42 for deterministic CI computation.
cohens_d Paired Cohen's d on the difference vector.
significant p < α (with Holm correction applied to the secondary family).

Power warning

_build_power_warning(n_pairs) returns a string like "Low statistical power at N=5: minimum possible two-sided Wilcoxon p-value is 0.0625, so p<0.05 cannot be reached even before correction." when fewer than 8 paired observations were used (it returns None at n ≥ 8). It is surfaced into the comparison JSON — nested under statistics — so reviewers can see it without re-reading the runbook.

Determinism

The bootstrap RNG is seeded explicitly (rng_seed=42 by default). The same input produces the same CI on every run. The benchmark-execution noise is bounded by Docker isolation but not eliminated; that's why we report mean ± std and a CI rather than a single point estimate.

Output JSON schema

A comparison JSON is structured as follows (truncated):

{
  "comparison_metadata": {
    "timestamp": "...",
    "tuning_session_path": "...",
    "evaluation_log_path": "...",
    "benchmark": "sysbench",
    "repetitions": 5,
    "pair_seed_base": 50000,
    "benchmark_parameters": { ... },
    "evaluation_environment": "docker",
    "resource_constraints": { "ram_bytes": ..., "cpu_cores": ..., "disk_type": "..." },
    "reproducibility": {
      "python_version": "3.11.x",
      "postgres_version": "16.x",
      "docker_image": "pbt-eval",
      "python_package_versions": { ... },
      "benchmark_binary_paths": { ... }
    }
  },
  "tuned_knobs":    { ... },
  "default_runs":   [ RunResult, RunResult, ... ],
  "tuned_runs":     [ RunResult, RunResult, ... ],
  "statistics": {
    "metrics": [ MetricComparison, MetricComparison, ... ],   // JSON LIST, one per endpoint
    "power_warning": "..." | null,
    "primary_endpoint": "score",
    "secondary_endpoints": ["throughput", "latency_p99", "memory_pressure", "scan_efficiency"]
  },
  "scoring_metadata":         { ... },   // policy used by THIS evaluation
  "session_info":             { "scoring_policy": ..., "scoring_policy_version": ..., "metric_reference_version": ... },
  "session_scoring_metadata": { "workload_features": {...}, "normalization_metadata": {...}, "score_breakdown": {...} },
  "system_info":              { ... }
}

Every field listed in the reproducibility checklist of the runbook is populated by the runner. A reviewer should be able to recreate the comparison from the JSON alone.

Multi-arm comparisons

ComparisonRunner.run_multi_arm() handles the case where multiple tuning sessions (e.g. several seeds, or PBT vs BO) need to be compared against the same default baseline.

python -m src.evaluation \
  --session results/.../trace_seed42.json \
  --session results/.../trace_seed123.json \
  --session results/.../trace_seed42.json \
  --repetitions 8

The runner:

  1. Runs the default configuration once for the full repetition budget; this is the shared control arm.
  2. Runs each tuned arm against its own paired seeds (still base_seed + i - 1).
  3. Computes pairwise statistics between every (arm, default) pair and between every (arm_a, arm_b) pair via compute_pairwise_statistics.
  4. Emits a MultiArmComparisonResult JSON with one PairwiseResult per pair plus the metadata block.

This is what feeds the multi-arm comparison plots produced by src/scripts/pbt_vs_bo_comarison.py and the comparison loader in src/visualization/loaders/comparison.py.

Design decisions

1. Paired design with shared seeds

Comparing against the default with independent seeds wastes statistical power; the workload variance dominates the configuration variance. Pairing repetitions on the same seed pulls the workload variance out of the comparison, leaving the configuration effect easier to detect with fewer reps.

2. Score is the primary endpoint, not throughput

A throughput-only primary endpoint biases the test toward configurations that trade tail latency for raw TPS. The composite score weighs latency, throughput, memory, and reliability together — testing on it answers the question the optimiser is actually asked to answer. Throughput remains in the secondary family for direction-of-win interpretation.

3. Holm correction on the secondary family only

Holm corrects for the multiple comparisons within the secondary family (latency, throughput, memory). The primary endpoint is a single confirmatory test and isn't part of any family. Bonferroni would over-correct given the secondary endpoints' known correlation structure.

4. Bootstrap CI seeded deterministically

Reviewers have to be able to reproduce the same CI from the same input — non-deterministic CIs would invite re-running until a "good" CI appears. Seeding the bootstrap RNG locks the result.

5. Docker preferred, bare-metal opt-in only

A reviewer reading a publication-facing comparison shouldn't have to wonder whether the result is contaminated by host noise. The runner defaults to Docker; bare-metal requires --no-docker and tags the output as reduced-isolation in evaluation_environment.

6. Output partitioning by (workload, tier)

Running comparisons across many sessions generates many JSONs. The path layout results/comparisons/{workload_kind}/{tier}/ keeps related artefacts adjacent, and downstream scripts can glob over a tier directory to assemble multi-arm plots without parsing every file.

7. Loader tolerates schema drift

A research repo accumulates session formats over months. The loader's compatibility branches mean today's evaluator can compare against a session from before scoring-v2 — the runner re-evaluates under the active policy and the comparison JSON records both versions for auditability.

File locations