Session JSON Schema¶

Every tuning run, evaluation comparison, and analysis pass emits or consumes one of three JSON shapes:

PBT session — produced by python -m src.tuners pbt
BO session — produced by python -m src.tuners bo
Comparison report — produced by python -m src.evaluation

This page is the schema reference for tooling authors and reviewers. The session loader in src/evaluation/loader.py is the authoritative implementation; this doc summarises what it expects.

All three schemas have evolved over the project's lifetime. The loader is schema-tolerant — sessions written under older versions still load with default values for fields added later. Fields described below as *added in vX* are absent in older artefacts.

Conventions¶

Times are ISO 8601 strings (UTC) unless otherwise noted.
Durations are floats in seconds.
Memory sizes follow the originating layer's unit — ram_bytes is bytes, shared_buffers is whatever unit PostgreSQL reported (typically 8kB pages).
Knob values are stored at the resolved (post-verify()) granularity, not the suggested granularity — see configuration-management §Verifying applied config.
null is used for "field absent / not applicable"; missing keys are equivalent to null for downstream tooling.

Timing instrumentation (v1.1)¶

Sessions written from 2026-06 onward carry tuning_session.timing_schema_version. v1.0 introduced per-component wall-clock instrumentation; v1.1 (current) is a non-breaking refinement that strips the redundant summary field at non-aggregating layers — see Changes from v1.0 below.

Clock. All durations come from time.monotonic() via the TimingRecorder / TimingRecord primitives in src/utils/timing.py. Wall-clock timestamps (filenames, log lines, ordering) come from session_timestamp() in src/utils/session_clock.py. Durations and wall-clock timestamps are deliberately decoupled — durations are immune to NTP drift, leap seconds, and DST.

Hierarchy. Timing data lives at three nested layers:

Session-level — bootstrap_breakdown (one-shot setup costs incurred before the gen loop) and timing_summary (mean/std/n/min/max/total per component, aggregated across every (generation, worker) tuple in the session). Both carry a summary field — this is where aggregation matters.
Round-level — whole-round work not attributable to any single worker (currently only PBT's evolve). There is no record-level timing block: these records are appended to the first worker's history[i].worker_scores[0].timing.records, after that worker's own components, so they still reach timing_summary. Older sessions carried them in a separate generation_history[i].timing block.
Worker-level — history[i].worker_scores[j].timing is a per-worker block covering everything from configuration apply through scoring. Shape: { "records": [...] } — no summary because each component appears once per worker per round (n=1).

The records field is the ordered list of TimingRecord entries (component name, duration, optional metadata) at every layer.

The summary field — present only at session level (timing_summary and bootstrap_breakdown.summary) — is aggregate() output keyed by component name: {n, mean, std, min, max, total}. std is population standard deviation (statistics.pstdev), 0.0 when n == 1.

Changes from v1.0¶

v1.0 emitted summary at every layer uniformly. At per-worker / per-gen scope each component appeared exactly once per evaluation, so the summary block had n=1, std=0.0, and mean=min=max=total=record.seconds — pure JSON noise (~5x the bytes of the records block) without information beyond records[i].seconds. v1.1 emits summary only at the session-level layers where pooling actually produces non-trivial aggregates (timing_summary over all (gen, worker) tuples; bootstrap_breakdown.summary over the bootstrap component set).

Backwards compatibility: v1.0 readers that walk worker_scores[*].timing.summary will see a missing key. The analysis script (src/analysis/timing_breakdown.py) already pulls from records (and falls back to flattening when timing_summary is absent), so it consumes both v1.0 and v1.1 unchanged.

Component reference¶

The components currently emitted by the orchestrator, population, and tuner bootstrap. The source-file pointer is where the bracket is opened; new components are added by following the contributor guide.

Component	Layer	Semantics	Source
`setup_instances`	bootstrap	Bring PostgreSQL workers up (containers, datadirs, schema load).	`src/tuners/base.py` — `BaseTuner._bring_up_instances`
`verify_instances`	bootstrap	Probe each worker for liveness, version, capability.	`src/tuners/base.py` — `BaseTuner._bring_up_instances`
`prune_knobs`	bootstrap	Drop knobs unsupported by the resolved PG server version.	`src/tuners/base.py` — `BaseTuner._prune_unsupported_runtime_knobs`
`setup_snapshots`	bootstrap	Create per-worker baseline snapshots for fast restart.	`src/tuners/pbt/tuner.py` — `PBTTuner` bootstrap block
`apply_only`	worker	`ALTER SYSTEM` writes only — no reload, no restart. Sets `restart_required`.	`src/utils/applicator.py` — `KnobApplicator.apply_only`, invoked from `WorkloadOrchestrator.apply_configuration`
`activate_reload`	worker	`pg_reload_conf()` for SIGHUP / user / superuser-context knobs. Metadata: `strategy="reload"`.	`src/utils/applicator.py` — `KnobApplicator.activate`
`activate_restart`	worker	`restart_instance()` for postmaster-context knobs. Metadata: `strategy="restart"`.	`src/utils/applicator.py` — `KnobApplicator.activate`
`snapshot_restore`	worker	`env.restore_snapshot(worker_id)` when `restore_due=True`. Replaces `activate_*` on restore-interval generations (the restore IS the restart).	`src/tuners/engine/orchestrator.py` — `WorkloadOrchestrator.evaluate_worker`
`knob_verify`	worker	Read-back via `KnobApplicator.verify()` — confirms PostgreSQL accepted and quantised the values.	`src/tuners/engine/orchestrator.py` — `WorkloadOrchestrator.evaluate_worker`
`workload`	worker	Full workload execution (warmup + measurement). Metadata: `executor="internal"` or `executor="benchmark"` (sysbench / tpch).	`src/tuners/engine/orchestrator.py` — `WorkloadOrchestrator.evaluate_worker`
`score`	worker	`engine.compute_breakdown()` over the captured metrics.	`src/tuners/engine/orchestrator.py` — `WorkloadOrchestrator.evaluate_worker`
`evolve`	generation	`execute_exploit_explore` + `env.clone_instances` — the PBT step itself.	`src/tuners/pbt/population.py` — `Population.train_generation`

Component names are snake_case and stable: they are the dimension key in the cost-decomposition table, so renaming one silently breaks reproducibility of older sessions through the analysis script.

Observed vs. configured¶

The workload component's semantics depend on the executor metadata:

executor="benchmark" (sysbench, tpch): warmup and measurement are not separately bracketed — the C-binary owns the warmup/measurement boundary internally. The reported seconds is the wall-clock duration of the subprocess call, which dominates both phases plus any process startup overhead. The configured sysbench_warmup_seconds / sysbench_duration_seconds (or tpch_warmup_passes / measurement-pass count) live in tuning_session and give the configured-vs-observed perspective.
executor="internal" (template-driven JSON workloads): warmup and measurement are not yet separately bracketed in the v1.1 schema — the orchestrator emits a single workload span for both (orchestrator.py:411). Splitting them will land in a later schema bump. Until then, the bracket reports the combined wall-clock of both phases together.

For sysbench specifically, warmup / measurement durations could in principle be reported as the configured values with observed=False metadata. The v1.1 emitter does not add that metadata; downstream tools that need the breakdown should consult the tuning_session configuration fields and treat the bracket as a single-block total.

Worker timing JSON example¶

{
  "timing": {
    "records": [
      {"component": "apply_only", "seconds": 0.124},
      {"component": "activate_reload", "seconds": 0.087, "metadata": {"strategy": "reload"}},
      {"component": "knob_verify", "seconds": 0.205},
      {"component": "workload", "seconds": 300.45, "metadata": {"executor": "benchmark"}},
      {"component": "score", "seconds": 0.012}
    ]
  }
}

On a restore-interval generation, the record list contains a single snapshot_restore entry instead of activate_reload / activate_restart. On PBT, worker 0's list additionally ends with the round's evolve record. Neither layer carries a summary field in v1.1 — see Changes from v1.0.

The corresponding aggregate lives at the session level — timing_summary[component] accumulates each component's durations across every (gen, worker) tuple and emits {n, mean, std, min, max, total}. For the worker example above, the workload entry contributes one sample to timing_summary["workload"].

Time accounting¶

tuning_session carries three duration fields:

Field	Meaning
`total_time_seconds`	End-to-end session wall-clock (bootstrap + tuning loop). Kept for backwards compatibility with v0.0 sessions.
`tuning_time_seconds`	Measurement-loop time only — captured from immediately before the gen loop to its end. This is what the paper's wall-clock-to-deployment-ready-config plot uses. new in v1.0
`bootstrap_seconds`	`total_time_seconds - tuning_time_seconds`. The cost paid before the algorithm starts learning. new in v1.0

Legacy sessions (timing_schema_version absent) carry only total_time_seconds; the loader treats tuning_time_seconds / bootstrap_seconds / bootstrap_breakdown / timing_summary as None for those sessions and the analysis script must handle that case explicitly.

PBT session schema¶

File location: results/sessions/{workload_dir}/pbt/{tier}/traces/trace_{timestamp}.json

Top-level layout¶

{
  "tuning_session":          { ... session metadata ... },
  "scoring_policy":          "feature_driven_v2",
  "scoring_policy_version":  "2.0",
  "metric_reference_version":"v2",
  "workload_features":       { ... feature vector ... },
  "normalization_metadata":  { ... normaliser state ... },
  "score_breakdown":         { ... }  (best worker's breakdown)
  "best_configuration":      { ... }
  "worker_resources":        { "ram_bytes": ..., "cpu_cores": ..., "disk_type": ... },
  "warm_start":              { "enabled": false, ... },
  "history":                 [ ... per-round records ... ],
  "convergence":             { ... },  (BO only)
  "system_info":             { ... host metadata ... }
}

The five scoring-related top-level keys (scoring_policy, scoring_policy_version, metric_reference_version, workload_features, normalization_metadata) are duplicated from the equivalent fields under tuning_session for downstream-tool convenience. Older sessions only have the nested copy; the loader handles both layouts.

`tuning_session`¶

Field	Type	Notes
`knob_tier`	str	One of `minimal` / `core` / `standard` / `extensive`.
`num_knobs`	int	Tier size at session start.
`workload_type`	str	`oltp` / `olap` / `mixed`.
`benchmark_name`	str \| null	`sysbench`, `tpch`, or null for custom workload files.
`scale_factor`	float \| null	TPC-H scale factor.
`sysbench_tables`, `sysbench_table_size`	int \| null	Sysbench schema.
`sysbench_workload`	str \| null	`oltp_read_only` / `oltp_read_write` / `oltp_write_only`.
`sysbench_duration_seconds`, `sysbench_warmup_seconds`	float \| null	Per-evaluation timing.
`tpch_warmup_passes`	int \| null	TPC-H query-pass warmup count.
`tuning_mode`	str	`online` / `offline` / `adaptive`.
`population_size`	int	Workers in the population.
`total_generations`	int	Generations actually run (may be < `max_generations` due to early stopping).
`num_parallel_workers`	int	Workers run concurrently. added in v2
`enable_snapshots`, `snapshot_restore_interval`	bool, int	Baseline-snapshot policy. added in v2
`resample_probability`	float	PBT explore step: per-knob probability of a full resample from the prior instead of a local perturbation. Single source of truth is the `PBTConfig`/preset default (`0.1`); recorded under `tuning_session.strategy_params` so a completed run states the value it used. added in v2
`seed`	int	Master random seed.
`total_time_seconds`	float	Wall-clock duration (bootstrap + tuning).
`tuning_time_seconds`	float	Measurement-loop wall-clock only (excludes bootstrap). added in timing-schema v1.0
`bootstrap_seconds`	float	Bootstrap wall-clock only. added in timing-schema v1.0
`timing_schema_version`	str	`"1.1"` (current) from 2026-06 onwards; absent in legacy sessions.
`timestamp`	str	`YYYYMMDD_HHMM`.
`workload_features`	object	Workload feature vector (mirrored at top level).

`best_configuration`¶

{
  "score": 0.9164,
  "knobs": {
    "shared_buffers": 65536,
    "effective_cache_size": 262144,
    ...
  },
  "metrics": {
    "latency_p50": 12.3,
    "latency_p95": 38.1,
    "latency_p99": 84.0,
    "throughput": 4250.7,
    ...
  },
  "score_breakdown": { ... }
}

score is on [0, 100] for v2 sessions — CompositeScorer.compute_breakdown() multiplies the weighted-utility total by 100.0 (scorer.py) — and in [0, 1] for legacy fixed_v1 sessions written before that scaling. Always check scoring_policy before interpreting the magnitude.

knobs keys are the post-verify() quantised values PostgreSQL actually ran with — not the optimiser's suggestion.

metrics is a PerformanceMetrics.to_dict() payload — the full field list lives in src/utils/metrics.py.

score_breakdown is a ScoreBreakdown serialised dict — resolved weights, per-metric utility scores, reliability gate value, and policy version.

`workload_features`¶

The feature vector consumed by FeatureDrivenWeightModel. Standard fields:

Field	Range	Meaning
`read_ratio`	[0, 1]	Fraction of read queries.
`write_ratio`	[0, 1]	Fraction of write queries.
`olap_complexity`	[0, 1]	Multi-table-scan / aggregation density.
`join_intensity`	[0, 1]	Cross-table joins per query.
`aggregation_intensity`	[0, 1]	GROUP BY / window-function density.
`sort_intensity`	[0, 1]	ORDER BY / sort-driven operator density.
`concurrency_pressure`	[0, 1]	Effective concurrent worker count.
`working_set_millions`	≥ 0	Approx working-set rows (millions).
`query_mix_entropy`	[0, log(N)]	Shannon entropy over the query weights.
`tail_latency_sensitivity`	[0, 1]	Heuristic weight for p99 over p50.

Older sessions may have an empty workload_features: {} — the loader falls back to default features in that case.

`normalization_metadata`¶

Snapshot of the QuantileUtilityNormalizer state at the end of the session, used by the post-hoc rescoring helper in src/utils/calibration.py.

Field	Type	Notes
`normalizer`	str	`"QuantileUtilityNormalizer"`
`metric_reference_version`	str	`v1` / `v2`.
`anchors`	dict[str, [low, high]]	Per-metric calibration anchors.
`n_calibration_samples`	int	Sample count used for anchors.
`out_of_support_rates`	dict[str, float]	Drift detection signal.
`latency_metric`	str	Active percentile (`p95` for OLTP, `p99` for OLAP).
`padding_factor`	float	Per-policy safety factor.
`ranges_calibrated`	bool	Whether the normaliser left fallback bounds.

`worker_resources`¶

{
  "ram_bytes": 4294967296,
  "cpu_cores": 2.0,
  "disk_type": "ssd"
}

The per-worker resource slice computed by detect_worker_resources. See hardware-aware-normalization. Reviewers can check that the slice matches the system_info host totals divided by num_parallel_workers × 0.8.

`history[]`¶

One record per round — a PBT generation, a BO iteration, or an LHS batch. Every strategy builds its records through the shared build_generation_record (session_assembly.py), so all three share one base shape. A PBT record:

{
  "generation": 0,
  "best_score": 79.1,
  "mean_score": 74.0,
  "std_score": 4.2,
  "restart_count": 1,
  "timestamp": "2026-03-26T00:46:48.849150",
  "wall_clock_seconds": 287.3,
  "generation_elapsed_seconds": 287.3,
  "pbt_overhead_seconds": 1.8,
  "num_exploited": 0,
  "worker_scores": [
    { "worker_id": 0, "score": 71.0, "metrics": { ... }, "score_breakdown": { ... },
      "timing": { "records": [ ... ] }, "actual_config": { ... } },
    { "worker_id": 1, "score": 79.1, "metrics": { ... }, "score_breakdown": { ... },
      "timing": { "records": [ ... ] }, "actual_config": {} }
  ],
  "worker_configs": [
    { "worker_id": 0, "config": { ... } },
    { "worker_id": 1, "config": { ... } }
  ],
  "exploitations": [],
  "exploit_cohort": { "intended": 2, "achieved": 0, "shortfall": 2 }
}

Per-strategy keys¶

The round-index, elapsed, and overhead keys use each strategy's own vocabulary. They are registered in HISTORY_KEYS (session_schema.py), the single source of truth for both writer and reader:

Strategy	Round index	Round elapsed (s)	Strategy overhead (s)	Flat strategy-only fields
PBT	`generation`	`generation_elapsed_seconds`	`pbt_overhead_seconds` (the `evolve` span)	`num_exploited`, `exploitations`, `exploit_cohort`
BO	`iteration`	`iteration_elapsed_seconds`	`bo_overhead_seconds` (ask / tell / relabel)	`phase` (`"bootstrap"` or `"bo"`)
LHS	`batch`	`batch_elapsed_seconds`	`lhs_overhead_seconds` (always `0.0`)	`evaluated` (design indices in the batch)

mean_score / std_score are written only by population strategies (PBT). BO records have length-1 worker_scores / worker_configs.

Reading records portably¶

Indexing a record by one strategy's key (gen["generation"]) raises KeyError on the other strategies' traces. The tolerant accessors in src.tuners.utils.session_schema read every strategy's key and the legacy names below. get_strategy_field checks the nested per-record strategy_params block first, then the flat key; the others check the current keys first, then the legacy names:

Accessor	Returns	Legacy fallbacks
`get_history(data)`	the `history` list	top-level `generation_history`, `evaluation_history`
`get_iteration_index(record)`	the round index	`round`
`get_iteration_elapsed(record)`	round elapsed seconds	—
`get_strategy_overhead(record)`	strategy overhead seconds	`strategy_overhead_seconds`
`get_strategy_field(record, name)`	a strategy-only field (e.g. `phase`)	nested per-record `strategy_params` block

Field notes¶

wall_clock_seconds is cumulative seconds since tuning started, at the moment the round ended. The round's own duration is the *_elapsed_seconds key.
best_score: PBT and LHS write the best score of that round; BO writes the best score so far (running maximum).
restart_count: PBT writes the cumulative number of generations so far in which at least one worker restarted PostgreSQL; BO writes 1 if that iteration restarted PostgreSQL, else 0; LHS writes 0.
worker_scores[].metrics is a PerformanceMetrics.to_dict(), or null for a failed evaluation. score_breakdown is a ScoreBreakdown dict (recomputed from metrics when the tuner had none). timing is the worker-level block above.
worker_scores[].actual_config holds the SHOW-verified knob values read back on the device after the evaluation (distributed mode). It is {} when unavailable, so consumers can read the key unconditionally.
worker_configs[].config is the configuration the worker evaluated that round. PBT and LHS store raw knob values; BO stores hardware-relative fractions.

Records carry no per-record converged or best_worker_id; convergence is session-level (tuning_session.converged), and older traces that still have these keys should not rely on them. BO traces written before #199 store wall_clock_seconds as that iteration's evaluation time rather than cumulative time, write metrics: {} (not null) for a failed evaluation, and omit worker_scores[].actual_config and worker_scores[].timing when unavailable.

num_exploited is the number of poor workers that adopted an elite this generation (ready-pool exploits and dead-config rescues); exploit_cohort (bug B2, ticket #165) breaks that count against intent: intended is the quantile cohort max(1, int(population_size × exploit_quantile)), achieved is how many ready-pool workers filled that quantile — dead-config rescues are excluded, so achieved ≤ num_exploited — and shortfall = max(0, intended − achieved). A positive shortfall means the ready pool could not fill the quantile — the expected steady state under the #164 readiness cooldown, which re-arms a worker's ready gate whenever it adopts a new configuration; the same condition also emits a WARNING at the selection seam. The sibling exploitations array records the { "elite_worker_id", "poor_worker_id" } pairs behind num_exploited.

`convergence`¶

BO only. PBT and LHS record convergence in tuning_session.converged.

{
  "converged": false,
  "generations_without_improvement": 7,
  "early_stopped": false,
  "early_stopping_patience": 50
}

generations_without_improvement is BO's stale-iteration counter for early stopping.

`system_info`¶

{
  "cpu_model": "AMD Ryzen 9 5950X 16-Core Processor",
  "cpu_cores": 16,
  "ram": "62.7 GiB",
  "disk_type": "ssd",
  "pg_version": "16.2",
  "os": "Linux 6.5.0-15-generic"
}

The first six fields are required; later additions (e.g. python_version, docker_image) are present in newer sessions and absent in older ones.

BO session schema¶

File location: results/sessions/{workload_dir}/bo/{tier}/traces/trace_{timestamp}.json (older pre-rename sessions may use bo_runs/{tier}/tuning_sessions/bo_results_{timestamp}.json).

The schema is structurally identical to the PBT session schema with one optimiser-specific addition under tuning_session:

Field	Type	Notes
`optimizer`	str	`"bo_smac3"`
`bo_library`	str	`"smac"`
`bo_surrogate`	str	`rf` (Random Forest) or `gp` (Gaussian Process).
`bo_acquisition`	str	`"EI"` or facade default.
`iterations`	int	Total iterations completed.
`num_parallel_workers`	int	Co-tenancy degree — concurrent instances during each BO measurement window.
`resource_equalization`	bool	Whether BO inherited per-worker resource slices from `--pbt-session`.
`reference_pbt_session`	str \| null	Path to the reference PBT session.
`reference_pbt_knobs`	list[str] \| null	Knob names copied from the reference session.
`pilot_size`	int	Sobol pilot iterations before `expand_ranges_for_metrics()` froze the normaliser.

history[] for BO has one record per iteration, keyed by iteration (not generation) with iteration_elapsed_seconds / bo_overhead_seconds and a flat phase field — see Per-strategy keys. worker_scores and worker_configs are length-1 arrays. Read records through the session_schema accessors so the consuming code (visualization, evaluation suite, comparison script) stays strategy-agnostic.

The population_size field is absent for BO (it's not population-based); consuming code should treat its absence as "single-worker iteration."

Comparison JSON schema¶

File location: results/comparisons/{workload_dir}/{tier}/comparison_{timestamp}.json

Produced by python -m src.evaluation. The full schema:

{
  "comparison_metadata":   { ... },
  "session_info":          { ... },
  "session_scoring_metadata": { ... },
  "scoring_metadata":      { ... },
  "tuned_knobs":           { ... },
  "default_runs":          [ RunResult, ... ],
  "tuned_runs":            [ RunResult, ... ],
  "statistics":            { ... },   // "power_warning" is nested INSIDE statistics
  "system_info":           { ... }
}

`comparison_metadata`¶

Field	Type	Notes
`timestamp`	str	`YYYYMMDD_HHMMSS`.
`tuning_session_path`	str	Source PBT/BO session JSON.
`evaluation_log_path`	str	Path to the HTML log.
`benchmark`	str	Sysbench / TPC-H.
`repetitions`	int	Number of paired (default, tuned) runs.
`pair_seed_base`	int	Repetition `i` uses `pair_seed_base + i - 1`.
`evaluation_environment`	str	`"docker"` or `"bare-metal"`.
`resource_constraints`	object	Per-worker `ram_bytes`, `cpu_cores`, `disk_type`.
`reproducibility`	object	`python_version`, `postgres_version`, `docker_image`, `python_package_versions`, `benchmark_binary_paths`.

`default_runs[]` and `tuned_runs[]`¶

Each element is a RunResult (src/evaluation/types.py):

{
  "config_type": "tuned",
  "run_number": 1,
  "pair_seed": 50000,
  "order_in_pair": 2,
  "metrics": { "latency_p50": ..., "latency_p95": ..., "throughput": ..., ... },
  "score": 78.21,
  "duration_seconds": 91.4,
  "container_id": "..."
}

The arrays are length-repetitions. Pair i of default_runs[i] and tuned_runs[i] shares its seed; the paired statistical tests in statistics are valid only because of this seed pairing.

`statistics`¶

{
  "alpha": 0.05,
  "primary_endpoint": "score",
  "primary_significant": true,
  "secondary_endpoints": ["throughput", "latency_p99", "memory_pressure", "scan_efficiency"],
  "secondary_correction_method": "holm",
  "correction_method": "holm_secondary",
  "n_pairs": 5,
  "power_warning": "..." ,
  "significant_metrics": ["score"],
  "overall_improvement_pct": ...,
  "overall_improvement_ci": [low, high],
  "metrics": [ MetricComparison, MetricComparison, ... ]
}

Each MetricComparison:

{
  "metric_name": "score",
  "default": { "mean": ..., "std": ..., "median": ..., "iqr_lower": ..., "iqr_upper": ..., "values": [...] },
  "tuned":   { "mean": ..., "std": ..., "median": ..., "iqr_lower": ..., "iqr_upper": ..., "values": [...] },
  "improvement_pct": ...,
  "improvement_ci": [low, high],
  "p_value": 0.0234,
  "p_value_corrected": 0.0468,
  "cohens_d": 1.23,
  "significant": true,
  "higher_is_better": true,
  "endpoint_role": "primary",
  "correction_method": null
}

The primary endpoint's p_value is uncorrected (its p_value_corrected equals p_value). Secondary endpoints' p_value_corrected is Holm-adjusted across the secondary family. See evaluation-suite §Statistical analysis for the methodology.

`scoring_metadata` vs `session_scoring_metadata`¶

These two blocks look similar but answer different questions:

session_scoring_metadata — what the source tuning session was scored under. Carries the session's scoring_policy, scoring_policy_version, metric_reference_version, workload_features, normalization_metadata, and best score_breakdown. In a multi-arm comparison this is keyed per arm (pbt, bo), each with that arm's own workload_features. Use this to attribute scores in the source session correctly.
scoring_metadata — what this evaluation used to compute the comparison scores. May differ from the session metadata if --scoring-policy was used to rescore historical sessions under a newer policy. Use this to interpret the default_runs[].score and tuned_runs[].score values.

If they match (the common case), there's no ambiguity. If they differ, downstream tooling should display both versions to avoid misleading interpretation.

The workload_features in these two blocks are expected to differ for a PBT arm, and that is not a version mismatch. A tuning session persists the vector as it stood at the end of the run — PBT moves its features while searching by design — whereas the evaluation scores every arm with one vector re-derived from its own benchmark parameters. scoring_metadata therefore also carries the provenance of that choice:

Field	Meaning
`workload_feature_policy`	Policy identifier, currently `eval_static_prior`.
`workload_feature_policy_version`	Version of the derivation rule.
`workload_feature_source`	Which extractor call produced the vector.
`workload_feature_inputs`	The effective benchmark parameters it was derived from, so the vector can be recomputed.

Comparison files written before ADR-007 carry "workload_features": {} and no workload_feature_policy key; their absence identifies a comparison scored under the older feature-blind rubric. See ADR-007.

Common patterns for consumers¶

"Load a session and get the best config"¶

import json
path = "path/to/session.json"
with open(path) as f:
    data = json.load(f)
best_config = data["best_configuration"]["knobs"]
best_score = data["best_configuration"]["score"]

"Load a session via the canonical loader"¶

For anything beyond trivial inspection, use load_tuning_session:

from pathlib import Path
from src.evaluation.loader import load_tuning_session
session = load_tuning_session(Path("path/to/session.json"))
# Returns a typed TuningSessionData with version compatibility applied.

The loader handles all schema-version differences, populates default fields, infers benchmark / workload_type from the file path when the JSON omits them, and warns on metric-reference-version mismatches without blocking.

"Convergence curve across multiple sessions"¶

from src.tuners.utils.session_schema import get_history, get_iteration_index

best_so_far = []
running = float("-inf")
for record in get_history(data):
    running = max(running, record["best_score"])
    best_so_far.append((get_iteration_index(record), running))

This works on PBT, BO, and LHS traces. The pbt_vs_bo_comarison.py script does the same for its convergence plot, with the additional step of using wall_clock_seconds for the time-axis and rescore_metrics_globally() for cross-method score parity.

"Per-round time split"¶

from src.tuners.utils.session_schema import (
    get_history,
    get_iteration_elapsed,
    get_strategy_overhead,
)

for record in get_history(data):
    elapsed = get_iteration_elapsed(record)
    overhead = get_strategy_overhead(record)
    evaluation = elapsed - overhead  # time spent outside the strategy machinery

Legacy traces with no *_elapsed_seconds key read as 0.0 elapsed, so evaluation comes out negative there; skip records where elapsed is 0.0.

"Per-knob lineage"¶

from src.tuners.utils.session_schema import get_history, get_iteration_index

# Each worker's config per round, then the rounds where a knob changed.
configs = {}  # (round, worker_id) -> config
for record in get_history(data):
    for wc in record["worker_configs"]:
        configs[(get_iteration_index(record), wc["worker_id"])] = wc["config"]

knob, worker_id = "shared_buffers", 0
rounds = sorted(r for r, w in configs if w == worker_id)
changed_in = [
    later
    for earlier, later in zip(rounds, rounds[1:])
    if configs[(earlier, worker_id)].get(knob) != configs[(later, worker_id)].get(knob)
]

There's no direct lineage field; lineage has to be reconstructed from worker_configs across generations. For automation, prefer the visualization loaders in src/visualization/loaders/session.py.

Field-presence cheat sheet¶

Field	First version	Notes
`tuning_session.scoring_policy`	`fixed_v1` era	Always present.
Top-level `workload_features`	feature_driven_v2	Empty `{}` in older sessions.
Top-level `normalization_metadata`	feature_driven_v2	Empty `{}` in older sessions.
`worker_resources`	hardware-aware-normalization	Absent in earliest sessions.
`tuning_session.num_parallel_workers`	parallel-worker isolation	Absent in earliest sessions.
`tuning_session.enable_snapshots`	snapshot lifecycle	Absent in earliest sessions.
`comparison_metadata.reproducibility`	reproducibility checklist	Absent in older comparisons.
`tuning_session.timing_schema_version`	timing-instrumentation v1.0 (current value `"1.1"`)	Absent in pre-2026-06 sessions; loader treats absence as `"0.0"`.
`tuning_session.tuning_time_seconds`, `bootstrap_seconds`	timing-instrumentation v1.0	Absent in pre-2026-06 sessions.
`bootstrap_breakdown`, `timing_summary`	timing-instrumentation v1.0	Top-level. Absent in pre-2026-06 sessions. Both carry `records` + `summary`.
`history[].worker_scores[].timing`	timing-instrumentation v1.0	Absent in pre-2026-06 sessions. v1.0 carried `records` + `summary`; v1.1 drops the redundant `summary` here (each component has n=1 at this scope). Older sessions also carried a record-level `generation_history[].timing` block, now folded into worker 0.
Top-level `history`	unified session schema	Older sessions use `generation_history` (PBT) or `evaluation_history`; `get_history()` reads all three.
Per-strategy record keys (`iteration` / `batch`, `_elapsed_seconds`, `_overhead_seconds`)	unified session schema	Older BO traces may key records by `generation`; `get_iteration_index()` reads any registered key.
`history[].worker_scores[].actual_config`	distributed SHOW read-back	Absent in older sessions and in BO traces written before #199.

When in doubt, run the file through load_tuning_session() — the loader's compatibility branches are the source of truth on what older sessions look like and how to interpret missing fields.