Skip to content

Feature-Driven Scoring

See also: Documentation Index, Performance Evaluation, Metrics Validation, Workload Orchestrator

Overview

This document describes the migration from the older fixed scoring policy to the current feature-driven scoring system used by tuning, post-hoc rescoring, and evaluation. The objective is still a single scalar reward for Population-Based Training, but workload features now influence metric importance instead of relying on benchmark name alone.

The runtime pipeline is composed of four parts:

  1. WorkloadFeatures capture static and runtime workload characteristics.
  2. QuantileUtilityNormalizer maps raw metrics to monotonic utilities in $[0, 1]$.
  3. FeatureDrivenWeightModel computes metric weights from workload features and the active policy.
  4. CompositeScorer combines weighted utilities with a reliability gate and emits the final score.

What Changed

The older design used fixed, workload-specific weights. The current design keeps a compatibility policy for historical sessions, but shifts new runs to feature-conditioned weights. The score now reacts to workload shape rather than just benchmark label.

Aspect Fixed policy (fixed_v1) Feature-driven policy (feature_driven_v2)
Weight source Static per workload type Computed from workload features
Metric set latency_p50, latency_p95, latency_p99, throughput, memory_utilization, error_rate latency_p95, latency_p99, latency_variance, tail_amplification, throughput, throughput_variance, error_rate, memory_pressure, scan_efficiency, buffer_miss_rate
Compatibility Default for legacy sessions Default for new feature-aware runs
Weight behavior Preset and deterministic Floor-constrained softmax over feature-conditioned logits

Scoring Contract

The runtime score is computed as:

$$ S = 100 \cdot G \cdot \frac{\sum_{i=1}^{n} w_i \cdot u_i}{1 - w_{\text{error}}} $$

where:

  • $G \in [0, 1]$ is the reliability gate.
  • $w_i$ is the active weight for metric $i$.
  • $u_i \in [0, 1]$ is the normalized utility for metric $i$.
  • The active weights produced by FeatureDrivenWeightModel sum to $1$ and each metric receives a configured floor.

The final score is scaled by $100$ into the $[0, 100]$ range (scorer.py: final_score = total * 100.0). error_rate is excluded from the sum and its weight is divided out through the $1 - w_{\text{error}}$ denominator, so the reported score reflects only the non-error utilities gated by $G$.

Workload Features

src/utils/scoring/workload_features.py defines the extraction layer that produces the feature vector consumed by the weight model. The canonical feature groupings are:

  • read_ratio and write_ratio
  • olap_complexity
  • join_intensity
  • aggregation_intensity
  • sort_intensity
  • concurrency_pressure
  • working_set_millions
  • query_mix_entropy
  • tail_latency_sensitivity

The extractor is deterministic for Sysbench, TPC-H, and template workloads.

Sysbench encodes read/write mode, concurrency pressure, and working-set scale. TPC-H emphasizes OLAP complexity, joins, aggregation, and tail sensitivity. Template workloads derive feature values from query text and schema metadata.

Weight Model

FeatureDrivenWeightModel converts workload features into weights using a floor-constrained softmax:

$$ z_i = b_i + \sum_j M_{ij} f_j $$

$$ w_i = \alpha_i + \left(1 - \sum_k \alpha_k\right) \cdot \mathrm{softmax}\left(\frac{z_i}{T}\right) $$

where:

  • $b_i$ is the per-metric base weight.
  • $M_{ij}$ is the feature coefficient matrix.
  • $f_j$ is the workload feature value.
  • $\alpha_i$ is the floor for metric $i$.
  • $T$ is the softmax temperature.

The current implementation applies logarithmic compression to working_set_millions before multiplication so large benchmarks do not dominate the score.

The floor set must satisfy $\sum_i \alpha_i < 1$. That keeps every metric in the score while allowing the remaining weight mass to move with the workload.

Normalization

QuantileUtilityNormalizer maps raw performance metrics to $[0, 1]$ using robust quantile anchors instead of single-point min/max bounds. This keeps normalization monotonic while reducing sensitivity to outliers and one-off failures.

The normalizer:

  • uses calibrated anchors when history is available,
  • falls back to sensible anchors before calibration,
  • keeps naturally bounded metrics clamped to $[0, 1]$,
  • tracks drift through out-of-support rates,
  • recalibrates when drift persists,
  • expands anchors when saturation is detected.

This is what keeps the scoring signal stable across generations while preserving discrimination between candidate configurations.

Asymmetric, Direction-Aware Anchoring

Calibration (QuantileUtilityNormalizer.fit()) anchors each metric with an explicit good end and bad end rather than trimming both tails symmetrically. The bad tail is still trimmed robustly (a one-sided iqr_filter pass followed by the p05/p95 quantile), so a pathological-but-not-failed measurement cannot distort the range. The good tail is never discarded: the good-end anchor is placed strictly at/beyond the best non-failed observation, with headroom.

  • HIGHER_IS_BETTER: good end = q_high, set to max(observed) + headroom; bad end = q_low, the robust low quantile.
  • LOWER_IS_BETTER / ZERO_IS_BEST: good end = q_low, set to min(observed) − headroom; bad end = q_high, the robust high quantile.

Symmetric trimming (the pre-fix behaviour) discarded the single best (elite) observation, so the elite sat permanently outside the range and clamped at maximum utility — the scorer went blind to any further improvement past it (bug B9). Anchoring the good end beyond the best observation guarantees the current champion scores just under 1.0 with room to grow. The headroom may extend the good-end anchor slightly past the metric's physical range (for example just below 0 for a non-negative metric); this is intentional and only ensures the best observation stays strictly interior. See ADR-010.

Saturation Detection and Anchor Expansion

When several workers clamp a metric to the same anchor, they collapse to an identical utility and their configurations tie in the composite score even when they are genuinely different. detect_metric_saturation() counts, per metric, how many workers pin the upper utility bound (utility ≈ 1.0) and how many pin the lower bound (utility ≈ 0.0), and expand_ranges_for_metrics() widens the saturated anchors so ranking is restored.

Two rules govern this:

  • Quorum of 2 (SATURATION_QUORUM in src/utils/metrics.py). A metric is treated as saturated on a bound as soon as two workers clamp to it. Two is the principled threshold: two workers clamped to the same bound already produce identical utility, so two distinct configurations tie — the smallest count at which a saturation-induced ranking tie can exist. The earlier half-population quorum (max(2, n // 2)) hid real ties whenever fewer than half the population clamped.
  • Both ends expand. A metric clamped at both ends simultaneously (a dominant elite pinned at utility 1.0 and a crippled worker pinned at utility 0.0 on the same metric) reports both bounds. detect_metric_saturation() returns a list of saturated bounds per metric, and the consumer calls expand_metric_anchor() once for each end. Expanding one end never silently discards the other: expand_metric_anchor() moves only the saturated raw anchor and preserves the opposite anchor, so a second per-end call cannot overwrite the first's expansion.

Direction-aware anchoring maps utility-space saturation to the correct raw anchor (HIGHER_IS_BETTER: utility 1 ↔ q_high; LOWER_IS_BETTER/ZERO_IS_BEST: utility 1 ↔ q_low), so a LOWER_IS_BETTER metric clamped at utility 1.0 relaxes q_low rather than pushing q_high further away.

Reliability Gate

The reliability gate prevents unstable runs from dominating ranking.

  • Fatal failures set the gate to $0$.
  • Error rates at or above the fatal threshold set the gate to $0$.
  • Non-zero error rates below the threshold decay the gate linearly toward $0$.
  • Successful runs keep the gate at $1$.

The gate is applied before score aggregation, so a failed worker cannot win because of incidental utility values on unrelated metrics.

Serialization and Reproducibility

The scoring layer defines explicit contracts for auditability:

  • WorkloadFeatures stores the feature vector, source, and version.
  • MetricSnapshot stores per-metric raw and normalized values, weights, and contributions.
  • ScoreBreakdown captures the final score, policy, reliability gate, and component list.
  • NormalizationState stores anchors and normalization metadata for reproducible rescoring.

Legacy sessions continue to deserialize with fixed_v1, policy version 1.0, and metric reference version v1 through the defaults in src/utils/scoring/constants.py.

Migration Guidance

Use fixed_v1 when you need historical comparability or want to replay old sessions exactly. Use feature_driven_v2 when you want the score to reflect workload shape rather than benchmark name alone.

When enabling the feature-driven policy:

  1. Keep the feature extractor deterministic for the benchmark family you are running.
  2. Ensure the floor sum remains below $1$.
  3. Allow the normalizer to calibrate before interpreting small score deltas.
  4. Persist ScoreBreakdown and NormalizationState so rescoring remains reproducible.

For debugging, inspect the resolved weights and component breakdown rather than only the final scalar. That makes it much easier to confirm that feature shifts are influencing the intended metrics.

Validation Checklist

The migration is complete when the following hold:

  • workload feature extraction is deterministic,
  • the feature-driven weight model returns weights that sum to $1$,
  • every configured metric respects its floor constraint,
  • the normalizer produces stable utilities after calibration,
  • drift and saturation handling preserve score variance,
  • legacy sessions still load under the compatibility policy,
  • the targeted unit tests pass.

Scoring Engine Factory

The scoring stack is reached through a single factory:

from src.utils.scoring import create_scoring_engine
scorer: CompositeScorer = create_scoring_engine(metric_config)

create_scoring_engine(metric_config) lives in src/utils/scoring/init.py and is the only place that wires the four scoring layers together. Given a MetricConfig, it:

  1. Reuses an existing QuantileUtilityNormalizer from metric_config._normalizer if one was attached (e.g. by post-hoc rescoring); otherwise constructs a fresh normalizer.
  2. For fixed_v1, builds a per-metric weight override dict from metric_config.weight_latency / weight_throughput / weight_memory / weight_error so legacy sessions remain bit-identical.
  3. For feature_driven_v2, lets FeatureDrivenWeightModel derive weights at scoring time from the workload features.
  4. Returns a CompositeScorer configured with the resolved policy_id, workload type, normalizer, and weight overrides.

The orchestrator constructs the engine lazily under a lock so the first per-worker thread to call evaluate_worker doesn't race with peers building their own engine — see WORKLOAD_ORCHESTRATOR.md §Lazy thread-safe scoring engine. Every subsequent worker thread shares the same engine.

The post-hoc evaluation suite uses the same factory: when --scoring-policy feature_driven_v2 overrides a session originally tagged fixed_v1, the suite calls create_scoring_engine with a re-tagged MetricConfig, then rescores the persisted raw PerformanceMetrics.

Which feature vector post-hoc evaluation scores with

Because feature_driven_v2 derives its weights from the feature vector alone, the vector a comparison scores with is the rubric. The arms of a comparison do not agree on one: PBT deliberately moves its vector while it searches, BO and LHS keep the static prior, and a default arm has no session vector at all.

Post-hoc evaluation therefore uses neither — it re-extracts the static prior from its own effective benchmark parameters, applies that single vector to every arm, and records the policy and derivation inputs in the comparison JSON. Passing no vector is not equivalent: the workload-type MetricConfig constants define no feature priors, so an omitted vector collapses feature_driven_v2 to its bare base logits and stops distinguishing OLTP from OLAP.

See ADR-007 and src/evaluation/feature_policy.py.

Outlier Filtering

Location: src/utils/scoring/outlier_filtering.py

Calibrating the normalizer's quantile anchors against raw observations is sensitive to extreme outliers — a single hung query producing a 600-second latency would push the 95th-percentile anchor far above the typical operating range and collapse score variance for the next dozen generations.

iqr_filter(values, k=2.5) removes observations outside [Q1 - k·IQR, Q3 + k·IQR] before they enter the calibration set. The choice of k=2.5 is a compromise:

  • Classic 1.5× — too aggressive for noisy database metrics; rejects legitimate variance from autovacuum bursts and checkpoint writes.
  • 3.0× — too lenient; lets a single hung query past the gate.
  • 2.5× (chosen) — rejects only the tails that demonstrably distort calibration on real PBT/BO traces.

The filter returns a (filtered_array, metadata_dict) pair where the metadata records n_removed, original_size, the bounds used, and a fallback_used flag. When the input has fewer than 4 observations or IQR == 0, the filter falls back to the unfiltered values rather than producing degenerate bounds; both cases are surfaced in the metadata so post-hoc analysis can audit when calibration was unfiltered.

The filter is applied inside QuantileUtilityNormalizer.expand_ranges_for_metrics() immediately before quantile estimation, and inside the global rescoring helper rescore_metrics_globally() used by the PBT vs BO comparison script. Inside QuantileUtilityNormalizer.fit() the same filter is applied one-sided, to the bad tail only — the good tail is preserved so the best observation always survives to anchor the good end (see ADR-010). The filter is not applied at scoring time — only at calibration time — because individual scoring calls must remain monotonic in their inputs.

Source References