polars_online.eval

Evaluation harness (docs/PLAN.md section 8).

Pure Polars over the output structs: per (spec, group, target) out-of-sample R^2, IC (correlation of prediction with realized target) and hit rate, either overall or in rolling windows measured in clock units. One group_by compares specs.

Everything here consumes the frame a ModelBank returns: the original columns plus one struct column per spec. It needs the whole frame – these are Polars aggregations over collected output – where a spec’s emit_metrics keeps the same numbers beside the model, O(state), for a stream too long to hold.

What each function raises is what unpack() raises, since each starts there: KeyError for a spec_name the frame has not got, TypeError for a column that is not a model’s prediction struct, ValueError for a slot whose target column cannot be found. A by or clock column the frame has not got is polars’ ColumnNotFoundError.

polars_online.eval.metrics(df: DataFrame, spec_name: str, *, by: Iterable[str] = (), targets: Sequence[str] | None = None, min_obs: int = 30, binary: bool = False) → DataFrame

Overall out-of-sample metrics per (slot, *by).

Rows where the prediction or the target is null are dropped, so warmup and skipped rows never enter the numbers; a group with fewer than min_obs rows left is dropped from the result rather than reported on too little. n is the rows counted, r2 out-of-sample R^2 against the realized mean, ic the correlation of prediction with target, hit_rate the fraction of sign agreements (rows with y == 0 excluded), mse the mean squared residual.

binary=True reads pred as a probability and y as a 0/1 label – the output of a sgd or ftrl fit with loss="logistic" – rather than a signed regression target (docs/PLAN.md task 76): hit_rate becomes accuracy at a 0.5 threshold and every row scores, an extra log_loss column is added (-(y*ln(p) + (1-y)*ln(1-p)), averaged), and r2/ic keep their formulas under names that mean something different on a 0/1 target – Brier skill score and point-biserial correlation. Nothing here reads the target’s values to decide which reading applies; name it explicitly, the way you chose the loss.

Raises as unpack() does; a by column the frame has not got is polars’ ColumnNotFoundError.

polars_online.eval.rolling_metrics(df: DataFrame, spec_name: str, *, clock: str, window: float, by: Iterable[str] = (), targets: Sequence[str] | None = None, min_obs: int = 30, binary: bool = False) → DataFrame

Metrics in non-overlapping windows of window clock units.

window_start is the left edge of each bucket (floor(clock/window)*window); the columns are metrics()’s, per window, and binary is metrics()’s. Raises as unpack() does, ValueError for a window that is not above 0, TypeError for a clock column that is not numeric, and polars’ ColumnNotFoundError for a clock or by column the frame has not got.

polars_online.eval.compare_specs(df: DataFrame, spec_names: Iterable[str], *, by: Iterable[str] = (), targets: Sequence[str] | None = None, min_obs: int = 30, binary: bool = False) → DataFrame

Stack metrics() for several specs, adding a spec column.

binary is metrics()’s, applied to every spec alike – compare specs that share a loss, not a logistic fit against a regression one. Raises as metrics() does for each; no specs give an empty frame.

polars_online.eval.unpack(df: DataFrame, spec_name: str, *, spec: dict | None = None, targets: Sequence[str] | None = None) → DataFrame

Long form: one row per (row, prediction slot).

Returns slot (the struct field name), target (the target column it predicts), pred, y and every other non-struct column of df. Input columns named like the output ones (see RESERVED) are dropped – a target column called y would otherwise collide with the y output.

Pass spec to resolve each slot’s target exactly (via polars_online.spec.output_index()); without it a name-based heuristic is used, which can misattribute a target whose name embeds another’s.

Raises KeyError for a spec_name the frame has not got; TypeError for a column that is not a struct, or a struct with no pred_* fields (an ew_cov, kmeans, micro or ew_class output); ValueError when a slot’s target column cannot be found – the frame no longer has it, or targets does not name it – and, with spec, whatever polars_online.spec.output_index() raises for it.

polars_online.eval.seqtest(df: DataFrame, *, targets: Sequence[str] | None = None, a: str | None = None, b: str | None = None, a_suffix: str = '', b_suffix: str = '', by: Iterable[str] = (), min_periods: float = 0.0, name: str = 'seqtest') → DataFrame

polars_online.spec.seqtest() in polars expressions, over a frame in memory: the same e-processes, the same fields, row for row.

Column mode (no a/b): targets name the columns whose sign is tested. Compare mode: a and b name two output structs of df (two specs the bank ran), targets the residuals both carry – t for resid_<t><a_suffix> of a against resid_<t><b_suffix> of b; every t they share when None – and the sign tested is that of |resid_b| - |resid_a|, positive when a was closer.

Returns df with a struct column name holding, per target t and read before the row, as the bank emits them: log_e_pos_<t>, log_e_neg_<t>, n_pos_<t>, n_neg_<t> in column mode, log_e_a_<t>, log_e_b_<t>, wins_a_<t>, wins_b_<t> in compare mode, then n_eff – the rows before this one in its by group – with every other field null until n_eff reaches min_periods. by runs one process per group, in row order (.over(by)). A null, zero or NaN value bets nothing and counts nothing, as in the bank; what the bank adds is the clock (session, on_clock_reset), which a frame in memory has not got.

Per target, with s the sign and the counts before the row:

lam_pos = max(0, (n_pos - n_neg) / (n_pos + n_neg + 1))
log_e_pos += log1p(lam_pos * s)         (lam_neg, log_e_neg likewise)

tests/test_seqtest.py holds the bank’s struct to this one to the last bit; the difference is that the bank is O(state) over a stream and this is O(rows) over a frame.

Raises ValueError for a without b (or the reverse), for column mode without targets, and for a target neither side has a residual for (naming the fields it does have); KeyError for a spec the frame has not got, TypeError for one that is not a struct.

polars_online.eval.sums(df: DataFrame, spec_name: str, *, by: Iterable[str] = (), targets: Sequence[str] | None = None, spec: dict | None = None, weight: str | None = None, binary: bool = False) → DataFrame

Reduce a chunk of output to the sufficient statistics of its metrics (docs/ENHANCEMENTS.md E49).

metrics() needs the whole frame. This needs one chunk at a time: ten doubles per (slot, target, *by), which merge_sums() adds together and from_sums() turns back into the same numbers. A run that compares fifty slots over a billion rows then keeps ten doubles per key instead of writing the rows out to evaluate them later.

The columns beside the keys are SUM_FIELDS: n rows and w weight behind them, the weighted means mean_y and mean_pred, the centred sums m2_y, m2_pred and cov, the residual sum of squares sse, and hits / signed for the hit rate – hits counting sign agreements and signed the rows with y != 0, or (binary=True, metrics()’s reading) hits counting agreement at a 0.5 threshold and signed every row, since every row scores. Chunks reduced with different binary settings must not be merged – merge_sums() sums whatever is in hits/signed without knowing which reading produced it.

Centred, not raw. The obvious form – keep sum(y) and sum(y**2) and subtract – is one addition simpler and loses the variance entirely when the mean is large relative to the spread: a unit-variance target around 1e8 has var / E[y**2] of about 1e-16, and the subtraction has nothing left. merge_sums() pays for the centring with a parallel-axis term, which is a multiply, and keeps every digit. It is the same choice EwCov makes for the same reason (E11b).

Rows where the prediction or the target is null are dropped, as metrics() drops them, so warmup and skipped rows never enter the numbers. weight names a column to weight rows by; without it every row counts 1 and w equals n. spec, targets and the errors are unpack()’s.

polars_online.eval.merge_sums(first: DataFrame, *rest: DataFrame) → DataFrame

Add the sufficient statistics of disjoint row sets.

Exact, whatever the split: the means are pooled by weight and the centred sums pick up the parallel-axis term for the distance between each part’s mean and the pooled one:

w    = sum(w_g)
mean = sum(w_g * mean_g) / w
m2   = sum(m2_g + w_g * (mean_g - mean)**2)
cov  = sum(cov_g + w_g * (mean_y_g - mean_y) * (mean_p_g - mean_p))

That is the n-way form of Chan, Golub and LeVeque’s merge – every part enters as a sum, never as a difference of running totals – so merging a thousand chunks loses no more than merging two.

Keys present in one part and not another are carried through as they are. Merging one frame returns it unchanged.

polars_online.eval.from_sums(s: DataFrame, *, min_obs: int = 30) → DataFrame

The metrics metrics() reports, from sums() instead of rows.

Same columns and same numbers: n, r2 (out-of-sample against the realized mean), ic (correlation of prediction with target), hit_rate, mse, and rmse beside it. Which reading hit_rate is – sign agreement or accuracy at 0.5 – was fixed when the sums were built (sums()’s binary); this just divides hits by signed, so there is nothing to choose here. There is no log_loss column here (metrics()’s binary=True has it); SUM_FIELDS would need a mean and a weight for it, not added since nothing has asked for the chunked form yet. A key with fewer than min_obs rows is dropped, as metrics() drops it.

r2 and ic are null where they are undefined – a key whose target or prediction never varied has no correlation to report, and dividing by its zero variance would give an infinity that reads as a number.