polars_online.gram

The accumulators, read back (docs/ENHANCEMENTS.md E46).

gram() hands back the matrices the models solve against, from one pass over data that is never materialized. This module is what to do with them afterwards: pool shards, take a subset, read a correlation, solve a ridge, walk a lasso path, put standard errors on coefficients, and diagnose collinearity.

Every function takes the mapping gram() produces – columns, means, comoments, cross_moments, target_weights, target_means, target_vars, n_eff, n_kish, target_n_kish – and merge() and subset() return one of the same shape.

The arithmetic is the models’ own, so solve() on a spec’s Gram reproduces that spec’s coefficients and lasso_path() reproduces the lasso model’s path. What it is not is the same arithmetic to the last bit: the models factorize with faer’s Cholesky and numpy with LAPACK’s LU, which round differently in the last place or two. The tests hold the two to a relative tolerance, not to equality, and say so.

Requires numpy, which is an optional extra of this package (pip install polars-online[numpy]) – not a dependency, as it is not one of polars’ either. Nothing here needs scipy or scikit-learn.

polars_online.gram.coef_stats(g: dict[str, Any], coef: Sequence[float], *, target: str | int = 0, features: Sequence[str | int] | None = None) → dict[str, Any]

Residual variance, standard errors and t-statistics for a fit.

This is what the target moments were added for (E45): with Var[y] in the Gram, a saved state answers “how good is this fit, and which coefficients are real” without the rows:

resid_var = Var[y] - 2 b' Cov[X, y] + b' C b
sigma2    = resid_var * n / (n - k)          # n = target_n_kish
se        = sqrt(diag(inv(C)) * sigma2 / n)
t         = b / se

Returns resid_var, sigma2, r2, n (the Kish size the correction and the errors use), se and t – the last two arrays over the same slots as coef, with the intercept’s entry nan (its standard error depends on the design’s centring, which the Gram has already absorbed).

n is Kish’s effective sample size, not n_eff: a weighted stream’s weight sum is not a count, and dividing by it would report standard errors too small by the factor the weights are unequal by. The rows behind an exponentially weighted fit are also neither independent nor identically distributed, so read a t here as a scale for comparing coefficients, not as a p-value.

ValueError if the Gram has no target moments – a state saved by 0.2.0 or earlier cannot answer this.

polars_online.gram.condition(g: dict[str, Any], *, features: Sequence[str | int] | None = None) → dict[str, Any]

Belsley’s collinearity diagnostics for the accumulated design.

Returns singular_values (of the column-scaled design, largest first), condition_indexes (s_max / s_j), kappa (the largest of them) and proportions – the variance-decomposition proportions, one row per component and one column per feature, each column summing to 1.

A component with a large condition index and a large share of two or more columns’ variance is a near-dependency between exactly those columns, which is what makes this worth more than a single kappa: it says which columns are the problem, where a VIF only says that one is. Belsley’s rule of thumb is an index above 30 with two proportions above 0.5.

The design is scaled to unit column length first (Belsley’s prescription), but not centred: the intercept is part of the collinearity when a column is nearly constant, and centring hides that. singular_values are of that scaled raw matrix, so they are the square roots of the eigenvalues of the scaled second-moment matrix.

polars_online.gram.correlation(g: dict[str, Any]) → Any

The correlation matrix of the columns, from the centred co-moments.

nan in the row and column of a constant one (the intercept included: a constant has no correlation with anything, and reporting 0 there would read as “independent”). The diagonal is 1 where the variance is positive.

polars_online.gram.from_row(row: Any) → dict[str, Any]

A closed group’s row (docs/ENHANCEMENTS.md E54) as the mapping gram() returns, so everything in this module works on it:

for row in bank.closed_groups().iter_rows(named=True):
    g = po.gram.from_row(row)
    beta = po.gram.solve(g, target=0)

Takes a one-row frame, a row of iter_rows(named=True), or a mapping. The row’s comoments is the upper triangle with the diagonal, row by row, and its cross_moments is row-major (n_targets, k); this expands both.

The result is what gram() would have returned for that group bit for bit, except the co-moment matrix’s lower triangle, which is the upper one mirrored. The two differ in the last bit or so and not more: the accumulator updates C[i][j] and C[j][i] with the same two products in the opposite order, which does not commute in IEEE arithmetic (docs/PERFORMANCE.md §14). Everything read off the matrix – a solve, a correlation, a condition number – is unaffected at that scale, and the packed half is what makes the closed row half the size.

A row of a kind that keeps no accumulators (its columns is null) raises ValueError: there is no Gram to make.

polars_online.gram.lasso_path(g: dict[str, Any], lambdas: Sequence[float], *, l1_ratio: float = 1.0, penalty_weights: Sequence[float] | None = None, target: str | int = 0, features: Sequence[str | int] | None = None, max_iter: int = 1000, tol: float = 1e-07) → Any

The elastic-net path from the Gram, one row of coefficients per lambda.

The lasso model’s coordinate descent (Lasso::solve), run offline: on the standardized (correlation-form) matrix, warm-started down the path in the order given, with

b_i = soft(rho_i, l * l1_ratio * pw_i) / (C_ii + l * (1 - l1_ratio) * pw_i)

where rho_i is the standardized cross-correlation less the other columns’ contributions and soft(v, t) = sign(v) * max(|v| - t, 0). Coefficients come back in original units with the intercept recovered from the means, so a row is directly comparable to bank.coef().

penalty_weights scales the penalty per feature (in features order, or columns order without the intercept): 0 leaves a column unpenalized, and a column the stream found constant is dropped whatever is asked for. The online model has no such parameter – it is the one thing here that the models do not also do, and it is cheap offline because the path is re-walked rather than carried.

Give lambdas from large to small, as a path is meant to be walked: the warm start makes that both faster and better conditioned. max_iter and tol are the model’s max_cd_iters and cd_tol.

polars_online.gram.merge(grams: Sequence[dict[str, Any]]) → dict[str, Any]

Pool the Grams of disjoint row sets into the Gram of their union.

Chan, Golub and LeVeque’s update: the pooled co-moments are the weighted average of the parts’ plus the spread between their means, and every quantity is a sum of parts rather than a difference of cumulative sums – so pooling a thousand shards loses no more precision than pooling two. With weights W_a, W_b and mean gap d = m_b - m_a:

W = W_a + W_b
m = m_a + (W_b / W) * d
C = (W_a * C_a + W_b * C_b) / W + (W_a * W_b / W**2) * outer(d, d)
Q = Q_a + Q_b

Use it to pool accumulators that share a weighting: one per shard of a pass, one per group being combined, one per worker. Not two halves of a decayed stream in time order – each part’s weights are relative to its own last row, so the earlier part is over-weighted by exactly the decay between them. Either run the parts under an infinite halflife, or scale the earlier part’s n_eff by lam**dt and its sum(w**2) by lam**(2*dt) before merging (the means and co-moments are unaffected, being weighted means already).

lags and lag_comoments come back None: a lagged cross-moment pairs a row with the row l back within its own part, and the pairings across a part boundary are what no part holds.

Every part must have the same columns and targets; a part with no n_kish or no target moments (a state saved by 0.2.0 or earlier) makes the merge report None for those, since the sums behind them are not there to add. group and instance come back as None: a pooled accumulator is no longer one group’s or one instance’s.

Merging one Gram returns it unchanged; merging none is a ValueError.

polars_online.gram.solve(g: dict[str, Any], *, ridge: float | Sequence[float] = 0.0, target: str | int = 0, features: Sequence[str | int] | None = None, standardize: bool = False) → Any

Ridge coefficients from the Gram, in the features’ original units.

The model’s own algebra (EwRidge::solve), so the result is the fit that spec would report on the same accumulator:

• standardize=False adds ridge to the diagonal of the raw second-moment matrix, leaving the intercept unpenalized;

• standardize=True centres, scales to correlation form, adds ridge there, then unscales and recovers the intercept from the means – so ridge means the same thing whatever the features’ units. A column with zero variance is dropped with a coefficient of 0 rather than making the system singular.

Pass the standardize the spec used, or the numbers will not match its coef(). With an intercept in columns the returned vector starts with it, in polars_online.spec.coef_index() order.

ridge may be a sequence, and then the return is one row per value. A grid rides a single eigendecomposition wherever the penalty is uniform in the basis being solved – always with standardize=True, and without an intercept otherwise: with V d V' in hand every ridge is V diag(1/(d + r)) V' b, which is what makes a grid of fifty cheap. An unstandardized fit with an intercept leaves that one column unpenalized, so its penalty is not a multiple of the identity and each value costs a factorization. That is the model’s arithmetic, and reproducing it is worth more here than the shortcut.

target picks the target by name or position; features narrows the regressors (equivalent to subset() first, and refusing the intercept, which the solve handles itself).

polars_online.gram.subset(g: dict[str, Any], cols: Sequence[str | int]) → dict[str, Any]

The Gram of a subset of the columns, in the order given.

Exact, not approximate: a marginal set of moments is a sub-block of the joint ones, so this is a selection rather than a recomputation, and a regression on the subset is the regression the full accumulator implies. That is the point – forward stepwise, an information criterion over feature sets, or an r-column fit read off a k-column stream all fall out of one pass.

Names or positions, and the intercept may be selected like any other column. Targets are untouched: they index a different axis.

polars_online.gram.vif(g: dict[str, Any], *, features: Sequence[str | int] | None = None) → Any

Variance inflation factors: 1 / (1 - R2_j) for each column on the rest, straight off the diagonal of the inverse correlation matrix.

The intercept is not a regressor and is left out by default (its VIF is undefined – a constant is perfectly explained by any other constant). A column the stream found constant reports inf.

Above about 10 the coefficient of that column is mostly noise; the fix is a ridge, a subset, or a feature set the spec already knows how to fit beside the full one.