Skip to content

Compare factors and parameters

To rank several factors or parameter values, evaluate each one on identical inputs and compare the flat metrics() of each result. Metric names follow <cs|ts>.h<horizon>.<method>.<statistic> (see results and metrics); undefined statistics are omitted rather than reported as zero.

import pandas as pd

from alphatonic import evaluate
from alphatonic.demo import synthetic_market

factor, prices = synthetic_market(n_assets=60, n_bars=900, ic=0.03, seed=5)


def smoothed(window):
    """A trailing mean of the raw score: a simple one-parameter family."""
    return factor.rolling(window).mean()


start = prices.index[59]  # all smoothing windows have scores from this bar
rows = {}
for window in (1, 5, 10, 20, 60):
    result = evaluate(
        smoothed(window),
        prices,
        start=start,
        horizons=[5],
        time_series={"bootstrap_samples": 0},
    )
    rows[window] = result.metrics()

board = pd.DataFrame(rows).T[
    ["cs.h5.spearman.mean", "cs.h5.spearman.hac_z", "cs.h5.spearman.n_periods"]
]
print(board.sort_values("cs.h5.spearman.mean", ascending=False).round(4))

Keep everything but the thing you vary fixed: the same prices, universe, window, horizons, delay and eligibility settings. The shared start above excludes the warm-up rows of the longest smoothing window. Also compare coverage: a shared window does not guarantee the same usable pairs when candidates have different missing scores.

The more you try, the less a good number means

A p-value is a statement about one pre-specified test. Evaluate twenty factors that have no skill and you may see p < 0.05 by chance:

import numpy as np

rng = np.random.default_rng(0)
p_values = []
for _ in range(20):
    noise = pd.DataFrame(rng.normal(size=factor.shape), index=factor.index, columns=factor.columns)
    row = evaluate(noise, prices, horizons=[5], time_series={"bootstrap_samples": 0}).summary
    p_values.append(row.query("method == 'spearman'").hac_p_value.iloc[0])
print(f"smallest of 20 null p-values: {min(p_values):.3f}")

One call to evaluate cannot know how many alternatives you tried. Run the candidates through a study, which counts them, and:

  • Hold out a final period that you evaluate once, after choosing: the study's holdout_start. Fix the model and parameter choices before the holdout; compute each holdout score using information available at its own timestamp.
  • Apply a multiple-testing correction, such as the study's adjusted_p (Bonferroni: p times the number of candidates), or report the number of candidates beside the best result.
  • Prefer parameters whose neighbours also work. An isolated peak deserves more scrutiny than a broad range of similar results.
  • Overlapping labels are handled by the HAC estimate within one evaluation. They do not make different horizons independent evidence.