Results and metrics¶
evaluate returns an Evaluation: tables you can inspect, save and render. This page
describes the result tables, status values, metric names and saved formats.
Missing statistics are NaN in tables, blank in CSV files and omitted from metrics(). They are
never replaced by zero. Summary and time-series tables have a method column: spearman,
pearson, or (time series only) cosine. The cross-sectional periods table stores spearman and
pearson as separate columns. Correlations are taken against each analysis's target: the forward
return, or the vol-scaled forward return (see targets and scaling).
In-memory labels¶
result.forward_returns maps each horizon to a DataFrame of raw forward returns on the evaluation
grid. These remain raw returns even when an analysis uses a volatility-scaled target.
result.volatility holds the one-bar volatility estimates when scaling is needed, otherwise None.
Neither is saved by save() or included in to_dict().
Cross-sectional tables¶
periods¶
One row per timestamp and horizon, horizons in the configured order.
| Column | Contents |
|---|---|
timestamp |
The bar (UTC). |
horizon |
Label length in bars. |
spearman, pearson |
The IC at that timestamp, or NaN. |
n_available |
Assets with a price at the timestamp. |
n_signal |
Assets with a finite score. |
n_pairs |
Assets with a finite score and a label. |
pair_coverage |
n_pairs / n_available; NaN if no asset is available. It does not measure coverage of a historical universe. |
status |
valid, insufficient_assets, constant_signal, constant_returns. |
summary¶
One row per horizon (ascending) and method; every statistic that cannot be computed is absent
(NaN).
| Column | Contents |
|---|---|
total_periods, n_periods, skipped_periods |
Timestamps, those with a valid IC, and those without. |
mean_pair_coverage, mean_n_pairs |
Averages over all timestamps. |
mean, std, positive_fraction |
Over valid timestamps, with equal weights. |
ic_ir |
mean / std, not annualised. |
hac_lags |
Lags used: max(horizon − 1, L), where L is the hac_lags option or, by default, floor(4·(n/100)^(2/9)) for n valid periods. |
hac_se, hac_z, hac_p_value |
Newey–West standard error of the mean, z-statistic and two-sided normal p-value. |
ci95_lower, ci95_upper |
mean ± 1.96·hac_se. |
quantiles¶
Per valid timestamp and horizon: quantile (1 is the lowest score bucket), mean_return of the
bucket (NaN if empty) and n_pairs. mean_return averages the cross-sectional target, so it is a
vol-scaled return when target="vol_scaled". The buckets are retrospective.
Time-series tables¶
assets, monthly and rolling share these window columns:
| Column | Contents |
|---|---|
asset, horizon, method |
The row's keys. |
window_start, window_end |
The half-open window [start, end). |
complete |
Whether the window lies fully inside the evaluation period. |
n_expected |
Bars whose label ends inside the window. |
n_pairs |
Of those, bars with a finite score and label. |
boundary_excluded |
Bars in the window whose label ends outside it. |
pair_coverage |
n_pairs / n_expected. |
long_fraction |
Share of the pairs with a positive score. |
status |
valid, partial_window, short_window, insufficient_pairs, low_coverage, then constant_signal or constant_returns for spearman and pearson, or zero_signal or zero_returns for cosine (first applicable). |
first_pair, last_pair |
First and last bar with a pair. |
ic |
The IC if status is valid, else NaN. |
ic_timing, ic_static |
For cosine: the parts of ic due to the covariance and to the means; they add up to ic. NaN for the other methods. |
assets is the full-period view, with extra columns:
| Column | Contents |
|---|---|
valid_months, skipped_months |
Complete UTC calendar months with and without a valid IC. |
monthly_mean_ic, monthly_std_ic, monthly_ic_ir |
Over valid months; the ratio needs min_months and variation. |
monthly_ir_status |
insufficient_months, constant_ic, preliminary (below twelve months) or available. |
ci95_lower, ci95_upper |
Percentile interval from the circular block bootstrap. |
bootstrap_valid, bootstrap_status, bootstrap_block_bars |
Draws used, outcome, block length. |
signal_autocorr_lag1 |
Correlation of the asset's score with its previous bar's, by the row's method (uncentred for cosine). |
hac_dof, hac_se, hac_t, hac_p_value |
Test that ic is zero (for cosine, that the mean of score × target is zero): the degrees of freedom of the equal-weighted cosine standard error, that standard error under the null, ic / hac_se, and the two-sided p-value from Student's t with hac_dof degrees of freedom. NaN where ic is, and when the bars are fewer than 2 × horizon (hac_dof is then 0). |
summary (time series)¶
One row per horizon and method, weighting eligible assets equally:
| Column | Contents |
|---|---|
n_assets, total_assets |
Assets with a valid full-period IC, and all assets. |
mean, median, q25, q75, positive_fraction |
Over eligible assets. Quartiles describe dispersion, not uncertainty. |
mean_timing, mean_static |
For cosine: the mean of the assets' timing and static parts. NaN for the other methods. |
mean_pair_coverage |
Over all assets. |
universe_valid_months, universe_skipped_months, universe_monthly_mean_ic, universe_monthly_std_ic, universe_monthly_ic_ir, universe_monthly_ir_status |
Average the eligible assets' IC within each month first, then summarise across months. The asset set can change between months. |
ci95_lower, ci95_upper, bootstrap_* |
Interval for the mean asset IC, using one block resampling shared by all assets, and its status: disabled, too_few_blocks, no_eligible_assets, insufficient_draws or valid. |
hac_dof, hac_se, hac_t, hac_p_value |
The same test as in assets, for mean. The assets' contributions are added bar by bar, so assets that move together are not counted as independent. NaN without eligible assets. |
quantiles (time series)¶
Per asset with a valid Spearman IC: asset, horizon, quantile, mean_return (of the
time-series target), n_pairs.
Metrics¶
result.metrics() flattens the summaries to finite numbers named
with analysis cs or ts, horizon in bars, and method spearman, pearson or (ts only)
cosine. For example cs.h5.spearman.mean is the mean cross-sectional rank IC at five bars, and
ts.h5.spearman.mean is the mean eligible asset time-series rank IC. They describe different
samples. The statistics are the columns of the two summary tables, so
cs.h5.spearman.hac_p_value, ts.h5.pearson.n_assets, ts.h5.cosine.mean_static and
ts.h5.cosine.hac_p_value exist.
String-valued columns (such as bootstrap_status) are omitted. Both analyses contribute their
available statistics.
Persistence metrics use a separate naming scheme; see signal persistence.
JSON documents¶
result.to_dict() and study.to_dict() return plain JSON values, for a page, a notebook widget or
another tool to read; the server sends the second as /study.json. Both pass json.dumps(...,
allow_nan=False):
- a missing value is
null, neverNaNand never zero; - timestamps are ISO 8601 strings in UTC, such as
2026-01-01T00:00:00+00:00; - durations are seconds;
- a table is an object of equally long columns,
{"column": [value, ...]}, named and ordered as in the table above, sopandas.DataFrame(table)rebuilds it.
Each document starts with format and format_version (1). Fields may be added within a version;
renaming, removing or reinterpreting one increments it.
Evaluation.to_dict()¶
| Field | Contents |
|---|---|
format |
"alphatonic.evaluation". |
format_version, alphatonic_version |
The layout's version, and the version of alphatonic that wrote the document. |
config |
The configuration, as EvaluationConfig.model_validate reads it back. |
start, end, bar_seconds |
The window [start, end), and the bar spacing (null for irregular grids). |
metrics |
result.metrics(). |
cross_section |
The tables summary, periods and quantiles. |
time_series |
The tables assets, monthly, rolling, summary and quantiles, or null. |
persistence |
The tables periods, quantiles, assets and summary, or null. |
Nothing is recomputed, so the document holds what the tearsheet draws. Forward returns and
volatilities are left out, as by save. The rolling table dominates its size, which grows with
assets, horizons and rolling windows: 30 assets, three horizons and 1,500 daily windows make about
120 MB of JSON.
Study.to_dict(horizon, sort=..., descending=...)¶
| Field | Contents |
|---|---|
format |
"alphatonic.study". |
format_version |
The layout's version. |
name, holdout_start |
The study's name and holdout boundary, or null. |
horizon, horizons |
The horizon of the statistics (default: the smallest), and the horizons of all trials. |
sort, descending |
The order of trials: a board column (default created_at) and its direction. |
context_keys |
The caller's context keys across all trials, in order of appearance. |
families |
Per family: trials and holdout_uses. |
trials |
One object per trial, in the board's order. |
unreadable |
Records that exist but cannot be read, relative to the study's directory. |
Each trial carries the record's id, created_at, signal, family, summary, parameters,
note, sample, start, end, horizons, delay and bar_seconds; the board's assets (a
count), cs_ic, cs_z, cs_p, adjusted_p, ts_ic, ts_t, ts_p, ts_adjusted_p,
ts_cosine, ts_cosine_t, ts_cosine_p, ts_cosine_adjusted_p, ts_timing, trials and
holdout_uses at the document's horizon; the 95% interval of cs_ic as cs_ci_lower and
cs_ci_upper (the Newey–West normal-approximation interval of the evaluation, null where it has no
inference at that horizon); the 95% block-bootstrap intervals of the mean over assets of ts_ic as
ts_ci_lower and ts_ci_upper, and of ts_cosine as ts_cosine_ci_lower and
ts_cosine_ci_upper (null where the bootstrap was off or had too few draws); its context as an object; and files, the paths of its
record and tearsheet relative to the study's directory. tearsheet is null if the trial has
neither a stored tearsheet nor tables to draw one from.
Files written by save¶
| Path | Contents |
|---|---|
evaluation.json |
format_version (5, which adds the hac_* columns to the time-series assets and summary tables; formats 1 to 4 remain readable, without those columns: format 4 gzips the tables, formats 1 to 3 wrote plain CSV files, and format 1 is read with return targets, which it predates), alphatonic_version, the configuration, the window (start, end) and bar_seconds (null for irregular grids). |
ic_by_timestamp.csv.gz, ic_summary.csv.gz, cs_quantiles.csv.gz |
periods, summary, quantiles. |
persistence_periods.csv.gz, persistence_quantiles.csv.gz, persistence_assets.csv.gz, persistence_summary.csv.gz |
Signal persistence tables, written for new evaluations. |
ic_monthly.csv.gz |
The mean and count of spearman and pearson IC per horizon and UTC month. |
ts_assets.csv.gz, ts_monthly.csv.gz, ts_rolling.csv.gz, ts_summary.csv.gz, ts_quantiles.csv.gz |
The time-series tables, always written for new evaluations. |
Every table is gzip-compressed CSV. Formats 1 to 3 wrote the same tables as plain .csv files
(ic_summary.csv, and so on); they load as before.
Files written by a study¶
Each trial of a Study is a directory named by its identifier, such as
20260115-093012-momentum-3f9a2c:
| Path | Contents |
|---|---|
record.json |
The trial's record, written last; a directory without one is an evaluation that did not finish. |
tables/ |
The files written by save, reloadable with Study.evaluation. |
tearsheet.html |
The tearsheet, only if Study.evaluate was called with report=True. Otherwise Study.render_tearsheet and the server draw it from tables/. |
record.json holds these fields. Fields it does not know are ignored when it is read, and a record
without family is counted under its signal.
| Field | Contents |
|---|---|
format_version |
1. |
id, created_at |
Identifier and creation time (UTC). |
signal, family, summary, parameters |
What was evaluated, the name its trials are counted under, a one-line description, the parameters. |
sample, holdout_start |
in_sample or holdout, and the study's holdout boundary at the time. |
start, end, horizons, delay |
The evaluated window [start, end) (UTC), horizons and delay in bars. |
assets, bar_seconds |
The assets of the price table, and the bar spacing (null if irregular). |
data_fingerprint |
SHA-256 fingerprint of the factor and price objects supplied to Study.evaluate, including rows outside the evaluation window. It does not store the data. |
versions |
alphatonic's version and any the caller added. |
context, note |
Facts and a note supplied by the caller. |
metrics |
The flat metrics above. |
Study.board(horizon) has one row per trial, indexed by id: created_at, signal, family,
parameters, sample, start, end, assets (a count), bar, horizon, then cs_ic, cs_z
and cs_p (mean cross-sectional Spearman IC, its Newey–West z-statistic and p-value), adjusted_p
(cs_p times trials, at most 1), ts_ic, ts_t, ts_p and ts_adjusted_p (mean time-series
Spearman IC, its t-statistic, its p-value, and that p-value times trials, at most 1), ts_cosine,
ts_cosine_t, ts_cosine_p and ts_cosine_adjusted_p (the same for the cosine), ts_timing (the
cosine's timing part), trials and holdout_uses (per family), note, and one column per context
key.
Signal persistence¶
Evaluation.persistence holds a PersistenceEvaluation with periods, quantiles, assets and
summary tables. It is None for older saved evaluations. These tables use no future-return labels
and have no horizon column. persistence.lags defaults to (1, 5, 10) price-grid bars;
persistence.min_pairs defaults to 2. Cross-sectional eligibility uses min_assets, and bucket
count uses quantiles. See definitions and interpretation.
| Table | Keys | Values |
|---|---|---|
periods |
timestamp, lag |
rank_autocorrelation; current/prior/common eligible asset counts (n_current, n_previous, n_common); pair_coverage = n_common / max(n_current, n_previous); eligible-universe n_entries, n_exits; status. |
quantiles |
timestamp, lag, quantile |
entry_fraction; n_current, n_previous, n_entries, n_exits; n_universe_entries, n_universe_exits count bucket changes involving absent endpoint scores/prices; status. |
assets |
asset, lag |
normalized_change, change_status; sign_change_fraction, n_sign_changes; zero_transition_fraction, n_zero_transitions; n_pairs, n_expected = max(window_bars − lag, 0), pair_coverage = n_pairs / n_expected; status for transition metrics. |
summary |
scope, lag, quantile, metric |
mean, n_valid, n_total, status. Scope is cs, quantile or ts; quantile 0 means not bucket-specific. CS/bucket candidates are all window timestamps (including boundary exclusions); TS candidates are all assets. |
Statuses: rank persistence uses valid, boundary_excluded, insufficient_assets or
constant_signal; bucket entry fraction uses valid, boundary_excluded, ineligible_endpoint or
empty_quantile. Asset transition metrics use valid or insufficient_pairs; change_status
additionally uses zero_signal. Summary status is valid if at least one statistic contributes,
otherwise unavailable. A zero denominator gives NaN. Counts remain descriptive even when a
comparison is unavailable; consult status before treating counts as transitions. Fractions are
decimals and no metric is annualised.
metrics() includes finite means and counts as
persistence.<scope>.lag<lag>[.q<quantile>].<metric>.<mean|n_valid|n_total>. Formats 3 to 5 save
four persistence tables (persistence_*.csv.gz from format 4); formats 1 and 2 remain readable
without them. The report renders stored tables without recomputing diagnostics.