CAFE

Main board

5 of 15 ranked models share tier 1, so the order inside that tier is no claim. The board rests on 25 C, 14 A and 17 FE datasets. Main board: one readout per model.

Forgis runs CAFE and developed 5 of the 15 ranked models, marked ◊, with CAFE test scores visible during their development. They get no special colour here. You can take them out of every table with the switch above the board.

◊ Forgis row. Kind tells unranked rows apart: baseline, reference, tabular. Click a column to sort. The glossary near the end explains every term.

0.9.5 and 0.9.6

0.9.6 adds 8 C datasets, 2 A datasets and 1 FE dataset to 0.9.5, and fits the C head over a wider grid of penalties (head H1). Every model keeps its 0.9.5 readout. Scores of the two versions are means over different datasets, so a change in score is not a change in the model; ranks and tiers are what can be compared. The full 0.9.5 page is kept here.

Before any 0.9.6 score was read, we fixed one test for which release becomes CAFE 1.0 (PREREG09 Amendment 11 item 7): count the pairs of the 15 ranked models that stay resolved when the datasets themselves are resampled (10,000 draws, Holm correction). 0.9.5 resolves 49 of 105 pairs and 0.9.6 resolves 51. The mark was 54: 0.9.6 does not reach it, and 0.9.5 stays the CAFE 1.0 candidate. No pair of models is called better in opposite directions by the two versions.

Pair verdicts on the main board, out of 105 pairs:

Heat maps

One row per model, one column per dataset. Lighter is lower and darker is higher, so the scale also reads in grey and for colour-blind readers. A thick edge marks the best ranked model on a dataset. The rows follow the switches above the board.

Breakdowns

Per dataset

Best and median are over ranked models. A flag says a baseline beats every model on that dataset, or reaches at least the median model. Those datasets stay in on purpose.

Dataset groups

Domain groups are for browsing and carry no weight in any score. Numbers are the main board's. To see the board without a group, use the switches at the top.

Pair verdicts

Row against column. Hover or tap a cell for the difference, its interval and the Holm p. Unresolved does not mean equal.

Tiers

Sensitivities

Noise

A null floors per dataset

Each null is normalised as the dataset's score is, so 0.5 is chance. They remove no dataset in 0.9.6: they are shown as unranked rows, and a dataset leaves A only if fewer than 5 of its series keep a prefix of 512 samples. These scores read no model.

14 A datasets are admitted; the rule asks for at least 5. The mean of their best data-free scores is 0.594, and 15 of 15 ranked rows score above it on those datasets; the rule asks for at least half. So A counts in the CAFE score.

Source: results/nulls096_summary.json (nulls095_summary.json plus PREDIST and NOBOOM from nulls096.json, centre assignment, Eval series).

Rules in plain words

Inputs

Every encoder gets real samples with no padding and no mask. What an encoder does inside, such as padding to its own context or normalising its input, counts as part of the model under test. The models table lists the input rules each row used.

On C, a recording of 512 samples or fewer is stretched to 512 by linear interpolation for every model, which changes the time scale in the same way for every encoder. A longer recording goes to the model whole: a model that accepts variable length gets it up to its maximum context, most recent samples kept, and a model with a fixed input length of 512 gets it block-averaged to 512. On A, the window is 256 real samples ending at the scored moment, never stretched; its score goes to the window's centre step, and a series needs a prefix of at least 512 samples or it leaves the list. On FE, every model gets the last 512 samples before the forecast time. Rows that embed no window, the tabular rows and hand-made references, keep their registered inputs.

Heads

The head is the only thing fitted, and it is the same for every model. C uses L2 logistic regression on one block per channel: a model's native width when it is at most 256 per channel, and wider embeddings projected to 256 per channel by a fixed random projection (at most 8,192 in total), with its penalty chosen by cross-validation inside the training split. A uses no labels: it scores a window by the mean of its 5 smallest Mahalanobis distances to the windows of the series' own prefix. FE fits one ridge regression per lead time on the embedding at the end of the context, with 3 head seeds.

Scores

Each dataset gives one number on the same scale, 0.5 for chance and 1 for perfect. C uses balanced accuracy rescaled for the number of classes, A uses VUS-PR after the prefix rescaled against random scores, and FE uses stratified AUROC averaged over lead times and head seeds. A task score is the plain mean over its datasets. The CAFE score is the mean of C, A and FE.

Readouts

Each model has one readout, the same on C, A and FE. The candidates are m0 (the released embedding), m1, m2, m3 and m6, as far as the model's code exposes them; the pick has the highest validation CAFE score, the mean over the tasks of cross-validation on the training split for C, the Tuning series for A and the inner cross-validation of the ridge head for FE. A candidate within 0.005 of the best loses to an earlier one. No test value is read.

Intervals

An interval is a paired 95% percentile bootstrap of the test material inside each dataset: test samples or splits on C, Eval series on A with near-duplicates resampled together, time blocks on FE. There are 10,000 draws, the same for every model. The datasets are held fixed, so an interval says nothing about how a model would do on other datasets; the resampled table under Sensitivities speaks to that.

Verdicts and tiers

Every pair of ranked models gets one of three verdicts. It is better if the Holm-corrected two-sided bootstrap p is below 0.05. It is practically equal if not resolved and the 95% interval of the difference lies inside plus or minus 0.01. Otherwise it is unresolved, which means the data do not say. A pair of single-seed models less than 0.009 apart is also marked within seed noise. Tier 1 holds the models above the first one resolved below the top model, and the next tier starts there, so a model in a lower tier can still be unresolved against the top one. Rank inside a tier is not a claim.

Admission checks

A dataset must come from an inanimate physical system, its label must be a state, type or identity of that system or an event observed in the record, it needs at least 100 test samples on C, 5 Eval series on A or 1,000 events on FE, and only the largest dataset of a recording campaign stays. A series needs the prefix above. The data-free nulls admit and remove nothing; they are shown beside the models. None of these checks reads a model's score, so datasets where a trivial row beats every encoder stay in and the boards show it.

Conflict of interest

Forgis maintains CAFE and developed 5 of the 15 ranked models. Those models were developed with access to CAFE test scores, and the owner of the benchmark asked for several rule changes while the order was known. Each amendment says what was known when it was written, and the boards under older rules stay published. Some of the latest rules help Forgis rows and some hurt them. Two decisions came after this board was seen. The 08:00Z deadline of 4 November, whose fallback would have made every model's readout m0, was withdrawn because 912 of the 920 jobs, the fallback's own m0 cells among them, finished after it (PREREG09, note of 2026-11-04 ~18:50Z); the m0 board is shown as a sensitivity. The note of 2026-11-04 ~22:05Z retired reporting-only sensitivities and gates that this version's rules make inapplicable, kept A at w = 512 as a sensitivity and scheduled the per-channel Forgis m0; both are pending for 0.9.5.1, as the dataset card lists. Forgis checkpoints that are not public carry a sha256, and the dataset card says which.

What the board does not show

The board does not show performance after fine-tuning, because nothing is fine-tuned. It does not cover datasets outside the list, and its intervals leave out pretraining seed, head training set and readout choice. It says nothing about value forecasting, because FE scores events only. Run time is reported and never scored. CAFE publishes scores and no data.

How to submit a model

CAFE has no upload form. To propose a model, open a discussion on the dataset page with a public checkpoint pinned to a commit, its licence, code that turns an array of shape (samples, channels, T) into an embedding, with the largest T it accepts, the list of readouts it exposes, and the number of variants you scored on CAFE datasets before submitting, which the protocol publishes with the row. Forgis then runs its cells under the rules above, and the row appears with its marks in a dated version.

Datasets and models

Version history

Protocol · Changelog · leaderboard.json

Figures and tables for reports

Every figure is a file in this Space. Use the links under each one. Tables are below in a form you can copy, and each also exists as a CSV file. Version 0.9.6.

Not offered here because they predate this board: baselines.png, cost.png, elo_cost.png, gaps.png, leaderboard.png, method_board.png, money_plot.pdf, money_plot.png, variance_breakdown.png, variance_pie.png.

Tables

Glossary

A

Anomaly detection: find the abnormal stretches in a series. Scored by VUS-PR after the prefix.

AUROC

The chance that a randomly picked positive case gets a higher score than a randomly picked negative one. FE uses a stratified version, compared within fold, unit and month.

balanced accuracy

Accuracy averaged over classes, so a large class does not dominate. C rescales it so that chance is 0.5.

baseline

A row with no learned encoder: ZERO on every task, and current value on FE. Shown on every board, never ranked. The random-init control and the other references are listed beside them.

best baseline

On one dataset, the highest-scoring of the baselines the board compares with.

bootstrap

Drawing the test material again with replacement to see how far a score moves. CAFE uses 10,000 draws.

C

Classification: say what a recording shows. Scored by balanced accuracy.

CAFE score

The mean of the task scores that count on this release: C, A and FE.

cell

One model's score on one dataset, stored with its bootstrap draws.

chance-normalised

Rescaled so that random output gives 0.5 and a perfect result gives 1.

conflict of interest

Forgis runs CAFE and also builds some of the models it ranks. Those rows carry the mark ◊.

embedding

The vector of numbers a frozen model returns for a window. It is all a head ever sees.

Eval series

The series of an A dataset that are scored. Tuning series are kept apart.

FE

Forecasting events: say whether an event will happen within the next lead time. Scored by AUROC.

foundation model

A network pretrained on many time series and then used for jobs it did not see.

frozen

The weights are not changed for CAFE. No fine-tuning on any dataset.

head

The small model fitted on the embedding for one task. It is the same for every model.

Holm

A step-down correction that keeps the error rate over a whole family of pair tests at the stated level.

input rules

What each encoder gets: on C the whole recording (block-averaged to 512 for fixed-length models; stretched to 512 when it is 512 samples or shorter), on A a window of 256 real samples, on FE the last 512 samples. No padding, no mask.

Kendall tau

Agreement between two orderings: 1 means the same order, 0 means no relation.

lead time

How far ahead an FE head has to warn.

Mahalanobis distance

Distance that takes account of the spread and correlation of the normal windows.

near-duplicate cluster

Eval series joined by a rule on the data alone and resampled as one block, so that copies do not narrow the interval.

null

A score that reads no model: elapsed time rising, elapsed time falling, or a random number per window.

null floor

A dataset's highest null. A model at or below it has shown nothing the clock or chance could not.

One readout

The one readout a model uses on C, A and FE, picked by the highest validation CAFE score without reading any test value. This is the main board's rule.

paired

Every model is scored on the same random draws, so a difference between two models does not depend on which material a draw picked.

practically equal

A verdict: not resolved, and the 95% interval of the difference lies inside plus or minus 0.01.

prefix

The normal stretch at the start of an A series, used to fit the head.

readout

Which internal tensor of a model is taken as its embedding.

reference

A hand-made row that embeds nothing, such as max-|z| or current value. Shown, never ranked.

tabular row

A gradient-boosted or MLP learner on hand-written statistics of the window. Shown, never ranked.

tier

Models in score order. Tier 1 holds the models above the first one resolved below the top model; the next tier starts there and is built the same way, so a model in a lower tier can still be unresolved against the top one.

Tuning series

Series of an A dataset kept out of scoring. Only readout selection reads them.

unresolved

A verdict: the data do not separate the two models, and the interval is too wide to call them equal.

VUS-PR

Volume under the precision-recall surface, the metric for A.

within seed noise

A pair of single-seed models closer than 0.009 on the CAFE score or on a task board, 1.96 times the total SD of a single-seed difference in the 0.9.2 noise budget (0.0044, of which pretraining seed is 0.0018), a normal approximation.

ZERO

A fixed random projection of the raw window with no learning. The no-encoder reference.

†

A mean over fewer datasets than the view has, because some cells do not exist in this readout mode.

◊

A model developed by Forgis with access to CAFE Eval scores.