Main board
6 of 15 ranked models share tier 1, so the order inside that tier is no claim. The board rests on 17 C, 12 A and 16 FE datasets. Main board: one readout per model.
Forgis runs CAFE and developed 5 of the 15 ranked models, marked ◊, with CAFE test scores visible during their development. They get no special colour here. You can take them out of every table with the switch above the board.
◊ Forgis row. Kind tells unranked rows apart: baseline, reference, tabular. Click a column to sort. The glossary near the end explains every term.
Heat maps
One row per model, one column per dataset. Lighter is lower and darker is higher, so the scale also reads in grey and for colour-blind readers. A thick edge marks the best ranked model on a dataset. The rows follow the switches above the board.
Breakdowns
Per dataset
Best and median are over ranked models. A flag says a baseline beats every model on that dataset, or reaches at least the median model. Those datasets stay in on purpose.
Dataset groups
Domain groups are for browsing and carry no weight in any score. Numbers are the main board's. To see the board without a group, use the switches at the top.
Pair verdicts
Row against column. Hover or tap a cell for the difference, its interval and the Holm p. Unresolved does not mean equal.
Tiers
CAFE score
Tier 1 pj_better_10m ◊, pj_better_3p7m ◊, Mantis, pjgood-warp-xview-combo-10M ◊, Chronos-2, LeSaturn-warppool-factorized-7.4M ◊
Tier 2 pj_better_1m ◊, MantisV2
Tier 3 NuTime, Timestral-209M, Timestral-123M, Timestral-5M, Timestral-28M, Toto-Open-Base-1.0
Tier 4 TimesFM-2.5-200M
C, classification
Tier 1 Chronos-2, Timestral-209M, pjgood-warp-xview-combo-10M ◊, Timestral-123M
Tier 2 Timestral-28M, pj_better_10m ◊, pj_better_3p7m ◊, LeSaturn-warppool-factorized-7.4M ◊
Tier 3 Timestral-5M, Mantis, NuTime
Tier 4 pj_better_1m ◊, MantisV2
Tier 5 TimesFM-2.5-200M
Tier 6 Toto-Open-Base-1.0
A, anomaly detection
Tier 1 pj_better_3p7m ◊, pjgood-warp-xview-combo-10M ◊, pj_better_10m ◊, MantisV2, Mantis, NuTime, pj_better_1m ◊, LeSaturn-warppool-factorized-7.4M ◊
Tier 2 Chronos-2, Timestral-209M, Timestral-123M, Timestral-28M, Timestral-5M, Toto-Open-Base-1.0, TimesFM-2.5-200M
FE, forecasting events
Tier 1 Mantis, Toto-Open-Base-1.0, pj_better_10m ◊, LeSaturn-warppool-factorized-7.4M ◊
Tier 2 Chronos-2, pj_better_3p7m ◊
Tier 3 pjgood-warp-xview-combo-10M ◊, pj_better_1m ◊
Tier 4 MantisV2
Tier 5 NuTime, Timestral-5M
Tier 6 Timestral-209M, Timestral-123M, Timestral-28M
Tier 7 TimesFM-2.5-200M
Sensitivities
With and without A
Computed from the per-dataset values on this page. Ranks count the ranked rows only.
Other rule sets and dataset choices
Each row is a whole board under one changed rule, from sens095.json.
Datasets resampled, and the chance of being first
The datasets are drawn again with replacement within each task. Computed by sens095.json (sens095.py), 10,000 draws, each also resampling the test material. This asks how much the order depends on which datasets are in the benchmark, which the board's own interval does not cover.
Noise
Signal-to-noise, this version
Recomputed from this version's draws: the spread of the ranked rows' scores divided by the test-material noise of a pair. The other sources below are not in it.
A null floors per dataset
Each null is normalised as the dataset's score is, so 0.5 is chance. They remove no dataset in 0.9.5: they are shown as unranked rows, and a dataset leaves A only if fewer than 5 of its series keep a prefix of 512 samples. These scores read no model.
12 A datasets are admitted; the rule asks for at least 5. The mean of their best data-free scores is 0.603, and 14 of 15 ranked rows score above it on those datasets; the rule asks for at least half. So A counts in the CAFE score.
Source: data/nulls095_summary.json (from nulls095.json, centre assignment, Eval series).
Rules in plain words
Inputs
Every encoder gets real samples with no padding and no mask. What an encoder does inside, such as padding to its own context or normalising its input, counts as part of the model under test. The models table lists the input rules each row used.
On C, a recording of 512 samples or fewer is stretched to 512 by linear interpolation for every model, which changes the time scale in the same way for every encoder. A longer recording goes to the model whole: a model that accepts variable length gets it up to its maximum context, most recent samples kept, and a model with a fixed input length of 512 gets it block-averaged to 512. On A, the window is 256 real samples ending at the scored moment, never stretched; its score goes to the window's centre step, and a series needs a prefix of at least 512 samples or it leaves the list. On FE, every model gets the last 512 samples before the forecast time. Rows that embed no window, the tabular rows and hand-made references, keep their registered inputs.
Heads
The head is the only thing fitted, and it is the same for every model. C uses L2 logistic regression on one block per channel: a model's native width when it is at most 256 per channel, and wider embeddings projected to 256 per channel by a fixed random projection (at most 8,192 in total), with its penalty chosen by cross-validation inside the training split. A uses no labels: it scores a window by the mean of its 5 smallest Mahalanobis distances to the windows of the series' own prefix. FE fits one ridge regression per lead time on the embedding at the end of the context, with 3 head seeds.
Scores
Each dataset gives one number on the same scale, 0.5 for chance and 1 for perfect. C uses balanced accuracy rescaled for the number of classes, A uses VUS-PR after the prefix rescaled against random scores, and FE uses stratified AUROC averaged over lead times and head seeds. A task score is the plain mean over its datasets. The CAFE score is the mean of C, A and FE.
Readouts
Each model has one readout, the same on C, A and FE. The candidates are m0 (the released embedding), m1, m2, m3 and m6, as far as the model's code exposes them; the pick has the highest validation CAFE score, the mean over the tasks of cross-validation on the training split for C, the Tuning series for A and the inner cross-validation of the ridge head for FE. A candidate within 0.005 of the best loses to an earlier one. No test value is read.
Intervals
An interval is a paired 95% percentile bootstrap of the test material inside each dataset: test samples or splits on C, Eval series on A with near-duplicates resampled together, time blocks on FE. There are 10,000 draws, the same for every model. The datasets are held fixed, so an interval says nothing about how a model would do on other datasets; the resampled table under Sensitivities speaks to that.
Verdicts and tiers
Every pair of ranked models gets one of three verdicts. It is better if the Holm-corrected two-sided bootstrap p is below 0.05. It is practically equal if not resolved and the 95% interval of the difference lies inside plus or minus 0.01. Otherwise it is unresolved, which means the data do not say. A pair of single-seed models less than 0.009 apart is also marked within seed noise. Tier 1 holds the models above the first one resolved below the top model, and the next tier starts there, so a model in a lower tier can still be unresolved against the top one. Rank inside a tier is not a claim.
Admission checks
A dataset must come from an inanimate physical system, its label must be a state, type or identity of that system or an event observed in the record, it needs at least 100 test samples on C, 5 Eval series on A or 1,000 events on FE, and only the largest dataset of a recording campaign stays. A series needs the prefix above. The data-free nulls admit and remove nothing; they are shown beside the models. None of these checks reads a model's score, so datasets where a trivial row beats every encoder stay in and the boards show it.
Conflict of interest
Forgis maintains CAFE and developed 5 of the 15 ranked models. Those models were developed with access to CAFE test scores, and the owner of the benchmark asked for several rule changes while the order was known. Each amendment says what was known when it was written, and the boards under older rules stay published. Some of the latest rules help Forgis rows and some hurt them. Two decisions came after this board was seen. The 08:00Z deadline of 4 November, whose fallback would have made every model's readout m0, was withdrawn because 912 of the 920 jobs, the fallback's own m0 cells among them, finished after it (PREREG09, note of 2026-11-04 ~18:50Z); the m0 board is shown as a sensitivity. The note of 2026-11-04 ~22:05Z retired reporting-only sensitivities and gates that this version's rules make inapplicable, kept A at w = 512 as a sensitivity and scheduled the per-channel Forgis m0; both are pending for 0.9.5.1, as the dataset card lists. Forgis checkpoints that are not public carry a sha256, and the dataset card says which.
What the board does not show
The board does not show performance after fine-tuning, because nothing is fine-tuned. It does not cover datasets outside the list, and its intervals leave out pretraining seed, head training set and readout choice. It says nothing about value forecasting, because FE scores events only. Run time is reported and never scored. CAFE publishes scores and no data.
Known limitations, as written on the dataset card
- Recording length carries label information in two C datasets, in the source data itself: a classifier that sees only the length scores uea/AsphaltPavementTypeCoordinates 0.720 and ucr/PLAID 0.596 (see Datasets admitted and removed); every other C dataset has one length per recording. The variable-length models see the length of recordings above 512 samples and the block-averaged models do not. The board with C resampled to 512 for every model is under Sensitivities.
- On W3 a score that reads only elapsed time reaches 0.9995, so a model that tracks drift scores well there without detecting anything. The A gate compares the ranked rows with 0.603, the mean over the A datasets of each dataset's best data-free score. The board without the five returned A datasets is under Sensitivities.
- On ESA_M1, ESA_M2, HAI, UKWIND and HOT some A Tuning series are other blocks of the same recordings as Eval series, though no series is in both, so the readout choice is less independent of the A test material than the split suggests.
- TiRex-2, a third-party model, was dropped with the 0.9.4.1 board known, where it ranked 8 in tier 1.
- FactoryNet's README warns that recording context partly predicts fault labels, so real fault evaluation should stay within one recording session.
mv06/vorausand the other pooled C datasets mix sessions, and a voraus split tests settings not seen in training. - voraus-AD comes from FactoryNet's copy, not the publisher's files, under CC BY-NC-SA 4.0. Forgis is a company, so CAFE publishes scores only, and a legal reading is still open.
- The weights of the Forgis rows are not public. Each carries a sha256 under Models, so a later release can be checked against it, and until then outsiders cannot rerun those rows.
- At least 66 distinct Saturn-family checkpoints, including the five ranked Forgis rows and two HEPA-lineage precursors, have CAFE or PrecursorBench 1.4 Eval scores on datasets of the 0.9.3.1 board, with records dated from 2026-08-04 to 2026-10-28. Before its submission, each ranked row had at least this many other variants scored: LeSaturn-warppool-factorized-7.4M 43, pjgood-warp-xview-combo-10M 59, pj_better_3p7m and pj_better_10m 64, and pj_better_1m 65. These are lower bounds counted from repository records, because some run trees and lane outputs are unavailable, and they exclude readout variants, controls, evaluation seeds, PrecursorBench v1 runs and development-harness runs.
- MantisV2 is read through the sweep's adapter, and its m0 is the sweep's released-readout rule, not independently confirmed.
- Hill of Towie labels are inferred stops, with rules made without any score. cdip_waves_c was rebuilt because
series.npydiffers from the pinned file; every ranked row was recomputed on the rebuilt file. - The near-duplicate rule clusters SMAP series with long constant stretches although their correlation is near zero, and four UKWIND turbines have identical series that still need checking at the source.
- Mantis and NuTime were pretrained on data that includes UCR and UEA training splits. Whether a Forgis checkpoint was pretrained on FactoryNet data was not established.
- Some unranked reference and tabular rows have no cell on a new C dataset, so they have no C score: CatBoost (Selected) on mv06/aursad095 and mv06/voraus095; XGBoost (Selected) on mv06/voraus095. These tabular runs hit the one-hour cap per dataset and mode (Amendment 3). No ranked row is affected.
- The reference rows raw-window LR, catch22 + LR and majority class on mv06/voraus095, and MiniRocket and MultiROCKET-Hydra on the three new C datasets, were computed on 4 November after the board was seen (x86 CPU,
c_rows06.pyunchanged; MiniRocket had failed earlier on a thread-count setting). They are unranked and change no ranked row. - The m0 candidate of the five Forgis rows pools over channels in this version (320 values for pj_better_10m on the 14-channel mv06/hydraulic), against the C head's one block per channel, and on the Fixed (m0) board the Forgis rows and their control are pooled differently. No Forgis row picked the pooled m0, but the per-channel m0 is a new candidate: on C it equals m2, and on FE it differs from m2, so the readout choice of a Forgis row, and with it the main board, may change in 0.9.5.1. The per-channel m0 and the readout choice redone with it are pending, to be added in 0.9.5.1 (PREREG09 note of 2026-11-04 ~22:05Z items 3 and 4).
- The C head charges for embedding width, not only for uninformative columns (found by Pratik Jawahar of Forgis, in an internal note). Appending 128 columns of pure noise to the m0 embedding of each of six Forgis rows lowered their training-split cross-validated balanced accuracy on C by 0.063 to 0.093. The cause is the head's single shared L2 penalty, chosen from four values (C in 0.01, 0.1, 1, 10): it cannot keep the informative columns while it shrinks the rest. Below the cap of 256 values per channel, models of different native width therefore face different effective penalties. The head (
c_head.py, sha256 0cc2a4bd) is the same in this version and nothing was rescored. The fix is planned for the next version as a head change registered before it is used and chosen without reading test scores. The options are a noise-column acceptance control (k junk columns may move the C score by no more than the fold-seed SD), a finer L2 grid or nested cross-validation, an elastic-net penalty, or a fixed-width PCA fitted on the training split. - Pending, to be added in 0.9.5.1: the A board at w = 512 with each model's readout, a sensitivity (Amendment 7; PREREG09 note of 2026-11-04 ~22:05Z items 2 and 4, and its addition of ~23:05Z). Its cells were queued on 4 November and had not run by the release.
- Retired by the PREREG09 note of 2026-11-04 ~22:05Z, reporting only: the board under the 0.9.3.1 rules, SKAB in its 28-series form, the Selected, Oracle and method-board comparisons, and the 0.9.1 KPI-pool SNR; see Sensitivities.
- Wall times are not comparable across jobs: the C and FE jobs that started from 14:45 UTC on 4 November ran with three workers per GPU (twelve per node) instead of one, with the same code, settings and thread counts per job, so their wall times are inflated against those of the earlier jobs. No score depends on wall time.
How to submit a model
CAFE has no upload form. To propose a model, open a discussion on the dataset page with a public checkpoint pinned to a commit, its licence, code that turns an array of shape (samples, channels, T) into an embedding, with the largest T it accepts, the list of readouts it exposes, and the number of variants you scored on CAFE datasets before submitting, which the protocol publishes with the row. Forgis then runs its cells under the rules above, and the row appears with its marks in a dated version.
Datasets and models
Datasets and licences
CAFE redistributes no data.
Models
Every model is frozen.
Version history
Figures and tables for reports
Every figure is a file in this Space. Use the links under each one. Tables are below in a form you can copy, and each also exists as a CSV file. Version 0.9.5.
Tables
Glossary
- A
- Anomaly detection: find the abnormal stretches in a series. Scored by VUS-PR after the prefix.
- AUROC
- The chance that a randomly picked positive case gets a higher score than a randomly picked negative one. FE uses a stratified version, compared within fold, unit and month.
- balanced accuracy
- Accuracy averaged over classes, so a large class does not dominate. C rescales it so that chance is 0.5.
- baseline
- A row with no learned encoder: ZERO on every task, and current value on FE. Shown on every board, never ranked. The random-init control and the other references are listed beside them.
- best baseline
- On one dataset, the highest-scoring of the baselines the board compares with.
- bootstrap
- Drawing the test material again with replacement to see how far a score moves. CAFE uses 10,000 draws.
- C
- Classification: say what a recording shows. Scored by balanced accuracy.
- CAFE score
- The mean of the task scores that count on this release: C, A and FE.
- cell
- One model's score on one dataset, stored with its bootstrap draws.
- chance-normalised
- Rescaled so that random output gives 0.5 and a perfect result gives 1.
- conflict of interest
- Forgis runs CAFE and also builds some of the models it ranks. Those rows carry the mark ◊.
- embedding
- The vector of numbers a frozen model returns for a window. It is all a head ever sees.
- Eval series
- The series of an A dataset that are scored. Tuning series are kept apart.
- FE
- Forecasting events: say whether an event will happen within the next lead time. Scored by AUROC.
- foundation model
- A network pretrained on many time series and then used for jobs it did not see.
- frozen
- The weights are not changed for CAFE. No fine-tuning on any dataset.
- head
- The small model fitted on the embedding for one task. It is the same for every model.
- Holm
- A step-down correction that keeps the error rate over a whole family of pair tests at the stated level.
- input rules
- What each encoder gets: on C the whole recording (block-averaged to 512 for fixed-length models; stretched to 512 when it is 512 samples or shorter), on A a window of 256 real samples, on FE the last 512 samples. No padding, no mask.
- Kendall tau
- Agreement between two orderings: 1 means the same order, 0 means no relation.
- lead time
- How far ahead an FE head has to warn.
- Mahalanobis distance
- Distance that takes account of the spread and correlation of the normal windows.
- near-duplicate cluster
- Eval series joined by a rule on the data alone and resampled as one block, so that copies do not narrow the interval.
- null
- A score that reads no model: elapsed time rising, elapsed time falling, or a random number per window.
- null floor
- A dataset's highest null. A model at or below it has shown nothing the clock or chance could not.
- One readout
- The one readout a model uses on C, A and FE, picked by the highest validation CAFE score without reading any test value. This is the main board's rule.
- paired
- Every model is scored on the same random draws, so a difference between two models does not depend on which material a draw picked.
- practically equal
- A verdict: not resolved, and the 95% interval of the difference lies inside plus or minus 0.01.
- prefix
- The normal stretch at the start of an A series, used to fit the head.
- readout
- Which internal tensor of a model is taken as its embedding.
- reference
- A hand-made row that embeds nothing, such as max-|z| or current value. Shown, never ranked.
- tabular row
- A gradient-boosted or MLP learner on hand-written statistics of the window. Shown, never ranked.
- tier
- Models in score order. Tier 1 holds the models above the first one resolved below the top model; the next tier starts there and is built the same way, so a model in a lower tier can still be unresolved against the top one.
- Tuning series
- Series of an A dataset kept out of scoring. Only readout selection reads them.
- unresolved
- A verdict: the data do not separate the two models, and the interval is too wide to call them equal.
- VUS-PR
- Volume under the precision-recall surface, the metric for A.
- within seed noise
- A pair of single-seed models closer than 0.009 on the CAFE score or on a task board, 1.96 times the total SD of a single-seed difference in the 0.9.2 noise budget (0.0044, of which pretraining seed is 0.0018), a normal approximation.
- ZERO
- A fixed random projection of the raw window with no learning. The no-encoder reference.
- †
- A mean over fewer datasets than the view has, because some cells do not exist in this readout mode.
- ◊
- A model developed by Forgis with access to CAFE Eval scores.