0.9.7.20.9.70.9.60.9.5

Versions compared · current version 0.9.7.2

CAFE

CAFE tests time-series foundation models with their weights frozen, on sensor recordings from machines, structures and the natural environment. Each model reads a recording on C (whole when it is longer than 512 samples, up to the model's context; block-averaged to 512 for fixed-length models; stretched to 512 when shorter), the last 512 values on FE and windows of 256 values on A, and returns a vector, and the same small head reads that vector for every model, to classify a recording (C), find abnormal stretches (A) or warn of an event before it happens (FE). A score of 0.5 is chance and 1 is perfect, and the CAFE score is the mean of C, A and FE.

Comparison across versions since 0.9.4

0.9.4 (0.9.4.1)0.9.50.9.60.9.70.9.7.212345678910111213141516pj_better_10m ◊: 0.9.4 (0.9.4.1) rank 6, 0.9.5 rank 1, 0.9.6 rank 1, 0.9.7 rank 1, 0.9.7.2 rank 1pj_better_10m ◊Mantis: 0.9.4 (0.9.4.1) rank 4, 0.9.5 rank 3, 0.9.6 rank 3, 0.9.7 rank 4, 0.9.7.2 rank 2Mantispj_better_3p7m ◊: 0.9.4 (0.9.4.1) rank 7, 0.9.5 rank 2, 0.9.6 rank 2, 0.9.7 rank 2, 0.9.7.2 rank 3pj_better_3p7m ◊pjgood-warp-xview-combo-10M ◊: 0.9.4 (0.9.4.1) rank 5, 0.9.5 rank 4, 0.9.6 rank 4, 0.9.7 rank 3, 0.9.7.2 rank 4pjgood-warp-xview-combo-10M ◊Chronos-2: 0.9.4 (0.9.4.1) rank 12, 0.9.5 rank 5, 0.9.6 rank 5, 0.9.7 rank 5, 0.9.7.2 rank 5Chronos-2LeSaturn-warppool-factorized-7.4M ◊: 0.9.4 (0.9.4.1) rank 10, 0.9.5 rank 6, 0.9.6 rank 6, 0.9.7 rank 6, 0.9.7.2 rank 6LeSaturn-warppool-factorized-7.4M ◊pj_better_1m ◊: 0.9.4 (0.9.4.1) rank 9, 0.9.5 rank 7, 0.9.6 rank 7, 0.9.7 rank 7, 0.9.7.2 rank 7pj_better_1m ◊MantisV2: 0.9.4 (0.9.4.1) rank 1, 0.9.5 rank 8, 0.9.6 rank 8, 0.9.7 rank 8, 0.9.7.2 rank 8MantisV2NuTime: 0.9.4 (0.9.4.1) rank 14, 0.9.5 rank 9, 0.9.6 rank 9, 0.9.7 rank 9, 0.9.7.2 rank 9NuTimeTimestral-209M: 0.9.4 (0.9.4.1) rank 3, 0.9.5 rank 10, 0.9.6 rank 10, 0.9.7 rank 10, 0.9.7.2 rank 10Timestral-209MTimestral-123M: 0.9.4 (0.9.4.1) rank 2, 0.9.5 rank 11, 0.9.6 rank 11, 0.9.7 rank 11, 0.9.7.2 rank 11Timestral-123MTimestral-5M: 0.9.4 (0.9.4.1) rank 13, 0.9.5 rank 12, 0.9.6 rank 14, 0.9.7 rank 14, 0.9.7.2 rank 12Timestral-5MTimestral-28M: 0.9.4 (0.9.4.1) rank 11, 0.9.5 rank 13, 0.9.6 rank 13, 0.9.7 rank 13, 0.9.7.2 rank 13Timestral-28MToto-Open-Base-1.0: 0.9.4 (0.9.4.1) rank 15, 0.9.5 rank 14, 0.9.6 rank 12, 0.9.7 rank 12, 0.9.7.2 rank 14Toto-Open-Base-1.0TimesFM-2.5-200M: 0.9.4 (0.9.4.1) rank 16, 0.9.5 rank 15, 0.9.6 rank 15, 0.9.7 rank 15, 0.9.7.2 rank 15TimesFM-2.5-200MTiRex-2: 0.9.4 (0.9.4.1) rank 8TiRex-2 (not ranked later)rank on the CAFE score
Rank of each ranked model on the CAFE score in each version. A diamond is a ◊ row. Ranks inside tier 1 are no claim; the tables give the tiers.

The versions differ in datasets, heads and, for 0.9.4, in the ranked rows (TiRex-2 is ranked in 0.9.4 only), so scores of different versions are means over different material; ranks, tiers and resolved pairs are what can be compared. 0.9.4 is the released 0.9.4 board, unchanged in 0.9.4.1 (board094_frozen.json).

Datasets and pairs resolved. Main board: datasets fixed, test material resampled. Datasets resampled: the measure of the 1.0 test.

versionC datasetsA datasetsFE datasetsranked modelstasks in CAFECAFE pairs resolved, mainofCAFE pairs resolved, datasets resampledof
0.9.4 (0.9.4.1)1671616C, A, FE39 (1 within seed noise)12016120
0.9.517121615C, A, FE73 (1 within seed noise)10549105
0.9.625141715C, A, FE73 (1 within seed noise)10551105
0.9.724101515C, A, FE85 (10 within seed noise)10561105
0.9.7.224131515C, A, FE90 (15 within seed noise)10562105

Per task. Pairs resolved on the task board, main / datasets resampled, and signal-to-noise (mean absolute gap between two ranked rows over the median bootstrap SD of that gap; separability096.py). A dash: no such file for that version.

versionC pairsA pairsFE pairsC signal-to-noiseA signal-to-noiseFE signal-to-noiseCAFE signal-to-noise
0.9.4 (0.9.4.1)73 / –24 / –69 / –––––
0.9.570 / 1518 / 487 / 678.392.4815.107.32
0.9.674 / 2137 / 687 / 668.983.0014.937.89
0.9.780 / 2560 / 987 / 6710.064.0416.2610.88
0.9.7.280 / 2563 / 1887 / 6710.064.8116.2612.20

No separability file exists for 0.9.4.

First place. The first row of each board; inside tier 1 the order is no claim.

versionCAFECAFEmodels in CAFE tier 1
0.9.4 (0.9.4.1)MantisV2Timestral-209MTimestral-123Mpj_better_1m ◊13
0.9.5pj_better_10m ◊Chronos-2pj_better_3p7m ◊Mantis6
0.9.6pj_better_10m ◊Chronos-2pj_better_3p7m ◊Mantis5
0.9.7pj_better_10m ◊Chronos-2pj_better_10m ◊Mantis1
0.9.7.2pj_better_10m ◊Chronos-2MantisMantis2

Rank and tier of each ranked model on the CAFE score, rank (tier):

model0.9.4 (0.9.4.1)0.9.50.9.60.9.70.9.7.2
pj_better_10m ◊6 (1)1 (1)1 (1)1 (1)1 (1)
Mantis4 (1)3 (1)3 (1)4 (2)2 (1)
pj_better_3p7m ◊7 (1)2 (1)2 (1)2 (2)3 (2)
pjgood-warp-xview-combo-10M ◊5 (1)4 (1)4 (1)3 (2)4 (2)
Chronos-212 (1)5 (1)5 (1)5 (2)5 (2)
LeSaturn-warppool-factorized-7.4M ◊10 (1)6 (1)6 (2)6 (3)6 (3)
pj_better_1m ◊9 (1)7 (2)7 (2)7 (3)7 (4)
MantisV21 (1)8 (2)8 (2)8 (4)8 (4)
NuTime14 (2)9 (3)9 (3)9 (5)9 (5)
Timestral-209M3 (1)10 (3)10 (3)10 (5)10 (6)
Timestral-123M2 (1)11 (3)11 (3)11 (6)11 (6)
Timestral-5M13 (1)12 (3)14 (3)14 (6)12 (7)
Timestral-28M11 (1)13 (3)13 (3)13 (6)13 (7)
Toto-Open-Base-1.015 (3)14 (3)12 (3)12 (6)14 (7)
TimesFM-2.5-200M16 (3)15 (4)15 (4)15 (7)15 (8)
TiRex-28 (1)––––

What changed, by the amendments of each version and the datasets added (+) or removed (-) against the version before:

versionrulesdatasets
0.9.4 (0.9.4.1)Amendment 5first version compared
0.9.5Amendments 6, 7, 8, 9 and 10C: +2 (aursad095, voraus095) -1 (voraus); A: +5 (ESA_M1, CARE_A, MSL, W3, SKAB)
0.9.6Amendments 11 and 12C: +8 (chillan5, haptic_texture8, openrainer_m, poseit_ft, px4_airframe3, soredd_abc, swiss_load10, ved_powertrain3); A: +2 (PREDIST, NOBOOM); FE: +1 (ngafid_hastall)
0.9.7Amendment 13C: +2 (bdg2_meter6, robo6_rb6) -3 (3w, bosch, soredd_abc); A: +5 (ESA, NASA, UAVSEAD, UR5E, KUKA) -9 (ESA_M1, ESA_M2, CARE_A, SMAP, MSL, CARE_C, RIGS, W3, SKAB); FE: +1 (heatpump_defrost) -3 (epa_aqs_pm25, substorm_omni, ngafid_hastall)
0.9.7.2Amendment 15A: +3 (UTAQ, RCFPOWER, REASSEMBLE)

Each version's own changes are in the change log.