File size: 2,184 Bytes
569aa3c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | task,protocol,chance,o2_sft,o2_base,gpt2_medium,pythia_410m,smollm2_360m,qwen2_5_0_5b,o2_sft_se,o2_base_se,gpt2_medium_se,pythia_410m_se,smollm2_360m_se,qwen2_5_0_5b_se
ARC-Challenge,25-shot,25,27.3333,30.0000,28.6667,26.0000,35.3333,31.6667,2.5774,2.6502,2.6152,2.5367,2.7644,2.6902
HellaSwag,10-shot,25,42.0000,39.3333,47.6667,47.6667,56.6667,50.6667,2.8543,2.8250,2.8884,2.8884,2.8658,2.8913
MMLU,5-shot,25,24.9825,25.0526,26.2456,26.0351,24.8421,49.5439,1.1404,1.1438,1.1502,1.1542,1.1477,1.2987
TruthfulQA MC2,0-shot,–,40.2831,41.3715,39.7467,40.6944,34.7438,39.9470,2.4146,2.4190,2.3740,2.3721,2.2122,2.3132
Winogrande,5-shot,50,54.0000,53.0000,54.0000,51.3333,55.3333,58.3333,2.8823,2.8864,2.8823,2.8905,2.8751,2.8511
GSM8K (strict),5-shot,0,4.0000,8.6667,1.0000,1.0000,4.3333,34.6667,1.1333,1.6271,0.5754,0.5754,1.1775,2.7522
GSM8K (flexible),5-shot,0,6.3333,8.6667,1.3333,1.0000,5.0000,35.0000,1.4086,1.6271,0.6633,0.5754,1.2604,2.7584
ARC-Easy,0-shot,25,48.3333,51.6667,43.3333,45.0000,66.3333,57.0000,2.8900,2.8900,2.8658,2.8771,2.7329,2.8631
PIQA,0-shot,50,67.6667,68.3333,68.3333,70.0000,73.0000,71.6667,2.7051,2.6902,2.6902,2.6502,2.5675,2.6060
SciQ,0-shot,25,89.6667,91.3333,68.0000,75.3333,87.0000,92.3333,1.7604,1.6271,2.6977,2.4929,1.9449,1.5387
OpenBookQA,0-shot,25,32.6667,33.0000,31.3333,32.6667,39.6667,36.3333,2.7123,2.7193,2.6825,2.7123,2.8291,2.7815
BoolQ,0-shot,62.2 (majority),47.0000,48.3333,62.6667,58.0000,61.0000,63.0000,2.8864,2.8900,2.7972,2.8543,2.8207,2.7921
LAMBADA accuracy,0-shot,–,26.3333,25.3333,46.6667,52.3333,52.6667,53.6667,2.5471,2.5152,2.8851,2.8884,2.8875,2.8838
LAMBADA perplexity ↓,0-shot,–,97.1431,106.1273,18.2225,10.9031,10.4561,10.7190,20.2186,20.7252,2.5501,1.3911,1.3151,1.4841
WikiText word perplexity ↓,0-shot,–,39.4240,36.9033,26.7578,20.1072,nan,18.4298,,,,,,
COPA,0-shot,50,65.0000,65.0000,68.0000,68.0000,78.0000,74.0000,4.7937,4.7937,4.6883,4.6883,4.1633,4.4084
RACE,0-shot,25,27.6667,27.6667,30.0000,32.3333,32.0000,33.0000,2.5871,2.5871,2.6502,2.7051,2.6977,2.7193
LogiQA,0-shot,25,,,,,,,,,,,,
CommonsenseQA,0-shot,20,22.6667,21.6667,19.0000,20.3333,17.0000,52.6667,2.4213,2.3825,2.2687,2.3276,2.1723,2.8875
|