[
  {
    "title": "All scenarios",
    "header": [
      {
        "value": "Group",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "Description",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "Adaptation method",
        "description": "Adaptation strategy (e.g., generation)",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "# instances",
        "description": "Number of instances evaluated on",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "# references",
        "description": "Number of references provided per instance",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "# prompt tokens",
        "description": "Total number of prompt tokens",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "# completion tokens",
        "description": "Total number of completion tokens",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "# models",
        "description": "Number of models we're evaluating",
        "markdown": false,
        "metadata": {}
      }
    ],
    "rows": [
      [
        {
          "value": "Core scenarios",
          "href": "?group=core_scenarios",
          "markdown": false
        },
        {
          "value": "The scenarios where we evaluate all the models.",
          "markdown": true
        },
        {
          "value": "generation, multiple_choice_joint",
          "markdown": false
        },
        {
          "value": 77.49230769230769,
          "description": "min=11, mean=77.492, max=114, sum=10074 (130)",
          "markdown": false
        },
        {
          "value": 3.2418461538461543,
          "description": "min=2, mean=3.242, max=4, sum=1264.32 (390)",
          "markdown": false
        },
        {
          "value": 458024.04421788757,
          "description": "min=133.92, mean=1174.421, max=3673.667, sum=458024.044 (390)",
          "markdown": false
        },
        {
          "value": 1319.410489510489,
          "description": "min=0, mean=3.383, max=80.72, sum=1319.41 (390)",
          "markdown": false
        },
        {
          "value": 16,
          "markdown": false
        }
      ]
    ],
    "links": []
  },
  {
    "title": "Scenarios",
    "header": [
      {
        "value": "Group",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "Description",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "Adaptation method",
        "description": "Adaptation strategy (e.g., generation)",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "# instances",
        "description": "Number of instances evaluated on",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "# references",
        "description": "Number of references provided per instance",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "# prompt tokens",
        "description": "Total number of prompt tokens",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "# completion tokens",
        "description": "Total number of completion tokens",
        "markdown": false,
        "metadata": {}
      },
      {
        "value": "# models",
        "description": "Number of models we're evaluating",
        "markdown": false,
        "metadata": {}
      }
    ],
    "rows": [
      [
        {
          "value": "NarrativeQA",
          "href": "?group=narrative_qa",
          "markdown": false
        },
        {
          "value": "The NarrativeQA benchmark for reading comprehension over narratives [(Ko\u010disk\u00fd et al., 2017)](https://aclanthology.org/Q18-1023/).",
          "markdown": true
        },
        {
          "value": "generation",
          "markdown": false
        },
        {
          "value": 25.0,
          "description": "min=11, mean=25, max=39, sum=650 (26)",
          "markdown": false
        },
        {
          "value": 2.0,
          "description": "min=2, mean=2, max=2, sum=156 (78)",
          "markdown": false
        },
        {
          "value": 265151.0979020979,
          "description": "min=1717.667, mean=3399.373, max=3673.667, sum=265151.098 (78)",
          "markdown": false
        },
        {
          "value": 721.5104895104893,
          "description": "min=0, mean=9.25, max=60.364, sum=721.51 (78)",
          "markdown": false
        },
        {
          "value": 13,
          "markdown": false
        }
      ],
      [
        {
          "value": "NaturalQuestions (closed-book)",
          "href": "?group=natural_qa_closedbook",
          "markdown": false
        },
        {
          "value": "The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.",
          "markdown": true
        },
        {
          "value": "generation",
          "markdown": false
        },
        {
          "value": 50.0,
          "description": "min=50, mean=50, max=50, sum=650 (13)",
          "markdown": false
        },
        {
          "value": 2.0600000000000014,
          "description": "min=2.06, mean=2.06, max=2.06, sum=80.34 (39)",
          "markdown": false
        },
        {
          "value": 5803.620000000002,
          "description": "min=133.92, mean=148.811, max=165.78, sum=5803.62 (39)",
          "markdown": false
        },
        {
          "value": 329.8799999999999,
          "description": "min=0, mean=8.458, max=80.72, sum=329.88 (39)",
          "markdown": false
        },
        {
          "value": 13,
          "markdown": false
        }
      ],
      [
        {
          "value": "NaturalQuestions (open-book)",
          "href": "?group=natural_qa_openbook_longans",
          "markdown": false
        },
        {
          "value": "The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.",
          "markdown": true
        },
        {
          "value": "generation",
          "markdown": false
        },
        {
          "value": 50.0,
          "description": "min=50, mean=50, max=50, sum=550 (11)",
          "markdown": false
        },
        {
          "value": 2.060000000000001,
          "description": "min=2.06, mean=2.06, max=2.06, sum=67.98 (33)",
          "markdown": false
        },
        {
          "value": 68683.38,
          "description": "min=1523.78, mean=2081.315, max=2406.28, sum=68683.38 (33)",
          "markdown": false
        },
        {
          "value": 73.02000000000001,
          "description": "min=0, mean=2.213, max=9.16, sum=73.02 (33)",
          "markdown": false
        },
        {
          "value": 11,
          "markdown": false
        }
      ],
      [
        {
          "value": "OpenbookQA",
          "href": "?group=openbookqa",
          "markdown": false
        },
        {
          "value": "The OpenbookQA benchmark for commonsense-intensive open book question answering [(Mihaylov et al., 2018)](https://aclanthology.org/D18-1260/).",
          "markdown": true
        },
        {
          "value": "",
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "value": 0,
          "markdown": false
        }
      ],
      [
        {
          "value": "MMLU (Massive Multitask Language Understanding)",
          "href": "?group=mmlu",
          "markdown": false
        },
        {
          "value": "The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://arxiv.org/pdf/2009.03300.pdf).",
          "markdown": true
        },
        {
          "value": "multiple_choice_joint",
          "markdown": false
        },
        {
          "value": 102.8,
          "description": "min=100, mean=102.8, max=114, sum=8224 (80)",
          "markdown": false
        },
        {
          "value": 4.0,
          "description": "min=4, mean=4, max=4, sum=960 (240)",
          "markdown": false
        },
        {
          "value": 118385.94631578946,
          "description": "min=373.26, mean=493.275, max=699.175, sum=118385.946 (240)",
          "markdown": false
        },
        {
          "value": 195.0,
          "description": "min=0, mean=0.812, max=1, sum=195 (240)",
          "markdown": false
        },
        {
          "value": 16,
          "markdown": false
        }
      ],
      [
        {
          "value": "GSM8K (Grade School Math)",
          "href": "?group=gsm",
          "markdown": false
        },
        {
          "value": "The grade school math word problems dataset (GSM8K) for testing mathematical reasoning on grade-school math problems [(Cobbe et al., 2021)](https://arxiv.org/pdf/2110.14168.pdf).",
          "markdown": true
        },
        {
          "value": "",
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "value": 0,
          "markdown": false
        }
      ],
      [
        {
          "value": "MATH",
          "href": "?group=math_chain_of_thought",
          "markdown": false
        },
        {
          "value": "The MATH benchmark for measuring mathematical problem solving on competition math problems with chain-of-thought style reasoning [(Hendrycks et al., 2021)](https://arxiv.org/pdf/2103.03874.pdf).",
          "markdown": true
        },
        {
          "value": "",
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "value": 0,
          "markdown": false
        }
      ],
      [
        {
          "value": "LegalBench",
          "href": "?group=legalbench",
          "markdown": false
        },
        {
          "value": "LegalBench is a large collaboratively constructed benchmark of legal reasoning tasks [(Guha et al, 2023)](https://arxiv.org/pdf/2308.11462.pdf).",
          "markdown": true
        },
        {
          "value": "",
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "value": 0,
          "markdown": false
        }
      ],
      [
        {
          "value": "MedQA",
          "href": "?group=med_qa",
          "markdown": false
        },
        {
          "value": "MedQA is an open domain question answering dataset composed of questions from professional medical board exams ([Jin et al. 2020](https://arxiv.org/pdf/2009.13081.pdf)).",
          "markdown": true
        },
        {
          "value": "",
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "value": 0,
          "markdown": false
        }
      ],
      [
        {
          "value": "WMT 2014",
          "href": "?group=wmt_14",
          "markdown": false
        },
        {
          "value": "WMT 2014 is a collection of machine translation datasets [(website)](https://www.statmt.org/wmt14/index.html).",
          "markdown": true
        },
        {
          "value": "",
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "markdown": false
        },
        {
          "value": 0,
          "markdown": false
        }
      ]
    ],
    "links": []
  }
]