EQ-Bench
Emotional Intelligence Benchmark for LLMs
Github | Paper | | Twitter | Leaderboard
About
📩How to Submit
At this time we only accept submissions of open weight models that are available to everyone via HuggingFace.
To submit, get in touch by email or twitter with:
- A link to your model on huggingface
- Optimal prompt format & generation config
- The EQ-Bench score that you got for your model
We will then verify the result on our end and add to the leaderboard. This project is self funded so please respect that we don't have unlimited compute!
💗EQ-Bench
EQ-Bench is a benchmark for language models designed to assess emotional intelligence.
Why emotional intelligence? One reason is that it represents a subset of abilities that are important for the user experience, and which isn't explicitly tested by other benchmarks. Another reason is that it's not trivial to improve scores by fine tuning for the benchmark, which makes it harder to "game" the leaderboard.
EQ-Bench is a little different from traditional psychometric tests. It uses a specific question format, in which the subject has to read a dialogue then rate the intensity of possible emotional responses of one of the characters. Every question is interpretative and assesses the ability to predict the magnitude of the 4 presented emotions. The test is graded without the need for a judge (so there is no length bias). It's cheap to run (only 171 questions), and produces results that correlate strongly with human preference (Arena ELO) and multi-domain benchmarks like MMLU.
You can run the benchmark on your own models or validate the leaderboard scores using the code in the github repo above.
If you would like to see a model on the leaderboard, get in touch and suggest it!
🧙MAGI-Hard
LLM Benchmarks are chasing a moving target and fast running out of headroom. They are struggling to effectively separate SOTA models from leaderboard optimisers. Can we salvage these old dinosaurs for scrap and make a better benchmark?
MAGI-Hard is a recently added metric to the leaderboard. It is a custom subset of MMLU and AGIEval, selected to have strong discriminatory power between top ability models.
Read more here.
You can use the MAGI test sets with this fork of EleutherAI lm-evaluation-harness.
🎨Creative Writing
This is a brand new leaderboard which uses a LLM judge (Claude 3 Opus) to assess the creative writing abilities of the test models on a series of writing prompts.
You can reproduce these results or run the benchmark on your own models with the EQ-Bench repo on Github.
There has been a distinct lack of automated benchmarks for creative writing because, put simply, it's hard to assess writing quality without humans in the loop. Asking a language model, "How good is this writing (0-10)" elicits poor results. Even if we had a good LLM judge, it's not immediately obvious how to formalise the assessment of creative writing objectively.
The release of Claude 3, in particular the flagship Opus model, has solved half of this equation: it's able to give meaningful & nuanced analysis of creative writing output, and it can tell the difference between a wide range of ability levels.
To solve the other half of the equation, we've come up with an assessment format that works to the strengths of LLM judges and avoids their weaknesses. LLM judges are typically bad at scoring nebulous metrics like "How well written is this piece?" They also find it hard to give consistent scores on an objective rating system if they don't have some exemplar or baseline reference to compare to.
Our test includes:
- 19 writing prompts (typically assessed over multiple iterations)
- 36 narrowly defined assessment criteria
- Including 6 question-specific criteria
- Several criteria targeting positivity bias which (in our opinion) contributes to bad writing
- Exemplar reference output for each question
This approach of breaking down the assessment task into a granular set of criteria and comparing to an exemplar has brought creative writing assessment into the purview of LLM judges. Our test is discriminative amongst a wide range of writing ability levels.
* A note on biases *
LLM judges have biases. LLM-as-a-judge benchmarks such as Alpaca-Eval can exhibit a strong length bias where the judge, (in Alpaca-Eval's case GPT-4), prefers longer outputs. Their approach involves presenting the output from two models to the judge, and the judge says which it thinks is better.
We attempt to mitigate the length bias by: A. assessing by 36 narrow criteria, and B. explicitly instructing the judge not to be biased by length (this seems to work for MT-Bench).
It's possible / likely that this & other biases might still be a factor in scoring (e.g. Claude might prefer its own and other anthropic models). So bear this in mind when interpreting the results.
We include the outputs that the model generated for each prompt so you can judge for yourself.
Alternative Judge Models
Yes, you can use other judge models than Claude Opus (although the results won't be directly comparable). Currently the benchmark pipeline supports Anthropic, OpenAI and Mistral models via their APIs. Soon we will support local models as judges.
* A note on variance *
This benchmark has a relatively small number of test questions (19). We specify generation temperature = 0.7 so each run is different. This means there is significant variation of scores between iterations (avg range: 3.35, std dev: 1.41). To reduce variance we recommend using 3 iterations or more. The leaderboard scores are averaged over 3 iterations.
It costs around $3.00 to bench a model over 3 iterations using Claude 3 Opus at current rates.
If you would like your model included on the creative writing leaderboard, please consider contributing to my compute costs, and get in touch!
⚖️Judgemark measures the ability of a model to judge creative writing using a numerical scoring system.
The Judgemark test incorporates a triple-threat of difficult tasks that LLMs typically struggle with: Evaluating writing quality; effectively using a multi-criteria numerical scoring system; and following complex instructions over a long prompt.
The benchmark requires the model to judge a series of pre-generated creative writing outputs from 19 test models, assigning scores to each of the test models based on a set of 36 narrow criteria for good & bad writing. This differs from other LLM-as-a-judge evals which involve comparing to test model outputs head to head, which is a relatively easier task for judge models to accomplish.
A minimum of 8k supported context length is required for this test. The judging prompts are complex, and incorporate the writing prompt, an exemplar response, the test response, and detailed scoring instructions.
Here's a quick rundown of the metrics:
The Judgemark score is computed like this:
([Average of Pearson Correlations normalised 0-1]+ [Average of Kendall Correlations normalised 0-1]
+ [ANOVA f-statistic normalised 0-1]
+ [Std. Dev. normalised 0-1])
÷ 4
The exact normalisation calculation is in lib/judgemark.py of the EQ-Bench pipeline.
A note on variance: The number of creative writing outputs that are scored per test model is quite low (19 items), to keep the cost of the test manageable. This means the results will vary somewhat between runs, and the 95% confidence intervals are quite high for the individual test model scores. The variance is mitigated to a degree by the fact that there are 19 models tested, so 19x19=361 prompts, each of which involves 36 scored criteria. It should also be noted that the creative writing test (that the judgemark test outputs are sourced from) runs 3x iterations, so the confidence intervals are tighter in the creative writing test than those shown in the judgemark test model score charts.
You can run Judgemark with the EQ-Bench pipeline with the code here.
Cite EQ-Bench:
@misc{paech2023eqbench,
title={EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models},
author={Samuel J. Paech},
year={2023},
eprint={2312.06281},
archivePrefix={arXiv},
primaryClass={cs.CL}
}