Emotional Intelligence Benchmark for LLMs

Github | Paper | | Twitter | Leaderboard


📩How to Submit

At this time we only accept submissions of open weight models that are available to everyone via HuggingFace.

To submit, get in touch by email or twitter with:

We will then verify the result on our end and add to the leaderboard. This project is self funded so please respect that we don't have unlimited compute!


EQ-Bench is a benchmark for language models designed to assess emotional intelligence.

Why emotional intelligence? One reason is that it represents a subset of abilities that are important for the user experience, and which isn't explicitly tested by other benchmarks. Another reason is that it's not trivial to improve scores by fine tuning for the benchmark, which makes it harder to "game" the leaderboard.

EQ-Bench is a little different from traditional psychometric tests. It uses a specific question format, in which the subject has to read a dialogue then rate the intensity of possible emotional responses of one of the characters. Every question is interpretative and assesses the ability to predict the magnitude of the 4 presented emotions. The test is graded without the need for a judge (so there is no length bias). It's cheap to run (only 171 questions), and produces results that correlate strongly with human preference (Arena ELO) and multi-domain benchmarks like MMLU.

You can run the benchmark on your own models or validate the leaderboard scores using the code in the github repo above.

If you would like to see a model on the leaderboard, get in touch and suggest it!


LLM Benchmarks are chasing a moving target and fast running out of headroom. They are struggling to effectively separate SOTA models from leaderboard optimisers. Can we salvage these old dinosaurs for scrap and make a better benchmark?

MAGI-Hard is a recently added metric to the leaderboard. It is a custom subset of MMLU and AGIEval, selected to have strong discriminatory power between top ability models.

Read more here.

You can use the MAGI test sets with this fork of EleutherAI lm-evaluation-harness.

🎨Creative Writing

This is a brand new leaderboard which uses a LLM judge (Claude 3 Opus) to assess the creative writing abilities of the test models on a series of writing prompts.

You can reproduce these results or run the benchmark on your own models with the EQ-Bench repo on Github.

There has been a distinct lack of automated benchmarks for creative writing because, put simply, it's hard to assess writing quality without humans in the loop. Asking a language model, "How good is this writing (0-10)" elicits poor results. Even if we had a good LLM judge, it's not immediately obvious how to formalise the assessment of creative writing objectively.

The release of Claude 3, in particular the flagship Opus model, has solved half of this equation: it's able to give meaningful & nuanced analysis of creative writing output, and it can tell the difference between a wide range of ability levels.

To solve the other half of the equation, we've come up with an assessment format that works to the strengths of LLM judges and avoids their weaknesses. LLM judges are typically bad at scoring nebulous metrics like "How well written is this piece?" They also find it hard to give consistent scores on an objective rating system if they don't have some exemplar or baseline reference to compare to.

Our test includes:

This approach of breaking down the assessment task into a granular set of criteria and comparing to an exemplar has brought creative writing assessment into the purview of LLM judges. Our test is discriminative amongst a wide range of writing ability levels.

* A note on biases *

LLM judges have biases. LLM-as-a-judge benchmarks such as Alpaca-Eval can exhibit a strong length bias where the judge, (in Alpaca-Eval's case GPT-4), prefers longer outputs. Their approach involves presenting the output from two models to the judge, and the judge says which it thinks is better.

We attempt to mitigate the length bias by: A. assessing by 36 narrow criteria, and B. explicitly instructing the judge not to be biased by length (this seems to work for MT-Bench).

It's possible / likely that this & other biases might still be a factor in scoring (e.g. Claude might prefer its own and other anthropic models). So bear this in mind when interpreting the results.

We include the outputs that the model generated for each prompt so you can judge for yourself.

Alternative Judge Models

Yes, you can use other judge models than Claude Opus (although the results won't be directly comparable). Currently the benchmark pipeline supports Anthropic, OpenAI and Mistral models via their APIs. Soon we will support local models as judges.

* A note on variance *

This benchmark has a relatively small number of test questions (19). We specify generation temperature = 0.7 so each run is different. This means there is significant variation of scores between iterations (avg range: 3.35, std dev: 1.41). To reduce variance we recommend using 3 iterations or more. The leaderboard scores are averaged over 3 iterations.

It costs around $3.00 to bench a model over 3 iterations using Claude 3 Opus at current rates.

If you would like your model included on the creative writing leaderboard, please consider contributing to my compute costs, and get in touch!

⚖️Judgemark measures the ability of a model to judge creative writing using a numerical scoring system.

The Judgemark test incorporates a triple-threat of difficult tasks that LLMs typically struggle with: Evaluating writing quality; effectively using a multi-criteria numerical scoring system; and following complex instructions over a long prompt.

The benchmark requires the model to judge a series of pre-generated creative writing outputs from 19 test models, assigning scores to each of the test models based on a set of 36 narrow criteria for good & bad writing. This differs from other LLM-as-a-judge evals which involve comparing to test model outputs head to head, which is a relatively easier task for judge models to accomplish.

A minimum of 8k supported context length is required for this test. The judging prompts are complex, and incorporate the writing prompt, an exemplar response, the test response, and detailed scoring instructions.

Here's a quick rundown of the metrics:

EQB-Corr: Correlation with EQ-Bench scores.
Arena-Corr: Correlation with LMSys Arena ELO.
Cost: in USD to run the Judgemark benchmark for this model.
Std. Dev.: Standard deviation of scores for the test models. This is a rough proxy for discriminative power, or how well the judge was able to reliably separate each model by ability level.
Self Bias: The deviation from the predicted score when the judge model scores its own output. The bias stats should be taken with a grain of salt as the sample size we've computed them from is quite low.
Family Bias: The deviation from the predicted score when the judge model scores models in its family (e.g., Claude opus scoring sonnet & haiku).
Judgemark: A final aggregate score derived from the correlation & std. dev. stats.
Stats: Test model scores and raw stats from the Judgemark run.
📊: Chart of the test model scores as evaluated by this judge.
(Not pictured) ANOVA f-statistic: A measure of how well the judge model was able to tell apart the different test models based on their writing, using cluster analysis.

The Judgemark score is computed like this:

   ([Average of Pearson Correlations normalised 0-1]
     + [Average of Kendall Correlations normalised 0-1]
     + [ANOVA f-statistic normalised 0-1]
     + [Std. Dev. normalised 0-1])
   ÷ 4

The exact normalisation calculation is in lib/judgemark.py of the EQ-Bench pipeline.

A note on variance: The number of creative writing outputs that are scored per test model is quite low (19 items), to keep the cost of the test manageable. This means the results will vary somewhat between runs, and the 95% confidence intervals are quite high for the individual test model scores. The variance is mitigated to a degree by the fact that there are 19 models tested, so 19x19=361 prompts, each of which involves 36 scored criteria. It should also be noted that the creative writing test (that the judgemark test outputs are sourced from) runs 3x iterations, so the confidence intervals are tighter in the creative writing test than those shown in the judgemark test model score charts.

You can run Judgemark with the EQ-Bench pipeline with the code here.

Cite EQ-Bench:

	title={EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models}, 
	author={Samuel J. Paech},
MAGI draws from the MMLU and AGIEval tests. Click to show citations

		title={Measuring Massive Multitask Language Understanding},
		author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},
		journal={Proceedings of the International Conference on Learning Representations (ICLR)},

		title={Aligning AI With Shared Human Values},
		author={Dan Hendrycks and Collin Burns and Steven Basart and Andrew Critch and Jerry Li and Dawn Song and Jacob Steinhardt},
		journal={Proceedings of the International Conference on Learning Representations (ICLR)},

		title={AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models},
		author={Wanjun Zhong and Ruixiang Cui and Yiduo Guo and Yaobo Liang and Shuai Lu and Yanlin Wang and Amin Saied and Weizhu Chen and Nan Duan},

		title = "Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems",
		author = "Ling, Wang  and
		Yogatama, Dani  and
		Dyer, Chris  and
		Blunsom, Phil",
		booktitle = "Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
		month = jul,
		year = "2017",
		address = "Vancouver, Canada",
		publisher = "Association for Computational Linguistics",
		url = "https://aclanthology.org/P17-1015",
		doi = "10.18653/v1/P17-1015",
		pages = "158--167",

		title={Measuring Mathematical Problem Solving With the MATH Dataset},
		author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},

		title={LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning},
		author={Jian Liu and Leyang Cui and Hanmeng Liu and Dandan Huang and Yile Wang and Yue Zhang},
		booktitle={International Joint Conference on Artificial Intelligence},

		title={JEC-QA: A Legal-Domain Question Answering Dataset},
		author={Zhong, Haoxi and Xiao, Chaojun and Tu, Cunchao and Zhang, Tianyang and Liu, Zhiyuan and Sun, Maosong},
		booktitle={Proceedings of AAAI},

		title={From LSAT: The Progress and Challenges of Complex Reasoning},
		author={Siyuan Wang and Zhongkun Liu and Wanjun Zhong and Ming Zhou and Zhongyu Wei and Zhumin Chen and Nan Duan},
		journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},