Emotional Intelligence Benchmark for LLMs

Github | Paper | | Twitter | About

EQ-Bench | Creative Writing | Judgemark

Model EQB-Corr Arena-Corr Cost Std. Dev. Self Bias Family Bias Judgemark

⚖️Judgemark measures the ability of a model to judge creative writing using a numerical scoring system.

The Judgemark test incorporates a triple-threat of difficult tasks that LLMs typically struggle with: Evaluating writing quality; effectively using a multi-criteria numerical scoring system; and following complex instructions over a long prompt.

The benchmark requires the model to judge a series of pre-generated creative writing outputs from 19 test models, assigning scores to each of the test models based on a set of 36 narrow criteria for good & bad writing. This differs from other LLM-as-a-judge evals which involve comparing to test model outputs head to head, which is a relatively easier task for judge models to accomplish.

A minimum of 8k supported context length is required for this test. The judging prompts are complex, and incorporate the writing prompt, an exemplar response, the test response, and detailed scoring instructions.

Here's a quick rundown of the metrics:

EQB-Corr: Correlation with EQ-Bench scores.
Arena-Corr: Correlation with LMSys Arena ELO.
Cost: in USD to run the Judgemark benchmark for this model.
Std. Dev.: Standard deviation of scores for the test models. This is a rough proxy for discriminative power, or how well the judge was able to reliably separate each model by ability level.
Self Bias: The deviation from the predicted score when the judge model scores its own output. The bias stats should be taken with a grain of salt as the sample size we've computed them from is quite low.
Family Bias: The deviation from the predicted score when the judge model scores models in its family (e.g., Claude opus scoring sonnet & haiku).
Judgemark: A final aggregate score derived from the correlation & std. dev. stats.
Stats: Test model scores and raw stats from the Judgemark run.
📊: Chart of the test model scores as evaluated by this judge.
(Not pictured) ANOVA f-statistic: A measure of how well the judge model was able to tell apart the different test models based on their writing, using cluster analysis.

The Judgemark score is computed like this:

   ([Average of Pearson Correlations normalised 0-1]
     + [Average of Kendall Correlations normalised 0-1]
     + [ANOVA f-statistic normalised 0-1]
     + [Std. Dev. normalised 0-1])
   ÷ 4

The exact normalisation calculation is in lib/judgemark.py of the EQ-Bench pipeline.

A note on variance: The number of creative writing outputs that are scored per test model is quite low (19 items), to keep the cost of the test manageable. This means the results will vary somewhat between runs, and the 95% confidence intervals are quite high for the individual test model scores. The variance is mitigated to a degree by the fact that there are 19 models tested, so 19x19=361 prompts, each of which involves 36 scored criteria. It should also be noted that the creative writing test (that the judgemark test outputs are sourced from) runs 3x iterations, so the confidence intervals are tighter in the creative writing test than those shown in the judgemark test model score charts.

You can run Judgemark with the EQ-Bench pipeline with the code here.