Emotional Intelligence Benchmark for LLMs
Github | Paper | | Twitter | About
EQ-Bench | Creative Writing | Judgemark
Model | EQB-Corr | Arena-Corr | Cost | Std. Dev. | Self Bias | Family Bias | Judgemark |
---|
⚖️Judgemark measures the ability of a model to judge creative writing using a numerical scoring system.
The Judgemark test incorporates a triple-threat of difficult tasks that LLMs typically struggle with: Evaluating writing quality; effectively using a multi-criteria numerical scoring system; and following complex instructions over a long prompt.
The benchmark requires the model to judge a series of pre-generated creative writing outputs from 19 test models, assigning scores to each of the test models based on a set of 36 narrow criteria for good & bad writing. This differs from other LLM-as-a-judge evals which involve comparing to test model outputs head to head, which is a relatively easier task for judge models to accomplish.
A minimum of 8k supported context length is required for this test. The judging prompts are complex, and incorporate the writing prompt, an exemplar response, the test response, and detailed scoring instructions.
Here's a quick rundown of the metrics:
The Judgemark score is computed like this:
([Average of Pearson Correlations normalised 0-1]The exact normalisation calculation is in lib/judgemark.py of the EQ-Bench pipeline.
A note on variance: The number of creative writing outputs that are scored per test model is quite low (19 items), to keep the cost of the test manageable. This means the results will vary somewhat between runs, and the 95% confidence intervals are quite high for the individual test model scores. The variance is mitigated to a degree by the fact that there are 19 models tested, so 19x19=361 prompts, each of which involves 36 scored criteria. It should also be noted that the creative writing test (that the judgemark test outputs are sourced from) runs 3x iterations, so the confidence intervals are tighter in the creative writing test than those shown in the judgemark test model score charts.
You can run Judgemark with the EQ-Bench pipeline with the code here.