Why AI benchmark comparisons break down - and how to get reliable answers

https://blogfreely.net/bailirbagw/h1-b-when-summaries-mislead-measuring-journalism-accuracy-in-production-llms

In a controlled evaluation I ran between 2024-03-01 and 2024-05-30 across 40 production-ready models, only 4 models scored better than a coin flip on a set of deliberately hard questions designed to separate summarization skill from factual knowledge