Language Models Agree With Each Other But Not Humans
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- Language models share selected choices at roughly 2.3 times the rate of human readers, demonstrating a much higher level of internal consistency among different models than among people.
- Rival models from competing laboratories, such as GPT-5.4 and Claude Opus 5, show surprisingly high agreement with each other, often matching their selections more strongly than a model matches its own self-evaluations.
- Newer models released in 2026 exhibit significantly higher mutual agreement than models released in 2024, showing that recent technological generations are converging even more tightly.
- Despite their high agreement with one another, language models show very weak alignment with human choices, with models barely exceeding human yardsticks and falling well short of human consensus ceilings.
Summary & Methodology Analysis
The paper investigates whether language models produce more uniform and predictable choices than real humans by analyzing how they select sentences from public web documents. The researchers gathered a corpus of 120 public web documents containing thousands of reader mark sets collected from a social highlighting platform. They tested 18 different model arms, feeding them the documents as numbered sentences to generate sentence rankings and keep sets. To measure agreement fairly, the team calculated excess agreement by comparing actual overlap against a randomized statistical baseline, using a simulation budget and a domain clustered bootstrap with 10,000 resamples to evaluate uncertainty and ensure that classical extractive algorithms did not falsely trigger high scores.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the main finding of the paper?
The paper found that language models agree with each other much more than human readers agree with each other, but these models show very little agreement with actual human choices.
Q2. How did the researchers measure agreement?
They measured agreement by having models and human readers select parts of web documents, then calculating excess agreement by comparing the overlap of their choices against an expected random baseline.
Q3. Did different companies' models agree with each other?
Yes, models from rival laboratories showed high agreement with each other, often matching their selections more strongly than a model matches its own paraphrased prompts.
Q4. What dataset was used in the study?
The study used a corpus of 120 public web documents from 78 domains, along with 2,523 reader mark sets collected from a social highlighting platform.
Q5. How were the models tested?
Each of the 18 model arms received the documents as numbered sentences and returned a ranking, with the top selections acting as the keep set.
Q6. What did the researchers do to make sure the statistical tests were correct?
They used a domain clustered bootstrap with 10,000 resamples, alongside calibration checks ensuring that random baselines landed near zero and classical extractive pairs did not exceed human yardsticks.
Q7. Were there any limitations regarding human participation?
Yes, the human side was never given a specific task or asked to do anything, creating an instruction asymmetry because the models were explicitly given a task while human marks were collected passively.
Q8. Did newer models agree more than older models?
Yes, 2026 models agreed with each other at a much higher rate than 2024 models, showing a clear generational increase in mutual alignment.
Q9. What training data information was available for the models?
The paper does not specify the training data for the models, meaning training data overlap between model arms and the evaluated documents could not be ruled out.