A Harvard study found that GPT-5.5 favored Claude-written feedback over paid expert reports in 132 decided comparisons, evaluating a tool for draft revision while leaving editorial publication decisions outside the experiment.
The paper, dated Oct. 28, was written by Gary King, an Albert J. Weatherhead III University Professor and director of Harvard’s Institute for Quantitative Social Science, and Steven Worthington, the institute’s director of data science. They examined 124 manuscripts submitted to rapidPeer, a double-blind service that gives authors feedback before formal journal submission. Ten papers received two human reviews, for a total of 134.
Field experts chose the paid reviewers, who worked at institutions other than the authors’. Reviewers were told strong work could lead to another paid assignment. In a follow-up survey, 65 of 74 authors rated their human review’s quality high or very high.
Claude Opus 4.8 generated a review for each manuscript at its maximum reasoning setting. In the web-enabled setup, it used a task-specific prompt and averaged 12.4 searches per review. GPT-5.5, from a different model family, preferred the AI report in 132 pairs. Two other comparisons had no consistent outcome because reversing the reports’ order changed the judge’s choice.
Three former journal editors each assessed 10 randomly selected pairs with the reports blinded. They selected the AI report in 26 of 30 judgments, though they had not read the manuscripts themselves.
In a separate audit of the no-tools (“matched”) AI reviews and human reviews, the paper counted 57 incorrect AI claim assessments out of 1,039 and 152 incorrect human assessments out of 956. The AI assessments covered 959 distinct claims. Unverifiable assessments remained in the totals but were not counted as errors; a claim was marked wrong when the manuscript provided evidence against it.
The paper contains a conflicting label: its main text and Table 3 report 132 AI preferences among 132 decided pairs, while Figure 1 appears to show 134 of 134. It does not explain the difference.
King and Worthington said the experiment did not track how authors changed their papers or evaluate editorial acceptance decisions. They said more evidence would be needed to determine whether the results apply beyond rapidPeer and the social sciences. The authors disclosed a conflict tied to their role in creating and operating rapidPeer, while saying the findings did not favor the service. They discuss possible gains in publishing speed and quality, as well as concerns that broad AI use could narrow research perspectives and concentrate influence among a few providers; the study did not measure those effects.
