We introduced AI, and performance improved. Good news, to begin with.
Then comes a slightly awkward question. How did it compare with letting AI do the task alone?
A comparison of human–AI collaboration needs three seats at the table: humans alone, AI alone, and humans with AI. Leave one empty, and the same result tells a different story.
A meta-analysis by Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone attends to that empty seat. Helping a human is one thing; a combination outperforming either component alone is another. The researchers distinguish the two. [1]
Improvement and synergy
The team collected experiments published between January 2020 and June 2023 that permitted a comparison of all three conditions. Their preregistered analysis brought together 370 effect sizes from 106 experiments in 74 papers.
Compared with humans alone, collaboration performed better: the standardized effect size, Hedges’ g, was 0.64. Compared with the better of humans or AI alone, the effect was −0.23. Neither 95% confidence interval crossed zero. [1]
The assisted humans, in other words, did better than before. But the combination did not outperform the most capable component. On average, it did worse.
Two people can carry a heavy load. Add a two-person planning meeting to a load that one person can carry quickly, however, and the situation changes. The agreeable sound of “collaboration” does not solve the question of how to carry it.
What does “the human makes the final decision” mean?
Jump from here to “people get in the way, so remove them,” and you land somewhere the study never examined.
More than 95% of the systems studied involved humans making final decisions after receiving AI input. Diverse forms of collaboration, such as dividing different tasks or repeatedly reframing a question together, were not evenly represented. Differences among studies were also substantial: the heterogeneity statistic I² for synergy was 97.7%.
For creative tasks, average synergy was positive, but its confidence interval crossed zero. That is no guarantee that creative collaboration will work. And the studies extend only to mid-2023. Reading them as a report card for today's models would be a stretch.
The paper's sharpness lies not in passing judgment on the future of humans and AI, but in asking what comparison would justify saying that a collaboration was worthwhile.
There are reasons for human final judgment beyond accuracy: responsibility, safety, and values among them. The need for that judgment and whether an added review stage actually reduces errors can be examined separately. It would be a pity to settle both with the sentence “A human checks it.”
Before removing a person, question the division of work
From here, the discussion turns to possible implications for practice.
AI answers. A person reads it and corrects anything that seems wrong. It is a natural arrangement. But to correct an answer reliably, the person needs clues that an error exists and enough capacity to investigate. Giving a reviewer a screen and responsibility does not automatically improve their ability to review.
If collaboration fails to improve performance, is the human not capable enough? Is the AI not capable enough? We might also ask what was shown, what was delegated, and at which stage.
A person forming an independent assessment before seeing AI's answer is doing a different job from someone who reads that answer first. AI delivering a conclusion differs from AI listing possible omissions. This meta-analysis cannot determine which arrangement is best. That is precisely why concrete workflows need testing.
The comparison also changes with the purpose. If the goal is to develop a person's skill, the AI's standalone performance at this moment is not the only standard. If explaining a decision or recovering from a mistake matters, those need measures too.
The answer to “One score cannot measure this” is not to measure nothing. It is to say what we want to improve, and then compare the person, the AI, and their combination. Respecting collaboration is different from marking its performance generously.
Some AI makes people more capable. Some people compensate for AI's weaknesses. But putting the two in the same place is not yet the same as building a relationship in which they complement each other. There is still design work to do.
Before saying, “It was better because we worked together,” ask: better than what? That question can be the beginning of the next collaboration.