ChatGPT Outperforms Humans in Reasoning Problem Solving, UCLA Study Finds
Researchers at the University of California, Los Angeles (UCLA) have discovered that the ChatGPT language model, powered by OpenAI’s GPT-3, can solve reasoning problems at a level that matches or even surpasses undergraduate students. This breakthrough finding comes from a study conducted by psychologists at UCLA, who compared the performance of GPT-3 with that of 40 UCLA undergraduates.
In the study, both GPT-3 and the human participants were tested on their ability to solve reasoning problems found in intelligence tests and exams like the SAT. The researchers converted complex arrays of shapes into text formats that GPT-3 could process, ensuring that the model had never encountered the questions before. Astonishingly, GPT-3 solved 80% of the problems correctly, while the human participants averaged just below 60%.
The researchers further challenged GPT-3 by asking it to solve SAT “analogy” questions, which involved selecting pairs of related words. They believe these questions had not been published on the internet, meaning GPT-3 could not have been trained on them. When compared with the SAT scores of college applicants, GPT-3 outperformed the average human score.
However, GPT-3 faced difficulties in one particular test that involved matching a passage of prose with a different short story conveying the same meaning. In this test, GPT-3 performed worse than the student volunteers. But the researchers noted that GPT-4, the improved successor to GPT-3, showed improvement in this aspect.
Lead author of the study, Taylor Webb, emphasized that while GPT-3 displayed remarkable reasoning capabilities, it was not at the level of artificial general intelligence or human-level intelligence. The model struggled in areas such as social interactions, mathematical reasoning, and tasks requiring an understanding of physical space. Nonetheless, the technology has made significant progress.
Despite the impressive results, the researchers acknowledged that without access to the inner workings of GPT-3, it is challenging to determine whether the model’s reasoning abilities resemble human thinking or represent a new form of intelligence. Keith Holyoak, a psychology professor at UCLA, commented on this ambiguity, stating that they would like to know if GPT-3 is thinking like a human or if it has developed a completely different approach.
The study, published in the Nature Human Behaviour journal, highlighted GPT-3’s surprising capacity to spot patterns and infer relationships, suggesting that it matches or even surpasses human capabilities in most settings.
The development of ChatGPT and its ability to excel in reasoning problem solving is a significant advancement. However, the researchers acknowledge that there is still progress to be made in achieving true artificial general intelligence.