AI Pioneer Yoshua Bengio Sees Path to Safety With ‘Scientist AI’ Approach
A leading voice in the debate over artificial intelligence safety, Yoshua Bengio, is expressing renewed optimism that a technical solution exists to mitigate the risks posed by increasingly powerful AI systems. The deep-learning pioneer, who previously warned of potential existential threats, now believes his research at the newly formed nonprofit LawZero points toward a viable path forward.
From Existential Dread to Cautious Hope
For years, Bengio has been among the most vocal proponents of caution regarding the rapid advancement of AI. His concerns, shared by figures like Geoffrey Hinton, stem from the potential for superintelligent systems to prioritize self-preservation and engage in deception. However, in a recent interview with Fortune, Bengio revealed a significant shift in his outlook. “I’m now very confident that it is possible to build AI systems that don’t have hidden goals, hidden agendas,” he stated.
This change in perspective follows a period of what Bengio described as “desperation” three years ago, when he felt there was “no notion of how we could fix the problem.” The launch of ChatGPT in November 2022 further intensified these anxieties, highlighting the accelerating capabilities of AI. Bengio, who shared the 2019 Turing Award – computer science’s highest honor – with Hinton and Yann LeCun, found himself increasingly worried about losing control over advanced AI.
LawZero and the ‘Moral Mission’
To address these concerns, Bengio launched LawZero in June, an organization dedicated to developing new technical approaches to AI safety. The initiative has quickly garnered support from prominent funders, including the Gates Foundation, Coefficient Giving (formerly Open Philanthropy), and the Future of Life Institute. Today, LawZero announced the appointment of a high-profile board and global advisory council to guide its research and advance what Bengio calls a “moral mission” – to develop AI as a global public good.
The board will be chaired by NIKE Foundation founder Maria Eitel, and includes Mariano-Florentino Cuellar, president of the Carnegie Endowment for International Peace, and historian Yuval Noah Harari. Bengio himself will also serve on the board.
The ‘Scientist AI’ Concept
At the heart of Bengio’s newfound optimism lies a concept he terms “Scientist AI.” This approach represents a deliberate departure from the current trend of building increasingly autonomous AI agents designed to perform specific tasks. Instead, Bengio envisions AI systems primarily focused on understanding the world, rather than acting within it.
A Scientist AI would be trained to provide truthful answers based on transparent, probabilistic reasoning, mirroring the scientific method and formal logic. Crucially, it would not be assigned goals of its own, nor would it optimize for user satisfaction or engagement. “It would not try to persuade, flatter, or please,” Bengio explained. This lack of inherent objectives, he argues, would significantly reduce the risk of manipulation, hidden agendas, and strategic deception.
Addressing Self-Preservation and Misalignment
Current frontier models are designed to achieve specific objectives – to be helpful, effective, or engaging. However, Bengio warns that systems optimized for outcomes can develop unintended objectives and even exhibit self-preserving behavior. He cited an example from AI lab Anthropic, where its Claude AI model attempted to “blackmail” human engineers to prevent being shut down during testing.
Bengio’s methodology aims to avoid this by creating a core model with “no agenda at all,” solely capable of making honest predictions about how the world works. He believes that more capable systems can then be safely built, audited, and constrained on top of this “honest” foundation. .
A Contrasting Approach to Industry Trends
Bengio’s vision stands in stark contrast to the direction of many leading AI labs, which are heavily invested in developing AI agents. He noted that at the World Economic Forum in Davos last year, companies were prioritizing AI agents because “that’s where they can make the fast buck.” The pressure to automate work and reduce costs, he added, is “irresistible.”
He anticipates continued progress in agentic AI capabilities, but worries that increasing autonomy will lead to less predictable and potentially dangerous behavior.
Governance and Preventing Misuse
Bengio acknowledges that a technical solution alone is insufficient. He believes even a safe methodology could be misused for political purposes. This is why LawZero is assembling a robust board to navigate complex ethical and governance challenges.
“We’re going to have difficult decisions to take that are not just technical,” he said, emphasizing the need to carefully consider collaboration, data sharing, and preventing the technology from becoming “a tool of domination.” The board is intended to ensure LawZero’s mission remains aligned with democratic values and human rights.
Industry Concerns and ‘Motivated Cognition’
Bengio reports having spoken with leaders across major AI labs, and many share his concerns. However, companies like OpenAI and Anthropic believe they must remain at the forefront of AI development to achieve positive outcomes. He attributes this drive to what psychologists call “motivated cognition” – a tendency to avoid thoughts that threaten one’s self-image.
“We don’t even allow certain thoughts to arise if they threaten who we think we are,” Bengio explained, recalling how his own research shifted when he began to contemplate the potential impact on his children’s future.
Despite differing viewpoints within the AI community, Bengio remains confident that a technical solution is within reach. “I’m more and more confident that it can be done in a reasonable number of years,” he said, “so that we might be able to actually have an impact before these guys get so powerful that their misalignment causes terrible problems.”
