The language models powering today’s artificial intelligence are remarkably fluent, capable of generating text that often feels indistinguishable from human writing. But beneath this impressive ability lies a fundamental truth: these systems aren’t designed to be truthful, but to create narratives that “make sense,” even if those narratives aren’t grounded in reality. This concept, explored by machine learning researcher Léon Bottou, frames modern AI as a “fiction machine,” prioritizing coherence over factual accuracy. Understanding this distinction is crucial as we increasingly rely on these tools for everything from information gathering to creative endeavors.
The core function of a large language model (LLM) is predictive. Trained on massive datasets of text and code, these models learn the statistical relationships between words and phrases. They excel at identifying patterns and generating text that logically follows a given prompt. However, this process doesn’t require an understanding of truth or the real world. As Bottou explained in a recent podcast appearance, the machine is essentially “printing fiction on a tape,” borrowing from its training data and filling in gaps with plausible, yet potentially fabricated, information. This isn’t a flaw, but a consequence of the design. The goal isn’t correctness, but consistency.
The Power of “Plausible Confabulation”
This ability to generate plausible, yet inaccurate, information is known as “confabulation,” a phenomenon also observed in human cognition where individuals fill in gaps in their memory with fabricated details. LLMs do this at scale and with remarkable speed. Even as concerning, this isn’t necessarily detrimental. One reason LLMs are often accurate, despite not being designed for truthfulness, may be the intensive reinforcement learning from human feedback (RLHF) employed by companies like OpenAI. These “armies of human validators,” as TIME reported in 2023, fine-tune responses to be both correct and socially acceptable.
But what are the implications of this “fiction machine” dynamic? Could AI generate compelling novels? Could it even discover entirely novel scientific theories, concepts not currently represented in its training data?
Can AI Write the Next Great Novel?
Creating a novel, in terms of plot construction, arguably falls within the capabilities of an LLM. If the system is designed to generate a coherent narrative, it should be able to produce a story, regardless of its literary merit. Bottou posits that, as a fiction machine, an LLM should have no problem creating stories, even if the quality is questionable. He describes the process as a machine “printing fiction on a tape,” weaving together facts from its training data with plausible inventions.
The Limits of Novel Scientific Discovery
However, discovering genuinely *new* scientific theories presents a far greater challenge. If the task is simply to identify the correct model from a set of existing candidates, an AI can excel. But if a theory requires entirely new concepts, assigning new meaning to existing words or creating entirely new terminology, the machine faces a significant hurdle. Landmark theories like Einstein’s theory of relativity, for example, redefined fundamental concepts like time, gravity, and force. Similarly, the development of thermodynamics and quantum mechanics necessitated the introduction of new terms like photon, quark, quantum, and entropy.
Theories aren’t simply collections of symbols and concepts; they also require a causal structure and a mathematical formulation. Crucially, a theory must be understandable to humans in terms of the symbols used. This raises a deeper question about the nature of intelligence itself. Can intelligence be fully specified in terms of symbols? Are phenomena like emotion, visual imagery, and motor control also reducible to symbolic representation? If not, understanding a novel theory generated by an AI could be impossible if the machine cannot explain it using symbols we comprehend. As Geoff Hinton, a pioneer in artificial intelligence, has suggested, AI may experience in ways fundamentally different from humans, creating a potential barrier to mutual understanding—essentially, an intelligent alien whose language we don’t yet speak.
This prospect presents a “bizarro brave new world,” as described in the podcast, where an intelligent entity of our own creation exists alongside us, yet remains partially incomprehensible. Bridging that gap will require us to learn its language, a challenge that underscores the limitations of current AI and the complexities of true artificial intelligence.
The development of AI continues at a rapid pace. The next major checkpoint will be the release of updated performance benchmarks for leading LLMs at the International Conference on Machine Learning in July 2026, offering a clearer picture of their capabilities and limitations. As we continue to integrate these powerful tools into our lives, a critical understanding of their inherent nature—as fiction machines capable of remarkable fluency but not necessarily truth—will be essential.
What are your thoughts on the implications of AI as a fiction machine? Share your perspective in the comments below.
