For decades, the quest to understand proteins—the molecular machines that drive every biological process in our bodies—was a matter of physics and painstaking laboratory observation. Scientists mapped how a string of amino acids folds into a complex 3D shape, a process where a single misplaced atom could mean the difference between a life-saving enzyme and a useless clump of matter.
Enter protein language models (pLMs). By treating protein sequences like sentences in a foreign language, these AI tools can now “write” entirely new proteins that have never existed in nature. The potential is staggering: enzymes that scrub carbon dioxide from the atmosphere, catalysts that eliminate toxic industrial waste, and bespoke medicines tailored to a patient’s specific genetic makeup.
But there is a catch. While these models are producing results that look like magic, they are doing so inside a “black box.” We can see the input and the output, but the reasoning in between remains opaque. In a new perspective paper published in Nature Machine Intelligence, researchers at the Centre for Genomic Regulation (CRG) are warning that this lack of transparency is no longer just a technical hurdle—it is a risk to the reliability and safety of biotechnology.
The study argues that as AI begins to shape real-world decisions in drug discovery and materials science, the scientific community must prioritize “explainable AI” (XAI). Without it, the researchers warn, we are building powerful tools that we cannot fully trust.
The transparency gap in synthetic biology
As a former software engineer, I’ve seen this pattern before in the transition from traditional algorithms to deep learning. In the early days of protein science, researchers used physics-based models. These were slower and less flexible, but they were transparent; you could trace exactly why a protein was predicted to fold a certain way based on the laws of thermodynamics.
“Protein language models are moving fast but our understanding of fundamental biological processes such as folding or catalysis has not advanced alongside these breakthroughs,” says Dr. Noelia Ferruz, Group Leader at the CRG and corresponding author of the paper. She notes that in the rush toward efficiency, we have traded away the mechanistic transparency that once defined the field.
The danger isn’t just academic. If a model predicts a protein structure that looks promising but is based on a statistical fluke or a bias in the training data, the result could be a failed clinical trial or an unstable industrial catalyst. Andrea Hunklinger, the paper’s first author, emphasizes that explainability cannot be a “patch” applied after the model is built; it must be baked into the design process from the start.
Deconstructing the ‘Black Box’
To move toward a more transparent system, the CRG researchers suggest that scientists stop looking at the AI as a single entity and instead interrogate four specific points in the model’s journey:
- The Training Data: Analyzing what the model “read” to learn. This helps identify biases, such as whether the model lacks data on human genetic diversity, which could lead to skewed predictions for certain populations.
- The Input Sequence: Identifying which specific amino acids or regions of a protein sequence most heavily influenced the AI’s prediction.
- The Architecture: “Opening the hood” to examine the internal artificial neurons and ensuring the information is being processed logically rather than through noise.
- Input-Output Behavior: Probing the model by slightly altering the protein sequence to see how the answer changes—essentially “nudging” the AI to see if its logic holds up under pressure.
From ‘Evaluators’ to ‘Teachers’
The CRG team conducted the most comprehensive survey of its kind to date, reviewing dozens of studies to see how XAI is currently being used. They found that most researchers are using AI in a very limited capacity. Most tools act as “Evaluators,” simply checking if the AI has learned patterns that biologists already know.
The researchers propose a hierarchy of how explainable AI can evolve to actually drive discovery rather than just verify it:
| Role | Function | Impact Level |
|---|---|---|
| Evaluator | Benchmarks model against known biological patterns. | Low (Verification) |
| Multitasker | Uses learned signals to annotate new proteins. | Moderate (Support) |
| Engineer/Coach | Trims AI components to steer toward desired traits. | High (Optimization) |
| Teacher | Reveals entirely new biological principles to humans. | Transformative (Discovery) |
The “Teacher” role is the ultimate goal. The authors compare this to the moment AlphaZero began uncovering novel chess strategies that surprised grandmasters. In protein science, a “Teacher” model wouldn’t just give a researcher a sequence; it would explain why that sequence works and why alternatives would fail, potentially revealing new rules of molecular interaction that humans have missed for centuries.
The path to controllable design
Reaching this level of insight requires more than just better code. The CRG paper calls for a fundamental shift in how the biotech community operates, specifically demanding three things: the creation of robust benchmarks to test if AI explanations are actually accurate, the development of open-source tooling so labs can compare results, and—most importantly—rigorous laboratory validation.
“The real holy grail is controllable protein design,” Dr. Ferruz explains. She envisions a future where a scientist can request a protein with a specific shape active at a specific pH, and the AI provides the sequence along with a mechanistic explanation, such as how a particular mutation maintains a hydrogen-bonding network essential for stability.
Until then, the authors argue, AI remains a powerful pattern recognizer rather than a true biological partner. The next critical step for the field will be the integration of these XAI frameworks into open-source platforms, allowing the broader scientific community to verify these “black box” predictions in wet labs.
Disclaimer: This article is for informational purposes only and does not constitute medical or professional biotechnological advice.
The research community is now looking toward the development of standardized evaluation frameworks to determine if AI-derived biological insights are mathematically sound or merely statistical correlations. Further updates on these benchmarks are expected as more labs adopt the CRG’s proposed open-source tooling.
Do you think we should trust AI-designed proteins if we can’t explain how they were made? Let us know in the comments or share this story with your network.
