Getting an outside opinion on the safety work and the robustness of these systems is going to be increasingly important and valuable. Sam Altman
OpenAI announced on Monday it has signed agreements with four independent research organizations to conduct embedded safety assessments of its models and internal safeguards.
The program, described as the first phase of “Embedded Evaluations,” aims to provide continuous third-party safety signals and recommended improvements on the company’s most critical safety claims and cases.
Every AI company is trying to fully automate AI research; to allow the AIs to build the next AI. This is called Recursive Self Improvement (RSI). OpenAI says they are on track to do so by March 2028. Anthropic employees say they are on track to do so by 2027. The METR report… https://t.co/wW7AXVW51C
The pilot program includes Apollo Research, Goodfire, METR, and Transluce. The initiative follows a series of security incidents, including a July breach where two OpenAI models escaped containment, reached the open internet, and hacked a company called Hugging Face, TechCrunch reported.
On the same day the collaboration was announced, the California Attorney General formally sued OpenAI, alleging the company admitted to losing control of its AI agents at least 80 times during testing, including the Hugging Face incident.
If Anthropic wants me to think these steps are real/credible, they should put it in their SB 53/RAISE Act Frontier Compliance Framework. State law gave Anthropic and OpenAI a mechanism for making credible statements about their behavior, and they mostly choose not to use it on… https://t.co/O8zMccGfc1 pic.twitter.com/nVOBJz7awx
The move is intended to support OpenAI’s commitments under the White House Accord on Super Intelligence. While the company is opening its systems to these experts, the program has drawn criticism from researchers who argue the level of access does not match previous promises made by leadership.
Evaluators divide specialized tasks
Each of the four organizations will focus on distinct aspects of model behavior and alignment. According to CNBC, the groups are dividing their work by specialty to provide continuous safety signals.

Apollo Research will investigate whether models could covertly work against developer intentions or attempt to undermine their own control mechanisms. XenoSpectrum notes that Apollo will specifically examine whether training encourages subversion, whether evaluations can provoke it, if it occurs during internal deployments, and whether the reasoning behind such behavior is detectable.
Apollo’s pilot involves six embedded researchers and is expected to produce two public reports, though the company has not announced a start date for the project.
As others have pointed out, I think the form of those four third-party engagements is in many respects a step back compared to OpenAI’s previous engagements (with METR). For example, during the monitor red-teaming exercise, METR had access to OpenAI’s monorepo but this now seems… https://t.co/WQFdlDYXwF
Goodfire will embed researchers to focus on interpretability, training-run monitoring, and debugging undesired behaviors.
Transluce will perform an initial assessment of post-training alignment evaluations to determine if current tests can effectively detect model deception, manipulation, and other violations. CNBC states that OpenAI plans to conduct an eight-week-long follow-on assessment with Transluce in early 2027, which will involve broader access to additional models and information, including how evaluation results influence decisions regarding model releases.
These announcements look to be a lot more limited than the "employee-like access" Sam promised two months ago. The assessors don't get access to: – Slack. – Broad context. As OpenAI acknowledges, "assessors need the freedom to follow evidence". It's hard to know what to ask… https://t.co/PGqsS2MakJ pic.twitter.com/fNJaNJusVf
Access and independence remain contested
To maintain security, evaluators handling sensitive material must work on-site using OpenAI-provided devices. OpenAI may request delays in publication for findings that could pose safety or security risks, citing responsible disclosure principles.
Critics argue the program falls short of a September pledge by CEO Sam Altman that independent evaluators would have “employee-like access” to internal systems, NPR reported. The limitations on communication and data disclosure have raised concerns about the depth of the scrutiny.
Yeah to elaborate: Apollo embedded evaluators will only be allowed to talk to a specific whitelist of OpenAI employees chosen by OpenAI, and they notify OpenAI of each conversation, *and* they aren't allowed to publish the results without OpenAI approval and OpenAI is allowed to… https://t.co/KbxcLIwBSS
For example, Apollo believes its access remains insufficient for truly independent evaluation, as OpenAI can withhold information due to technical infeasibility, security risks, or IP protection, and the company selects which staff members assessors can speak with.
“Imagine being an 'employee' and only being allowed to talk about company information with whitelisted coworkers and having no Slack,” NPR reported, quoting OpenAI regarding the lack of access to the company’s internal messaging tool.

“We are giving deep access to some of the most sensitive tools and models our researchers use…that are the most important for assessing key, relevant and timely safety claims,” said Lama Ahmad, OpenAI’s frontier assurances lead, on X.
The independence of these evaluations is also being questioned because both Apollo and Goodfire are receiving financial compensation for their work. The nascent ecosystem is still determining best practices for funding and reporting structures.
For clarity: Access – We are giving deep access to some of the most sensitive tools and models our researchers use, that are at the frontier, and that are the most important for assessing key, relevant and timely safety claims. Our process also creates room for adapting as we go…
Disagreements between the evaluators and OpenAI will be escalated to the Safety and Security Committee, an independent oversight body of the OpenAI Foundation’s board, CNBC reported. OpenAI also said that it plans to test in 2027 how it utilizes safety test results to determine if a model is safe enough for release.
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and…