Anthropic is widely regarded as one of the most safety-focused major AI laboratories. Its public identity is closely associated with responsible development, alignment research and the effort to understand the risks posed by increasingly capable AI systems. That makes one result from the Future of Life Institute’s AI Safety Index particularly difficult to ignore:
Anthropic received a D in existential safety.

The index scores companies across several safety domains on the basis of public evidence, reviewed by an independent expert panel. The existential safety domain assesses whether a company has a coherent, stated plan for controlling systems substantially more capable than today’s, not whether its current models behave safely. 

The obvious reaction is to ask whether Anthropic is doing enough. The more important question is what the result says about the field itself. If the laboratory most strongly associated with AI safety receives a poor score on existential safety, there are at least three possible explanations. Perhaps existential safety is an extraordinarily difficult technical problem. Perhaps the metrics used to evaluate safety are inadequate. Or perhaps the industry has a persistent gap between what its institutions say about safety and what they are actually able or willing to do.

The uncomfortable possibility is that all three explanations are true.

The paradox of the safety leader

Anthropic’s D is interesting precisely because it does not fit easily into a simple narrative of responsible versus irresponsible AI companies. If a company with little public commitment to safety received such a score, it would be easy to interpret the result as evidence of neglect.

Anthropic presents a harder case.

The company has invested heavily in alignment research and has positioned safety as a central part of its identity. Yet a safety-focused laboratory can still receive a poor evaluation of its ability to address existential risk. That suggests that the problem may lie deeper than corporate priorities.

It raises a fundamental question:

What does it mean to be “good at AI safety” when nobody has yet demonstrated that the hardest safety problems can actually be solved?

Perhaps the problem is simply extremely difficult

The first interpretation is also the least cynical. Advanced AI safety may be genuinely difficult.

No one yet has a rigorous method for guaranteeing that more capable systems will pursue the objectives people intend. Alignment techniques have known limits. A system far more capable than the people testing it is hard to test at all.

In this interpretation, a D does not necessarily mean that Anthropic is failing. It may mean that the underlying task is still beyond the state of the art. This distinction matters.

A company can invest seriously in safety research and still fail to solve a problem that the scientific community does not yet know how to solve. The existence of a serious safety program is therefore not equivalent to evidence that existential risk has been adequately controlled.

Or are we measuring the wrong thing?

A second interpretation concerns the evaluation itself. Safety is easier to measure when the relevant harms are observable.

For current AI systems, researchers can evaluate bias, harmful outputs, privacy violations, robustness and security failures. These problems can often be tested against defined benchmarks. Existential AI safety is different.

The most serious risks involve systems that do not yet exist, capabilities not yet demonstrated, and failures that could not be undone. None of that can be benchmarked.

How do you objectively score an organization’s preparedness for a catastrophe that has never happened?

A safety index can evaluate documented policies, research programs, governance mechanisms and commitments. These are useful indicators. But they are proxies. A laboratory can have an excellent safety policy and still fail to anticipate a new failure mode. Conversely, a laboratory might receive a poor score because its approach does not fit the evaluator’s framework while still making important technical progress.

This does not make safety indices useless. It means they should be treated as instruments for identifying weaknesses, not as definitive measurements of how safe a company or model actually is.

The institutional problem

The third interpretation is more uncomfortable.

Perhaps the problem is not only that safety is difficult to measure or solve. Perhaps AI institutions consistently struggle to prioritize safety when it conflicts with technological and commercial competition.

The history of the field provides reasons for concern. The dissolution of OpenAI’s Superalignment team in 2024, and Jan Leike’s account of safety culture losing ground to product priorities, is the clearest public instance.

One case does not establish institutional bad faith. But it illustrates a structural problem.

Safety research competes with product development for money, computing resources, talent and organizational attention. The incentives are not symmetrical. A new capability can produce a product advantage immediately. A safety breakthrough may prevent a hypothetical future failure.

Organizations operating under competitive pressure will naturally find it easier to justify investment in the first than the second.

The rhetoric-practice gap

This is where the D becomes particularly revealing.

AI companies can sincerely believe that safety matters and still systematically underinvest in it. There is no contradiction. Institutional incentives can produce outcomes that are inconsistent with an organization’s stated values even when the people inside the organization genuinely support those values.

This is a familiar problem in other high-risk industries. Aviation, pharmaceuticals and nuclear energy did not become safe simply because companies declared safety to be important. They developed systems in which safety became embedded in engineering processes, regulation, reporting structures and organizational incentives.

Boeing shows that the embedding is reversible.

The 737 MAX was built by a company that had already made safety structural, and the failures around it were not accidents of engineering. They followed from an orientation shift toward financial performance, in which certification work, internal reporting and design review read as cost rather than as the condition of the aircraft being permitted to fly at all. A firm outside a safety-critical industry making the same choice produces a bad product. A firm inside one produces a grounded fleet, and the damage lands on the category rather than the manufacturer, because every operator and regulator in aviation now carries the consequences of one company’s decision about what it was for. 

The critical distinction is between safety as a value and safety as a constraint.

A value can be overridden. A constraint is something an organization is not permitted to ignore, even when ignoring it would be commercially advantageous. The AI industry has made substantial progress in talking about safety. The harder question is whether safety has become a genuine constraint on frontier development.

The safety race has an incentive problem

This issue becomes particularly difficult because AI development is competitive. Imagine two laboratories. Laboratory A invests heavily in safety research and delays deployment until uncertainty is reduced. Laboratory B accepts greater uncertainty and deploys faster. If the market rewards speed, Laboratory A may be punished for behaving responsibly. This creates a classic collective-action problem.

Individual companies may have incentives to move faster even if the industry as a whole would benefit from stronger safety standards. That is why voluntary commitments alone may not be sufficient.

If safety depends entirely on individual companies consistently choosing restraint over competitive advantage, the system remains vulnerable to the incentives of the next product cycle.

What the D does (and doesn’t) tell us

The Anthropic grade should therefore not be read simply as “Anthropic is unsafe.” Nor should it be dismissed because the Future of Life Institute’s methodology is imperfect. The more useful interpretation lies between those extremes. The D is evidence of a significant mismatch between the ambitions of existential AI safety and the industry’s demonstrated ability to guarantee it.

That mismatch can arise from several sources:

  • the technical problem remains unsolved;
  • evaluation methods remain immature;
  • organizational incentives favor capability development;
  • governance mechanisms are still developing;
  • and safety commitments are not always matched by equivalent resources or constraints.

None of these explanations excludes the others. In fact, their combination may be the most accurate description of the current state of the field.

So, what would a meaningful safety standard look like?

The implication is not that AI laboratories should stop developing models until existential risk has been eliminated. That standard may be impossible to meet.

A more realistic objective is a managed safety regime in which safety capabilities develop alongside AI capabilities.

Such a regime would require several things:

  • Safety research would need resources commensurate with frontier capability research.
  • Governance mechanisms would need to exist before dangerous capabilities are deployed, rather than being created after incidents. 
  • Independent evaluation would need to complement internal safety assessments.
  • And safety commitments would need to survive moments of competitive pressure, when the economic incentives to ignore them are strongest.

This is where Project Glasswing becomes relevant.

Glasswing represents an attempt to establish governance mechanisms before a powerful capability is broadly deployed. Whether those mechanisms are sufficient is uncertain. But the underlying principle is important: governance should not begin after the technology has already escaped the laboratory.

The deeper lesson

The most important conclusion from Anthropic’s D is not that the industry has failed. It is that there may currently be no clear definition of what success in existential AI safety looks like.

If the leading safety-focused laboratory can receive a poor score, we should be cautious about treating corporate safety branding as evidence of safety itself. The field needs better technical methods, better evaluations and stronger institutions. But it also needs something more difficult: a safety culture that remains effective when safety conflicts with speed, market share and technological competition.

That may ultimately be the real test.

The question is whether any laboratory can currently demonstrate that it knows how to make frontier AI safe and whether the institutions surrounding the industry are strong enough to compensate for what researchers do not yet know.

If the answer is no, the D is not merely a grade for one company.

It is a warning about the state of the field.