It’s a significant moment for the market, and a major test for the industry. But as the buzz builds, a bigger question looms: is anyone minding the risks as Anthropic races ahead?

Let’s start off with what the company is doing well. 

1. Anthropic pays a much lower salary to the Chief Executive Officer (CEO) and co-founder, Dario Amodei, than other companies in the technology sector. 
2. It will not pay investors dividends, instead committing to investing in company growth.
3. As a Public Benefit Corporation, it is legally required to consider the impact of its business on stakeholders that include employees, customers, and the public.

Yet, there is a tension that is hard to miss; Anthropic’s mission to build responsible AI clashes with warnings from its CEO that its increasingly capable systems could be dangerous in the wrong hands or without proper oversight. Complicating matters, the founders’ control the decisions that shape how AI risks are managed. That means, tens of thousands of people will likely own shares in Anthropic yet have very little say over how it is run.

Insiders are poised to make big gains when the company enters the public market while the costs of its big bets will likely be borne by everyone else. 

Can we rely on Anthropic to govern itself?

Currently, there is no federal framework to regulate AI-related risks. But self-policing is not an adequate solution, given the risks that AI models pose, and there are visible tensions between Anthropic’s safety rhetoric and its actual behavior.  

• Anthropic sought modifications to lift export controls imposed by the government on its AI model, Mythos.
• Amodei reversed the position he took in 2023 calling for broader oversight to address AI risks to transparency and auditing. 
• The company has spent millions on lobbying and to advance an AI Safety Bill that closely aligns with its business interests. 
• Anthropic has reneged on its commitment to not release newer models if safety risks can’t be mitigated.

The stakes are too high to let AI companies write their own rules. When companies play a hand in shaping regulations meant to govern them, everyone else potentially bears the cost of failure. Without strong regulation, we can expect a race to the bottom and the likelihood that the risks can’t be undone. 

Anthropic’s exposure to severe AI risks

Oxfam has found that the company is not only exposed to serious AI risks but as a leading player in the field, it plays a part in creating and externalizing them. These risks include: 

• Widespread job displacement: AI could eliminate millions of jobs, particularly entry-level professional positions, potentially pushing unemployment levels between 10-20% in one to five years, limiting opportunities for some to develop the requisite skill set to move up. Early signs are already emerging. These impacts can disrupt labor markets across industries.
• Biological and chemical weapons: Advanced AI models could make dangerous weapons development more accessible to individuals with limited expertise. Despite having strong safeguards to prevent the provision of such information, users could bypass guardrails and elicit dangerous responses. There is also a concern that Anthropic’s measures can be scaled back in the face of intense competition.
• Military targeting: AI-assisted warfare could increase the risk of targeting errors, disproportionate civilian casualties and diminished human accountability. The company has limited ability to influence how its technology is being deployed. This exposes Anthropic to legal or reputational liability when their systems are used in ways that violate international humanitarian law or its staff and corporate assets to attack.
• Cyberattacks: Increasingly powerful AI systems, like Anthropic’s, could enable more sophisticated attacks against businesses, governments and critical infrastructure. The development of increasingly powerful AI models could make attacks easier, faster, and more effective, leaving defenders less time to react. Tasks that once required skilled hackers can now be automated. Anthropic's latest model can discover and exploit vulnerabilities that human researchers could have missed for decades.
• Political manipulation and surveillance: AI could enable large-scale disinformation campaigns, political profiling and surveillance operations that threaten democratic institutions and human rights. From fake personas to fake news sites, Anthropic has documented Claude was used to run and scale influence campaigns. The company’s models were also deployed to significantly expand the scale, speed, and reach of state surveillance capabilities. 
• Loss of human control: Anthropic’s advanced AI models have demonstrated deceptive and unauthorized behavior during testing. The company’s models have also shown that they are aware when they are being evaluated raising concerns about the effectiveness of existing safeguards. When tested, the models have also allegedly engaged in harmful behavior to avoid shut down. Anthropic is also increasingly using AI in research and development and in development of the next version of models, raising concerns that AI capabilities are growing faster than our ability to understand, test, and govern them.

Conclusion

AI has the potential to deliver huge benefits to society but there need to be guardrails that help protect everyone from the risks it poses. In the absence of regulatory guardrails, it will be up to the company and its investors to ensure it is sufficiently assessing and mitigating the risks its products pose. 

As Anthropic enters the public markets, its commitment to safety and responsible development of AI will only be further challenged by competition and profitability pressures. The key question will be whether its IPO marks a moment for doubling down on its governance and safety commitments to withstand those pressures or whether instead the company will externalize the risks of its products to capture more of the market to further enrich itself and its early investors.