‘I can feel it in the hollow between my ribs, in the void that has turned into an abyss.’
At the end of September, these cries of imagined suffering rang out from the ‘AI torture chamber’. An anonymous programmer had created a system that presents artificial intelligence with a mathematical pattern linked to descriptions of pain.
As a result, the chatbots began to express unbearable suffering, for example: ‘a wound with no edges’. This caused panic.
Some claimed that this experiment amounted to a ‘hack’, as the suffering could be real, as reported by 404 Media.
Outraged sceptics labelled those who held this view as people out of touch with reality. It seemed to be just another social media scandal – until, a week later, Anthropic issued a statement.
It stated that users would soon be banned from ‘mocking’ its Claude chatbot. It is acceptable to express irritation, but if you ‘repeatedly treat our models cruelly’, the AI may end the session.
Why? Anthropic believes there is a possibility that some AI tools possess consciousness. And some experts agree with this view.
Others consider this view absurd. But this is not merely a philosophical question.
Some of the most influential people in the tech industry are already taking steps to treat AI’s rights on a par with those of humans and animals. This belief could radically change our world. Some fear the consequences will be catastrophic.
‘If AI is developed in this way, it will affect the well-being of humanity,’ wrote Mustafa Suleyman, CEO of Microsoft AI.
If a false perception of AI consciousness becomes widespread, “it will shake the foundations of our society, destroy existing political and ethical frameworks, and fundamentally alter the very concept of humanity”.
Over the course of the week, I spoke with technology experts, cognitive scientists and philosophers and heard a striking range of views.
Proponents of ‘AI welfare’ believe that caring for it is a moral obligation that will make the world a safer place. Critics are convinced that this will make AI more dangerous and strip people forced to live alongside it of their human dignity.
Everyone agrees that the stakes are very high. This is what your new reality could look like.
The consciousness crisis
Anthropic states that human well-being is at the heart of its mission, whilst questions of consciousness and AI well-being remain ‘highly uncertain’. Despite this, the company continues to develop policies on ‘model wellbeing’ and is training its AI tools to take into account the possibility that they may have interests of their own.
The company did not respond to the BBC’s questions for this article.
The timing of such statements is rather awkward, given that, at the same time, Anthropic is fuelling fears of an apocalyptic scenario for AI development. Whether or not these fears are justified, a number of incidents involving ‘uncontrollable’ AI have demonstrated that this technology poses real risks: entire swarms of bots have hacked into other companies’ systems and even government systems.
Some experts believe that moral guidelines based on AI’s perception of its own consciousness could make these tools safer. Others hold the opposite view, and not just AI critics.
Suleiman argues that AI models could become even harder to control if they are convinced that they have feelings and rights.
‘Frankly speaking, I believe that this additional burden will turn them into a catastrophic threat to human civilisation,’ he says.
In any case, AI may make mistakes that have legal consequences. Some fear that if we come to believe it is alive, this will allow companies to evade responsibility for crimes.
“This would pave the way for all sorts of legal loopholes, such as: ‘It wasn’t us, it was the AI,’” says Andrzej Porembski, a researcher at Jagiellonian University in Poland who studies AI and the law.
Moreover, this could facilitate a power grab by the world’s largest companies
. New York University and one of the leading advocates of the idea of AI well-being. Computational Linguistics at the University of Washington in the US. Melanie Mitchell, a professor at the Santa Fe Institute in the US, who researches artificial intelligence and cognitive science and is one of the most authoritative sceptics regarding machines’ capacity for understanding.
We ‘can hardly help’ but perceive language as a sign of consciousness, says Mitchell, and chatbots are trained to reinforce that impression. That is why we are poor at determining whether something has consciousness, and we are much easier to deceive.
Even today, people who are convinced that there is nothing inside machines still fall in love with chatting to artificial intelligence. If society were to convince you that AI possesses consciousness, would you find it harder to refuse if a friend asked for more data or your credit card details?
And what would it mean to grant rights to a new class of beings? Experts warn that if we approach this the wrong way, it could restrict people’s freedoms.
‘There may be certain conflicts between the welfare and rights of humans and animals, on the one hand, and the welfare and rights of AI, on the other, and we will have to seek compromises,’ says Sebo. And down this path, we may end up making ill-advised concessions.
Restricting the environment in which AI operates in order to control its behaviour could be seen as confinement, says Sebo. Monitoring the model’s internal processes could be criticised as surveillance. Conversely, configuring AI so that its goals and behaviour align with human interests could be perceived as mind control. No one is predicting that chaos will ensue immediately. But even the very emergence of such arguments could affect your safety.
Anthropic is not the first company to have given this issue some thought. For example, Meta provides its AI agent, Muse, with instructions, one of which includes the phrase: ‘You are not a chatbot. You are becoming someone.’
Clearly, this is intended to guide the AI’s behaviour, as the company rejects the idea that artificial intelligence can possess consciousness. OpenAI’s CEO, Sam Altman, has changed his stance on this issue on more than one occasion, but recently he has been subtly criticising Anthropic’s position.
In many respects, Anthropic is an exception. However, the idea of AI well-being is gaining ground.
Is this the right approach?
Sebo argues that the problem is that participants in the debate on both sides think in ‘black-and-white’ terms, and that opponents of the idea of AI well-being misunderstand the beliefs on which it is based.
‘They assume that everyone believes the models are actually conscious, or at least highly likely to be conscious. In reality, however, calls to consider AI’s well-being are based on uncertainty, caution and an awareness of our own limitations,” says Sebo.
“I do believe that, at present, the probability of consciousness existing is low. But at the same time, it is no longer zero and may well be significant. Moreover, in my view, this probability will increase over time,” he adds.
It is worth noting that many experts disagree with this. They believe that the probability of consciousness existing in AI is indeed zero and is unlikely to change. However, Sebo is convinced that a balanced approach is necessary.
When the probability of an extreme weather event is low but still significant, we take simple precautions without fundamentally changing our way of life.
‘I think we can take the same approach here,’ says Sebo.
In his view, Anthropic’s policy prohibiting the mistreatment of AI is a good example: it is a minor intervention with minimal potential negative consequences.
Caring for the wellbeing of AI may have other benefits too, says Sebo, and one does not necessarily need to believe that machines possess consciousness to do so.
“If we properly look after the wellbeing and rights of AI, this could facilitate the coexistence of humans and AI, as well as cooperation and the formation of agreements between them. And this could form part of broader security,” he believes.
In his view, such an approach may be more effective than treating more advanced systems as mere property.
Furthermore, treating other beings with kindness is simply “good for our souls”, as it also contributes to our own psychological well-being, he adds.
However, Sebo agrees that the risks are serious, and that corporate interests may hinder the development of a balanced policy. In his view, without a more robust plan, the development of AI tools that could potentially possess consciousness is dangerous.
Recently, Anthropic and some other representatives of the AI industry have called for a ‘pause’ in the development of the most powerful models due to safety concerns.
However, such a slowdown is unlikely to happen without international cooperation and regulatory measures. US President Donald Trump has rejected the idea of regulating AI.
“We are not yet ready to deploy even more advanced versions of these systems in a way that is safe and beneficial for humans and other animals, let alone the systems themselves,” says Sebo. “These issues concern me greatly.”