Anthropic bans users from 'cruel behaviour' towards AI chatbot Claude
Anthropic has updated its user policy to prohibit sustained and needless abusive behaviour towards its Claude AI models, citing uncertainty over whether advanced AI systems could have some form of moral status.
Anthropic has banned users from subjecting its AI models to "sustained and needless abusive or cruel behaviour", as the company continues to investigate whether artificial intelligence systems could have some form of consciousness.
The change, first reported by The Verge, was added to Anthropic's online user policy.
The company has not immediately clarified what specific behaviour could be considered abusive or cruel.
However, the policy says the restriction does not cover ordinary frustration with Claude, testing models or the use of "dark creative themes".
Anthropic's rules already contained restrictions on abusive conduct, but the updated language comes as the company increasingly examines the possibility that AI systems could have some form of welfare or moral status.
The San Francisco-based company introduced a feature last August allowing Claude to end conversations when users repeatedly behave in harmful ways.
At the time, Anthropic said the measure was intended partly as a safeguard for the potential welfare of its models.
The company said: "We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future."
It added that it was researching the issue and exploring "low-cost interventions" that could reduce potential risks to model welfare if AI systems were capable of experiencing something resembling distress.
The question of whether AI can be conscious remains deeply contested within the technology industry.
Anthropic chief executive Dario Amodei has said he cannot rule out the possibility that advanced AI systems could eventually possess some form of consciousness.
The company's position contrasts with that of OpenAI chief executive Sam Altman, who has expressed discomfort with people attributing human-like or religious significance to AI.
Altman previously wrote on X: "I am very uncomfortable about people trying to ascribe religious force or a surrender of human judgment to AI models."
The debate has intensified as AI companies develop increasingly capable systems that can hold lengthy conversations, use computers and operate with greater autonomy.
Anthropic has also previously said Claude can decide to leave interactions it considers potentially distressing, although the company stresses that it remains uncertain whether the models actually experience distress.
For now, the new policy appears to be less about establishing that Claude is conscious than about preparing for the possibility that AI welfare could eventually become a serious ethical issue.
Anthropic's approach could nevertheless prove controversial, particularly among users who view AI models as software rather than entities capable of being harmed.
The company will now face questions over how the new rule will be enforced, and where it draws the line between legitimate testing, creative experimentation and genuinely abusive behaviour.
