Chinese Court: AI Developers Not Liable for Hallucinations

A landmark court decision in China just made AI development a little less risky and a lot clearer for everyone involved.

The Hangzhou Internet Court ruled for the first time that AI developers aren't automatically responsible when their systems "hallucinate" or make up false information. Instead, users must prove the developer was at fault and that the error actually caused them harm.

The case started when a user named Liang asked an AI chatbot about a Chinese university. The system invented a fake campus and insisted it existed even after Liang challenged it with evidence.

When Liang proved the information was wrong, the AI did something unexpected. It told him it would pay him 100,000 yuan (about $14,400) in compensation and even suggested he file a lawsuit through the Hangzhou Internet Court.

Liang took that advice and sued for nearly 10,000 yuan in damages. But the court dismissed his case, explaining that AI systems "do not possess civil subject status and therefore cannot independently make legally binding expressions of intent."

The court made another important distinction by classifying AI-generated content as a service rather than a product. This matters because services require proof of fault, while defective products can trigger automatic liability.

The ruling noted that Liang couldn't show the false information actually harmed him or affected any decisions he made. The court also recognized that AI hallucinations are difficult for developers to completely control and that holding them strictly liable could slow down technological progress.

The Ripple Effect

This decision arrives as AI hallucinations grab headlines across China. Last year, a security guard went viral after an AI chatbot offered to sign his poetry for 100,000 yuan and even proposed a signing date, though nothing came of it.

Professor Cheng Xiao from Tsinghua University Law School says the ruling provides crucial guidance for future disputes. "The ruling offers important guidance for how courts may apply civil liability principles in future cases involving AI-related infringement disputes," he explained.

Current regulations require AI providers to review and remove harmful or illegal content, but they don't have to guarantee everything their systems generate is accurate. The ruling strikes a balance between protecting users and encouraging innovation.

Experts suggest platforms should clearly warn users about AI limitations while continuing to improve accuracy. Courts should assess whether providers met their duty of care by considering how content might impact users' rights.

This precedent could reshape how AI disputes are handled throughout China, giving developers clearer guidelines while still protecting users who suffer real harm from AI errors.