When a deepfake image becomes real harm: Opinion

Rita’s story is a composite of many who have had to endure such harrowing experiences. According to UNICEF, at least 1.2 million children globally have had their images manipulated into sexually explicit deepfakes within a single year. In some jurisdictions, that is one in 25 children.

While Singapore-specific prevalence data remains limited, the increasing accessibility of “nudifier” tools – artificial intelligence applications that generate fabricated nude or sexualised images from ordinary photographs – and the global scale of such abuse suggest that no digitally connected society is insulated from these risks.

Even where laws are enforced and platforms respond promptly, removal takes time. During that interval, images may be copied, modified and redistributed across different channels, with a single file giving rise to multiple versions requiring separate attention.

For victims, the most acute harm arises when they sense a loss of control once the image circulates, multiplied with every screenshot duplicate.

Legal remedies exist, but it is striking that the same platforms capable of making content go viral within seconds still struggle to remove it at comparable speed.

Legal systems are often described as struggling to keep pace with technology. But the deeper problem may be that online platforms have not done enough. The argument that removal is technically difficult sits uneasily alongside the sophistication of the systems already used to maximise visibility and engagement.

Legal avenues

Existing provisions in the Penal Code already address sexually explicit deepfakes where an image or recording has been altered to depict a victim. These provisions cover, among other things, the possession or gaining of access to such material, as well as its distribution or threatened distribution without consent, subject to the statutory elements being satisfied.

The Criminal Law (Miscellaneous Amendments) Act 2025 extends the definition of an “intimate image or recording” to include generated images or recordings, and introduces an express offence of intentionally producing such material without consent. A person who uses AI to generate a fabricated nude image of a classmate from an ordinary social media photograph, and then circulates it within a chat group, may now fall within the amended framework.

These developments are significant because modern deepfake abuse often does not involve authentic intimate photographs at all. AI systems are now capable of generating realistic sexualised images from ordinary photographs obtained through social media accounts, school profiles or messaging applications. 

In a recent Singapore case, a man allegedly used a woman’s photograph to generate an AI-created nude image resembling her before later installing hidden cameras at her workplace to facilitate voyeuristic recordings. The law therefore increasingly focuses not on whether the image was originally real, but on the non-consensual sexualisation of a person’s likeness.

Separately, Singapore’s new Online Safety Commission will begin operations on June 29 under the Online Safety (Relief and Accountability) Act 2025, giving victims a dedicated avenue to seek rapid relief against specified online harms and imposing clearer obligations on platforms to respond to harmful content.

In most cases, harms must first be reported to the online service provider. But for urgent categories such as intimate image abuse and child abuse material, victims may seek relief directly from the commission, which may issue directions requiring harmful content to be taken down, access to be disabled or accounts to be restricted.

Platforms the laggards

With online platforms, the technical infrastructure already exists. Content matching technology, hash-matching databases for known child sexual abuse material and AI-powered moderation systems operate at scale.

Singapore’s new framework increasingly reflects this reality by placing clearer obligations on platforms to respond to harmful content. Yet responses to synthetic intimate imagery have often remained slow and inconsistent.

A 2024 audit study of X found that AI-generated intimate images reported through copyright channels were removed within about a day, while reports submitted as non-consensual nudity remained online for more than three weeks. This suggests that existing moderation systems remain poorly calibrated to respond quickly to synthetic sexual abuse.

A 2025 study by the Infocomm Media Development Authority found that platforms take about five days or more to act on valid reports of online harm.

Even where action is taken, it has often followed public pressure rather than as standard practice. Requests involving synthetic intimate imagery may still be processed through the same broad reporting systems used for other forms of unwanted content. 

The deeper difficulty is structural. Engagement-maximising algorithms do not distinguish between content that spreads because it is entertaining and content that spreads because it is harmful. Viral circulation is often rewarded, while rapid containment remains secondary.

Industry responses have also been uneven. Major technology companies including Meta, Google and Microsoft have announced voluntary commitments relating to AI-generated abuse and synthetic content. But such commitments do not necessarily create binding timelines, uniform removal standards or clear accountability when enforcement is delayed or inconsistent.

Other jurisdictions have moved more aggressively. The European Union’s Digital Services Act and the UK’s Online Safety Act 2023 impose stronger duties on large online platforms to address systemic online risks.

Singapore’s legal framework is moving in a similar direction through the Online Safety (Relief and Accountability) Act 2025, and the new Online Safety Commission. But its effectiveness will ultimately depend on whether platforms treat rapid containment as a core operational responsibility rather than a secondary moderation issue.

An equally important issue is whether perpetrators can be identified quickly enough to create meaningful deterrence. Recent reviews and consultation papers underpinning Singapore’s online safety reforms recognised that anonymity must be addressed, particularly in the cases involving AI-generated abuse.

Offline containment

While online platforms remain central to containment, much still depends on what happens in the first hours after an incident emerges. The difference between early intervention and delayed response may determine whether an image remains confined to a small group or spreads across multiple platforms.

Schools, families and online services are often the first points of contact. Clear reporting pathways, prompt escalation, and rapid moderation can significantly reduce harm before formal legal processes run their course.

Regulatory mechanisms will complement such efforts through structured avenues for intervention. But their effectiveness will depend not only on legal powers, but also on how seamlessly they integrate with early-stage responses across the broader ecosystem.

The emergence of “nudifier” tools has dramatically lowered the barrier to creating harmful content. What may appear to some as experimentation, online humour or technological curiosity can instead inflict serious and lasting harm.

Families and schools therefore play a defining role. Responses to such incidents do more than address individual cases. They shape  expectations about what conduct is acceptable.

Clear messaging that the non-consensual creation or sharing of sexualised images is not merely inappropriate but unacceptable helps establish boundaries that law alone cannot fully enforce. 

Over time, these norms become part of the wider environment in which technology is used, reducing the likelihood that harmful tools are treated casually and reinforcing the understanding that such conduct carries real consequences.

Need for urgency

For victims, the central question is more immediate: Are they protected when protection matters most?

AI has reduced the cost of producing harmful content, while increasing its realism and scale. Legal reforms are increasingly recognising the realities of synthetic abuse. 

If online services can optimise virality with extraordinary precision, they should also be expected to develop equally robust systems for rapid containment. The burden cannot rest solely on victims, families, schools or law enforcement agencies reacting after the damage has already multiplied across the digital environment. 

A fabricated image is capable of inflicting real injury. In an era where a person’s likeness can be manipulated within minutes using widely accessible AI tools, protection depends not only on punishing wrongdoing after the fact, but on stopping harmful content before it becomes impossible to retrieve. 

Ben Chester Cheong is a law lecturer and MOE START scholar at the Singapore University of Social Sciences. He is also a counsel at a Singapore law firm and an associate academic fellow at the NUS Asia-Pacific Centre for Environmental Law.

Source: The Straits Times © SPH Media Limited. Permission required for reproduction.

AI will change only part of your job. That is exactly why it matters: Opinion S’porean man linked to global insider trading scheme has appeal for bail denied by High Court