Meta's Oversight Board, which acts as a semi-independent policy council, now has a more pointed focus: how its social platforms treat explicit, AI-generated images. Tuesday, it said it is opening investigations into two separate cases over how Instagram in India and Facebook in the U.S. dealt with AI-generated images of public figures in the wake of a particularly egregious failure by Meta's systems to catch and respond to explicit content.

In both cases, the sites have scrubbed the media now. The board isn't identifying the people targeted by the AI images "to avoid gender-based harassment," according to an email Meta sent to TechCrunch.

The board takes cases about Meta's moderation decisions. Users have to appeal to Meta first about a moderation move before approaching the Oversight Board. The board is due to publish its full findings and conclusions in the future.
 
The cases
In the first case, the board said a user reported an AI-generated nude of a public figure from India as pornography on Instagram. The account posting the image posts images of only Indian women created by AI, and most of the users who react to these images are from India.

Meta did not take down the image after the initial report, and the ticket for the report automatically resolved 48 hours after the company made no further investigation into the report. When the original complainant appealed the decision, Meta automatically closed the report once more with no review whatsoever. That means that following two reports, the explicit AI-generated image remained on Instagram.

It was thus that the user appealed to the board. Only after that did the company act by deleting the offending content and deactivating the image for violating its community standards on bullying and harassment.

Then there was Facebook: the other user had posted an explicit, AI-generated photo that seemed to depict an American public figure in the AI creations group. On the social network, it removed the photo, however, as it was earlier posted by another user. Meta included it in a Media Matching Service Bank, now in the category of "derogatory sexualized photoshop or drawings.".

The board said it selects cases that are "analogues" of broader issues across Meta's platforms. Such cases can enable the advisory board to see the global effectiveness of Meta's policy and processes on various topics, as indicated by the company in its response to a query from TechCrunch on why the board chose a case where the company had taken down an explicit AI-generated image.

The Oversight Board said it is considering one case from the US and another from India to see if Meta is protecting all women globally equitably. "We know that Meta is faster and more efficient at moderating content in some markets and languages than others," co-chair Helle Thorning-Schmidt said in a statement.

"The Board believes it's worth investigating whether Meta's policies and enforcement practices are adequate to counter this issue."

Problems of deepfake porn and online gender-based violence
Some  generative AI tools in the last few years have grown to enable the generation of pornography. As TechCrunch has previously reported, groups such as Unstable Diffusion are making money off of AI porn that blur lines of ethics and engage in bias around data.

In the regions of India, deepfakes are also a cause of concern. Last year, the BBC report said that the videos of Indian actresses created using deepfakes have seen tremendous growth in recent times. According to data, women are the common subject for deepfaked videos. Earlier this year, Rajeev Chandrasekhar, a Deputy IT Minister, voiced his dissatisfaction regarding the approach being adopted by the tech giants to fight against the deepfakes.

"If a platform feels they can get away without removing deepfake videos, or just thinks they can take a lax attitude toward it, we have the power to protect our citizens by blocking such platforms," Chandrasekhar had said at that time in a press conference.

Though India has considered putting specific deepfake-related rules within the law, nothing is in place yet.

While the country has provisions about reporting online gender-based violence under law, experts note that it could be very tedious and that there is often little support. As a study, last year, noted from IT for Change-an Indian advocacy group-courts in India need to have robust processes to address online gender-based violence and not trivialise these cases.

According to Aparajita Bharti, co-founder of The Quantum Hub-an India-based public policy consulting firm-"There need to be some limitations for AI models such that they do not generate explicit content that harms.".

"It's the volume of that type of content that would increase because it can be produced with such a high sophistication degree easily. So, first off, we need to prevent it from being produced by training AI models to cap output once the intent to harm someone is already there, Bharti said during an email conversation with TechCrunch. And now we need to introduce default labeling also to be easy to catch on.

According to Devika Malik, who has worked for Meta's South Asia policy team in the position of a policy expert for platforms, while social networks have policies against non-consensual intimate imagery, it is mostly their users who enforce them through reporting:.

This puts a burden on the wronged user to prove their identity and lack of consent (as in Meta's policy case). This gets more error-prone when it comes to synthetic media and to say, the time taken to capture and verify these external signals enables the content to gain malicious traction, Malik said.

However, as of now, there are not many laws in the world that target the production and distribution of porn produced with AI tools. Some U.S. states have laws against deepfakes. The U.K. is introducing this week a law that criminalizes the creation of sexually explicit AI-powered imagery.
 Meta's response and next steps
Meta says it removed both posts, but the Oversight Board did not ask whether the social media company did not act on Instagram after users first reported the content or how long the posts were up on the platform before being removed.

Meta said it employs a combination of AI and human review to flag sexually suggestive content, and the company said it does not recommend that kind of content in, say, Instagram Explore or Reels recommendations.

The Oversight Board invited public comments-on the matter that refers to harms by deepfake porn, contextual information about the diffusion of such content in regions like the U.S. and India, and possible pitfalls of Meta's approach in detecting AI-generated explicit imagery-with a deadline of April 30.

The board will investigate the cases and public comments and post the decision on the site in a couple of weeks.

These cases reveal that though the big platforms are struggling to come to terms with older moderation processes, AI has made it possible for the user to create and distribute content much faster and with much less effort. For instance, companies like Meta have been trying new tools that use AI for content generation while trying some measures to detect the same. In April, the company announced that it would apply "Made with AI" badges to deepfakes if it could detect the content using "industry standard AI image indicators" or user disclosures.

Platform policy expert Malik said labeling is often inefficient because the system to detect AI-generated imagery is still not reliable.

"Labelling has proven to be very little deterrent in the potential limitation of distribution of harmful content. As we can recall, when millions of users were diverted by X through its own trending topic 'Taylor Swift AI, about AI-generated images of Taylor Swift, people and the platform were well aware that the content was not original," noted Malik. 

However, perpetrators always come up with ways to evade these detection systems and post problematic content on social platforms.