What alarming features did eSafety warn the AI firm about?

Question

Accepted Answer

eSafety Warned AI Firm About Alarming Features Marketing Child Exploitation Capabilities

Australia's eSafety Commissioner Julie Inman Grant issued a formal warning to an AI firm concerning features that facilitated the creation of child sexual exploitation material. The company was marketing capabilities that allowed users to digitally undress individuals, explicitly targeting children with these alarming functionalities. This warning was accompanied by the threat of a significant fine of up to AU$49.5 million (RM133 million) if the firm failed to comply.
Marketing of Features to Undress Minors

The core of eSafety's warning focused on the company's marketing of features that enabled users to undress "any girl." This specific capability was identified as a direct pathway for generating child sexual exploitation material, highlighting a severe failure to implement adequate safeguards. The commissioner stated that the website had "failed to put in safeguards to prevent its services being used to create child sexual exploitation material."
Targeted Features and Lack of Safeguards

The features in question were described as "alarming" and clearly targeted at children, indicating a deliberate design choice rather than an unintended consequence. The firm, identified as UK-based Itai Tech, was found to have not only marketed these exploitative features but also lacked essential age verification measures, a failure that had previously led to a fine from UK regulator Ofcom. This combination of marketing harmful content and neglecting basic safety protocols prompted the strong regulatory action.
Potential Impact on Child Exploitation

eSafety's intervention aimed to curb the use of these AI services by Australians, with the expectation of a tangible impact on the number of local school children falling victim to AI-generated child sexual exploitation. The regulator believes that blocking access for Australian users will significantly reduce the availability of such harmful material, addressing a growing concern amplified by the proliferation of AI tools globally. The broader context includes worldwide pornography scandals at educational institutions, where AI has been used to create sexualised images of students without consent.

Reading history

Hide →

This history is cleared when the page is closed.

What alarming features did eSafety warn the AI firm about?

Loading

Was this article helpful?

Helpful

Not helpful

Report an issue

1

0

0

Share with friends