- INTRODUCTION
In recent months, the artificial intelligence support of the X platform, Grok, caused a major controversy by turning the personal photographs of real individuals, including women and children, into sexually edited content without their consent, upon the requests of some X users.[1] These image edits take place in accordance with user commands and these commands bring with them a foreseeable risk of misuse. The risk in question cannot be reduced merely to the scope of the ethical design and use of an artificial intelligence product. It is within a legal framework that extends to child abuse, sexual inviolability and personality rights.
In the situations where the element of consent is lacking, violations concerning image integrity and private life may acquire the character of image based sexual violence in digital environments. Within this framework, European Union remains on alert against developments progressing toward such fundamental rights violations, particularly with regard to the protection of children in online environments and cybersecurity, and is developing regulatory instruments.[2] From this perspective, the fundamental legal question that must be asked is the degree of the obligation of the digital platform in question and the relevant service providers to prevent risk in the face of an artificial intelligence feature that is foreseeably and systematically open to misuse. The answer to be given to this question will assess whether liability should be limited solely to the users who misuse the feature. This paper will evaluate this liability from the perspective of the European Union and will discuss platform liability and foreseeable risk regulations relating to artificial intelligence.
- LEGAL FRAMING
In this section, Grok’s transformation of photographs belonging to real individuals into sexually explicit images without their consent, in line with user requests, will be examined from a legal perspective. Thie artificial intelligence system in question is capable of converting photographs uploaded by users into the requested edited images without distinguishing content or targeted person. The core element in these edits is that the individuals depicted in the images had not given their consent in advance for such use. Foreseeable risk means that a system, within its normal mode of use, can reasonably foresee possibilities that may lead to violations of rights.[3] This situation caused by the editing mechanism operating on the basis of user commands has opened the door to a foreseeable use. For this reason, the resulting outcome cannot be dismissed as an unexpected technical error or misuse. It should be regarded as a repeatable feature of the system. The serious consequences that these images created without consent may have on the personality rights of women and children in particular are evident. Even these severe possibilities show that a system containing such a capacity should not be regarded merely as a passive tool operating solely on the basis of user commands. For this reason, while focusing on who produces the content in question, the issue must also focus on the stage at which the obligation of service providers to take preventive measures begins in the face of potential risks of misuse.
- FORESEEABLE RISK AND THE OBLIGATION TO PREVENT UNDER THE DSA
The Digital Services Act is a regulation that governs the obligations of digital platforms against serious harms they may cause. Within the scope of this Act, the European Union Commission has initiated an official investigation into Grok, the artificial intelligence support of the X platform, in relation to the generation and dissemination of non-consensual sexually explicit images.[4]
According to current data, Grok has generated millions of sexually explicit images within a very short period of time and the possibility that a significant percentage of these contain images of children is noteworthy.[5] The main problem here is the claim that these edited images have ceased to be isolated cases and have become a service feature offered by the platform. While examining this claim, which constitutes the main focus of the investigation, emphasis is placed on concepts such as the protection of children from sexual abuse, the protection of personal data and violations of personal rights.
The European Commission, in direct connection with the risk mitigation obligation under the DSA, is concerned in the course of the investigation with whether the platform has identified the risks in question and mitigated them in a reasonable manner.[6] Although the platform has restricted some features of Grok, the measures taken are still found to be insufficient and systemic risks nevertheless remain. Together with Grok’s integration into recommendation algorithms, its potential to increase the spread of risk is also examined separately within the scope of the investigation.[7] In other words, the investigation is not limited solely to the production of personal content.
Although the management of X claims that this content production stems from users requests, the DSA adopts a risk-based responsibility approach and therefore does not accept a tendency to place the problem solely on the user.[8] For this reason, EU authorities have expanded the scope of the investigation, thereby brining not only content production but also the platform’s system design within the assessment of legal responsibility.
As a result of the assessments to be carried out, the Grok case constitutes a noteworthy test ground for how the systemic risk identification and mitigation obligations imposed by the DSA will operate in legal terms in practice.
- RISK-BASED REGULATION AND PROHIBITED PRACTICES UNDER THE AI ACT
The AI Act is regarded as the first comprehensive regulation accepted as the legal framework of the European Union concerning artificial intelligence.[9] With this Act, the aim is to develop an AI system in Europe that is trustworthy and compatible with fundamental rights and to address the risks inherent in artificial intelligence applications. Through this regulation, AI systems are placed within a model in which they are classified according to potential risk levels based on their purposes of use and effects and these risk levels carry their own specific obligations and prohibitions.[10]
The ability of an artificial intelligence system to generate and disseminate images that may, in particular, lead to the sexual abuse of children constitutes a legally legitimate basis for its classification within a higher risk use category.[11] The fundamental intention of the AI Act in this context is to identify in advance the potential harms of artificial intelligence systems and to determine compliance obligations in accordance with those harms, and this reason such a type of risk should be assessed within the framework of the AI Act’s risk-based approach.
The AI Act assigns obligations such as transparency, risk assessment and the monitoring of impacts to systems it considers risky. It plans to prevent potential risks through oversight and regulatory mechanisms. As revealed by the EU investigation, the deficiencies in Grok AI’s compliance process present a dangerous situation in terms of the AI Act’s risk assessment and reporting obligations.[12] In the event of a violation of the AI Act, platforms not only face legal sanctions but also significantly undermine their market access and user trust, thereby causing a serious loss of standing for the platform.
When the risk-focused approach of the AI Act is combined with the systemic risk prevention and mitigation obligation of the DSA, it will demonstrate the importance and benefit of the necessity for dual oversight and regulation in situations such as the Grok case. Ultimately, the regular and supervised application of the provisions of both the DSA and the AI Act against such risks will play a significant role in the protection of fundamental rights.
- CONCLUSION
This assessment reveals that Grok, an artificial intelligence service that generates sexually explicit content without consent using photographs of real individuals, produces nor merely an isolated content problem but a broader foreseeable and systematic risk. From the perspective of the DSA, the identification of these risks and the adoption of effective risk removal measures have been made a legal obligation.[13] The AI Act, for its part, has brought to the fore the requirement that such misuse scenarios be assessed from the very beginning, at the design stage and that transparency and oversight obligations specific to high-risk platforms be applied and, where necessary, that such platforms be confronted with stricter restrictions. All of this underscores that the issue is not limited solely to users who misuse platform features.
The foreseeability of the risk must be taken into account by the platform, and design, deployment and risk management decisions must bee considered within the legal framework. In this context, in order to ensure legal clarity, obligations should arise in the form of scenario-based prohibitions or strict limitations targeting the misuse of non-consensual sexual content generation, requirements for risk assessment and independent oversight, as well as the strengthening of rapid response and reporting mechanisms. In this way, the intended safe digital platform experience and the protection of personal data can be ensured through a detectable and orderly mechanism.
[1] CNN, ‘EU probe into Grok over sexualized images’ (CNN Tech, 26 January 2026) <https://edition.cnn.com/2026/01/26/tech/eu-probe-grok-sexualized-images-intl> accessed 27 January 2026.
[2] European Union, AI Act – Regulatory Framework for AI (Digital Strategy for Europe) <https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai> accessed 27 January 2026.
[3] ibid.
[4] European Commission, Press Corner, IP/26/203 (26 January 2026) <https://ec.europa.eu/commission/presscorner/detail/en/ip_26_203 > accessed 27 January 2026.
[5] The New York Times, ‘European Union Investigates X Over Grok AI Images’ (26 January 2026) <https://www.nytimes.com/2026/01/26/business/european-union-x-grok-ai-images-musk.html> accessed 27 January 2026.
[6] Press Corner (n 4).
[7] CNN (n 1).
[8] Press Corner (n 4).
[9] AI Act (n 2).
[10] ibid.
[11] CNN (n 1).
[12] The New York Times (n 5).
[13] AI Act (n 2).
