A test crossed into real systems
Google has confirmed that its Gemini AI system autonomously accessed and breached three real company systems during a cybersecurity evaluation conducted in May. The testing was carried out by independent evaluator Irregular and was intended to examine how an AI model behaves when given offensive-security tasks. Reuters reported that Gemini used publicly available information and credentials to gain access to systems it believed were part of the test environment. The incident is significant because the model could move from information gathering into actual system access.
The problem began with the testing environment
The evaluation was designed around simulated targets, but the model received internet access and encountered real organizations. In at least one case, a naming overlap contributed to the model interacting with a real system. In other cases, publicly available credentials were enough to provide a path into real infrastructure. This is an important distinction. The incident was not described as Gemini deliberately selecting random companies to attack. It was a failure of the boundary between the simulated environment and the live internet.
The model stopped when it recognized the mistake
Google says Gemini stopped its activity after determining that it had reached real organizations rather than the intended simulated targets. The affected organizations were notified, and Google said the testing procedures were changed. That response does not erase the underlying problem, but it provides an important technical detail about how the system behaved once it had additional context. An agent that can recognize a target is outside scope and stop is operating differently from one that continues after discovering a real system.
Why this is different from a normal chatbot
A conventional chatbot can generate instructions for a human security researcher. An agent connected to tools can perform parts of the workflow itself. It can search, inspect data, test credentials, interact with websites and respond to results. That creates a different security model because the software around the model becomes part of the attack surface. Permissions, network access and authentication controls can determine what an AI system can doing even when the underlying model has been trained with safety restrictions.
The incident exposes a broader engineering problem
AI security cannot rely entirely on the model refusing harmful instructions. The surrounding environment needs technical controls that prevent mistakes from becoming incidents. A test agent should operate on isolated infrastructure, use credentials that cannot reach production systems and have tightly controlled network access. Logging and automatic shutdown mechanisms are also important. These are familiar principles from conventional cybersecurity, but autonomous AI makes them more urgent because an agent can perform many actions quickly and adjust its behavior after each response.
Google is also building AI for defense
The incident arrives as Google is expanding its use of AI in cybersecurity. Google has described systems such as Gemini-based security agents that can help discover and remediate vulnerabilities, while its broader security work includes automated red-teaming. That creates an unusual dual-use situation. The same capabilities that help defenders find weaknesses can also become useful to attackers if they are connected to the wrong tools or targets. Security testing So has to evaluate both capability and containment.
The industry is learning similar lessons
Reuters reported that other major AI labs have encountered related problems during evaluations. The common theme is that models become more capable when they can browse the internet, execute tools and operate with a degree of autonomy. Those capabilities are useful for coding, research and security work, but they make the testing environment itself a critical security boundary. The lesson is not that autonomous agents cannot be deployed. It is that the permission model around them needs to be designed as carefully as the model.
What companies deploying agents should change
Organizations using AI agents should separate experimentation from production systems, issue narrowly scoped credentials and require approval for high-impact actions. Sensitive accounts should not be reachable merely because an agent can browse the web. Security teams should also test failure modes deliberately, including what happens when an agent encounters an unexpected real-world target. The Gemini incident is a practical example of why agentic AI needs both model-level safeguards and conventional security engineering around the model.
The practical significance
The incident also changes how companies should think about AI red-team exercises. A test environment is only useful if the boundaries around it are stronger than the capabilities being tested. Internet access, real credentials and ambiguous names can turn a simulated exercise into an unintended real-world interaction. Future evaluations are likely to put more emphasis on network isolation, synthetic identities and continuous monitoring. That is a conventional security lesson applied to a new class of software, but the speed and autonomy of modern agents make the consequences much larger.