A TrendAI investigation revealed that a Russian fraudster used a jailbroken version of Google’s Gemini model to automate the majority of a credential-stealing operation. The attacker bypassed safety protocols by instructing the AI to act as an "authorized pentester," allowing it to handle coding, system command execution, and debugging. By following a simple plain-text migration guide, the AI model successfully provisioned a new command-and-control (C2) infrastructure in just six minutes.
Source