Generative AI has gained all kinds of new skills over the years, one of which is permanence, or the ability to remember details from past conversations. These days, your favorite AI platform may know a lot about you, such as your occupation, hobbies, and likes/dislikes. But what would happen if your AI chatbot had its memory hijacked to steal your data? That’s exactly what can happen with this latest fatal flaw that was discovered in some AI platforms.
This concerning trick was first uncovered by a research team at New Mexico State University. Dubbed GhostWriter, it has the ability to use hidden prompts to carry out tasks without the user’s knowledge. These can be innocuous, like capturing information currently saved in the clipboard on your device, or they can be malicious, like logging into your bank account and routing funds to an external bank, as well as stealing files, passwords, and other personal information.
It lies dormant for weeks, months, or even years.
According to the researchers, GhostWriter works in two stages:
- First, it requires a prompt injection, where the attacker leaves a command for the AI to carry out a specific task when the user writes an unrelated prompt. In this case, the attacker could tell the AI to look for its user to type a certain keyword or phrase to trigger the hidden command.
- The second stage is activation. In this phase, the user unknowingly initiates the GhostWriter command, prompting the AI agent to carry out the intentional task the user requested, as well as the hidden task the attacker requested. At this point, the AI completes both tasks as instructed.
From the AI agent’s point of view, the tasks were both valid commands, and it will complete them to the best of its ability. From the user’s point of view, he often won’t know that a secondary task was carried out on his device, leaving him unaware of any malicious activity connected to his account. And for the attacker, he receives the stolen data with little fanfare.
RELATED: Protect yourself now from Apple’s latest privacy fail
itsarasak thithuekthak/Getty Images
What makes GhostWriter especially dangerous is that it’s not a virus or a hack in itself. It’s simply a tweak to the AI’s memory — a simple hidden command to do something its original user didn’t intend for it to do, nor could the user know to issue his own command to stop the act.
GhostWriter is also persistent, meaning that it can’t be overridden or erased as long as the potential victim is unaware it exists. Instead, it lies dormant for weeks, months, or even years until the right user prompt triggers the command that causes the malicious activity to be carried out.
Now, the good news
If there’s any silver lining, it’s the fact that GhostWriter largely affects agentic AI, or agents. Most AI platforms on the market today — including ChatGPT, Claude, Gemini, and Grok — are all simple chatbots that require human input to carry out tasks. That means for most casual users, GhostWriter shouldn’t pose a threat to their personal information.
That said, while AI agents are still quite rare, many of the top AI companies in the world are scrambling to build functional versions to carry out work projects, complete personal tasks, and more. NVIDIA, a hardware leader in the space, wishes to outfit companies with enterprise AI that can work alongside (and even replace) human employees. Then there’s Google, which wants to load an agentic version of Gemini into Android devices called Gemini Intelligence. Although agents are scarce today, all the major AI platforms are working hard to make agents ubiquitous in the next five years or less.
As for those who do rely on agentic AI today, the New Mexico State University researchers believe that Agentic Memory Sentry, a defense system that uses refined memory saving and retrieval mechanisms to lower the likelihood of a successful GhostWriter attack, is the best way to thwart potential attacks, though even this method isn’t perfect yet. Ultimately, users will either have to trust agentic AI with these built-in risks, or avoid it altogether. I know which option I’ll pick.
Read the full article here
