AI/LLM poisoning

Summary

As AI services become increasingly common and available to the average user through AI-powered user support chatbots, AI-summarised content, and widespread LLM use, the risk of data feeds being poisoned to include malicious content will continue to rise. Attack scenarios include:

• LLM responses including links to phishing pages and other malicious content
• AI summarised content containing phishing links

As AI-assisted (and autonomous) web browsing becomes more common, the risk of an automated service accessing malicious content and entering sensitive information (e.g. credentials, banking information) or downloading malware will increase further.

Examples

• Example 1: Poisoning “summarize this email” in Gemini for Workspace — Security researchers revealed that attackers could embed hidden HTML/CSS prompts in emails that Gemini faithfully executes, generating fake security alerts (e.g., “Your GMail password has been compromised. Call 1‑800‑555‑1212”) directly in the summary pane.
• Example 2: Netcraft’s GPT‑4.1 Misleads Users with Fake Login URLs — Netcraft found that ⅓ of login links generated by GPT-4.1 for major brands were inaccurate, with some pointing to phishing sites.

Further reading

• How attackers can use Computer-Using Agents to automate identity attacks
• How attackers can exploit system prompt leaks with phishing attacks targeting AI assistants
• Poisoning plugins used by LLMs to launch phishing attacks