Using ChatGPT to research cyber threats has backfired on bad actors, OpenAI revealed in a report analyzing emerging trends in how AI is currently amplifying online security risks.
Not only do ChatGPT prompts expose what platforms bad actors are targeting—and in at least one case enabled OpenAI to link a covert influence campaign on X and Instagram for the first time—but they can also reveal new tools that threat actors are testing to evolve their deceptive activity online, OpenAI claimed.
OpenAI's report comes amid heightening scrutiny of its tools during a major election year where officials globally fear AI might be used to boost disinformation and propaganda like never before. Their report detailed 20 times OpenAI disrupted covert influence operations and deceptive networks attempting to use AI to sow discord or breach vulnerable systems.
"These cases allow us to begin identifying the most common ways in which threat actors use AI to attempt to increase their efficiency or productivity," OpenAI explained.
One case involved a "suspected China-based adversary" called SweetSpecter, which used ChatGPT prompts to attempt to engage both government and OpenAI employees with an unsuccessful spear phishing campaign.
In the email to OpenAI employees, SweetSpecter posed as a ChatGPT user troubleshooting an issue with the platform detailed in an attachment. Clicking on that attachment would have launched "Windows malware known as SugarGh0st RAT," OpenAI said, giving SweetSpecter "control over the compromised machine" and allowing them "to do things like execute arbitrary commands, take screenshots, and exfiltrate data." Fortunately for OpenAI, the company spam filter deterred the threat without any employees receiving the emails.
OpenAI believes that it uncovered SweetSpecter's first known attack on a US-based AI company after monitoring SweetSpecter's ChatGPT prompts boldly asking for help with the attack.
Prompts included asking for "themes that government department employees would find interesting" or "good names for attachments to avoid being blocked." SweetSpecter also asked ChatGPT about "vulnerabilities" in various apps and "for help finding ways to exploit infrastructure belonging to a prominent car manufacturer," OpenAI said.
Another case involved an adversary suspected to be affiliated with the Iranian armed forces called CyberAv3ngers, which is a "group known for its disruptive attacks" against public infrastructure in the US, Ireland, and Israel.
Monitoring their prompts—which were used for research, reconnaissance, and to debug code—helped OpenAI "identify additional technologies and software that they may seek to exploit" to disrupt water, energy, and manufacturing systems.
And OpenAI reported similar findings after disrupting an Iranian threat actor, STORM-0817, flagging the group's apparent first time using AI models and uncovering "unique insights" into "infrastructure and capabilities that were being developed and weren’t yet fully operational." For example, after reading ChatGPT prompts showing STORM-0817 researching how to "debug code to scrape Instagram profiles," OpenAI confirmed that the group appeared to be testing the code on an Iranian journalist "critical of the Iranian government."
OpenAI’s report appears to downplay AI harms
ChatGPT wasn't the only OpenAI tool abused by threat actors, OpenAI said. Its image generator DALL-E was also used by a suspected Russian-based operation called Stop News that was "unusually prolific in its use of imagery." Even relying on cartoonish images with "bright color palettes or dramatic tones to attract attention" didn't help that campaign go viral, though, OpenAI reported.
The campaign with the biggest reach, OpenAI noted, was curiously a hoax, where ChatGPT was initially used to generate social media posts on a Russian "troll" X account later taken over by a human attempting to deceive audiences into believing later posts were AI-generated.
"This was an unusual situation, and the reverse of the other cases discussed in this report," OpenAI said. "Rather than our models being used in an attempt to deceive people, likely non-AI activity was used to deceive people about the use of our models."
Despite detailing major threat actors already experimenting with its tools, OpenAI's report perhaps predictably seems to downplay AI's capacity for harm in this moment. It repeatedly emphasized that OpenAI's models "did not appear to provide" any threat actors detected with "novel capabilities or directions that they could not otherwise have obtained from multiple publicly available resources."
Instead of radically altering the threat landscape, OpenAI tools like ChatGPT are mostly used to take shortcuts or save costs, OpenAI suggested, like generating bios and social media posts to scale spam networks that might previously have "required a large team of trolls, with all the costs and leak risks associated with such an endeavor." And the more these operations rely on AI, OpenAI suggested, the easier they are to take down. As an example, OpenAI cited an election interference case this summer that was quickly "silenced" because of threat actors' over-reliance on OpenAI tools.
"This operation’s reliance on AI... made it unusually vulnerable to our disruption," OpenAI said. "Because it leveraged AI at so many links in the killchain, our takedown broke many links in the chain at once. After we disrupted this activity in early June, the social media accounts that we had identified as being part of this operation stopped posting" throughout the critical election periods.
OpenAI can’t stop AI threats on its own
So far, OpenAI said, there is no evidence that its tools are "leading to meaningful breakthroughs" in threat actors' "ability to create substantially new malware or build viral audiences."
While some of the deceptive campaigns managed to engage real people online, heightening risks, OpenAI said the impact was limited. For the most part, its tools "only offered limited, incremental capabilities that are already achievable with publicly available, non-AI powered tools."
As threat actors' AI use continues evolving, OpenAI promised to remain transparent about how its tools are used to amplify and aid deceptive campaigns online. But the AI company's report urged that collaboration will be necessary to build "robust, multi-layered defenses against state-linked cyber actors and covert influence operations that may attempt to use our models in furtherance of deceptive campaigns on social media and other Internet platforms."
Appropriate threat detection across the Internet "can also allow AI companies to identify previously unreported connections between apparently different sets of threat activity," OpenAI suggested.
"The unique insights that AI companies have into threat actors can help to strengthen the defenses of the broader information ecosystem, but cannot replace them. It is essential to see continued robust investment in detection and investigation capabilities across the Internet," OpenAI said.
As one example of potential AI progress disrupting cyber threats, OpenAI suggested that, "as our models become more advanced, we expect we will also be able to use ChatGPT to reverse engineer and analyze the malicious attachments sent to employees" in phishing campaigns like SweetSpecter's.
OpenAI did not respond to Ars' request for comment.
