Still, the findings spurred the researchers to think more deeply about how AI-as-a-service may play a role in phishing and spearphishing campaigns moving forward. OpenAI itself, for example, has long feared the potential for misuse of its own service or other similar ones. The researchers note that it and other scrupulous AI-as-a-service providers have clear codes of conduct, attempt to audit their platforms for potentially malicious activity, or even try to verify user identities to some degree. 

“Misuse of language models is an industry-wide issue that we take very seriously as part of our commitment to the safe and responsible deployment of AI,” OpenAI told WIRED in a statement. “We grant access to GPT-3 through our API, and we review every production use of GPT-3 before it goes live. We impose technical measures, such as rate limits, to reduce the likelihood and impact of malicious use by API users. Our active monitoring systems and audits are designed to surface potential evidence of misuse at the earliest possible stage, and we are continually working to improve the accuracy and effectiveness of our safety tools.”

OpenAI does its own studies on anti-abuse measures and the Government Technology Agency researchers notified the company about their work.

The researchers emphasize, though, that in practice there’s a tension between monitoring these services for potential abuse and conducting invasive surveillance on legitimate platform users. And what’s even more complicated is that not all AI-as-a-service providers care about reducing abusive uses of their platforms. Some may ultimately even cater to scammers.

“Really what surprised us was how easy it is to get access to these AI APIs,” Lim says. “Some like OpenAI are very strict and stringent, but other providers offer free trials, don’t verify your email address, don’t ask for a credit card. You could just keep using new free trials and churning out content. It’s a technically advanced resource that actors can get access to easily.” 

AI governance frameworks like those in development by the Singaporean government and European Union could aid businesses in addressing abuse, the researchers say. But they also focused a portion of their research on tools that could potentially detect synthetic or AI-generated phishing emails—a challenging topic that has also gained attention as deepfakes and AI-generated fake news proliferate. The researchers again used deep learning language models like OpenAI’s GPT-3 to develop a framework that can differentiate AI generated text from that composed by humans. The idea is to build mechanisms that can flag synthetic media in emails to make it easier to catch possible AI-generated phishing messages.

The researchers note, though, that as synthetic media is used for more and more legitimate functions, like customer service communications and marketing, it will be even more difficult to develop screening tools that flag only phishing messages.

“Phishing email detection is important, but also just generally be prepared for messages that are coming that may be extremely appealing and then also convincing,” Government Technology Agency cybersecurity specialist Glenice Tan says. “There’s still a role for security training. Be careful and remain skeptical. Unfortunately, those are still important things.”

And as Government Technology Agency researcher Timothy Lee puts it, the impressive human mimicry of AI-generated phishing emails means that for potential victims the challenge is still the same as the stakes grow ever higher.

“They still only need to get it right once, it doesn’t matter if you receive thousands of phishing messages written all different ways,” Lee says. “Just one that caught you off guard—and boom!”

More Great WIRED Stories