LockInCS

Topic 1.4 · Unit 1: Introduction to Security · 15 minute session

1.4 AI-Based Cybersecurity Attacks

Adversaries use artificial intelligence (AI) tools to impersonate people with cloned voices and avatars, write fluent phishing in any language, extract sensitive data from large language models (LLMs), plant false information in training data, research targets and write malware. Defenses include shared secrets, multifactor authentication (MFA), keeping personal data out of AI tools, and verifying AI output with reputable sources.

Big idea AI makes old attacks faster, cheaper and more convincing.

Course and exam description (CED) codes covered: 1.4.A.11.4.A.21.4.A.31.4.A.41.4.A.51.4.A.61.4.B.11.4.B.21.4.B.31.4.B.4

Last reviewed against the CED:

By the end of this session you can

  • 1.4.A Explain how adversaries use AI-powered tools to augment cyberattacks.
  • 1.4.B Explain how to protect against some AI-augmented cyberattacks.

Already know this? Jump to the checkpoint.

1. Six ways adversaries use AI (1.4.A)

AI-powered move What it enables
Cloned voice or digital avatar built from existing voice and image samples 1.4.A.1 Impersonating someone on a phone or video call, leading to financial loss or the sharing of sensitive or private information. The more organizations use voice-based authentication, the bigger the impact.
Generative AI writing phishing 1.4.A.2 Fluent messages in any language. Unnatural language used to help spot phishing; that clue is fading.
Prompts that extract data from an LLM 1.4.A.3 Sensitive information can leak from what users typed in or from the huge data sets used to train the model.
False information planted online 1.4.A.4 New or edited websites carry lies into LLM training sets, so the LLM repeats them.
AI reconnaissance 1.4.A.5 Tools scan social media and public websites to build a profile of a target fast.
AI-enhanced coding tools 1.4.A.6 Writing new malware, changing legitimate code to act maliciously, or finding vulnerabilities in large code bases.

Case file: the call from “home”

A student gets a video call. It looks and sounds like their older brother: “I lost my wallet, can you send me the code that just came to your phone?” The face moves naturally. The voice is right. Then the student asks for the family word. The call drops.

Attack chain in four steps: AI scrapes voice clips from social media, builds a digital avatar of the voice, calls a relative with an urgent fake story, causing financial loss or shared sensitive data. Defenses under the steps: check online contacts are who they claim, turn on MFA because a voice is not enough, and use a shared secret phrase to prove who is calling.
Anatomy of an AI voice-cloning scam, and where each defense stops it.

2. Protecting against AI-augmented attacks (1.4.B)

  • Set up shared secrets with close friends and family: a word or phrase only the two of you know, used to verify identity in high-stakes moments 1.4.B.1.
  • Enable multifactor authentication (MFA). If a cloned voice gets past voice authentication, a second factor can still block the login 1.4.B.2.
  • Keep personal or sensitive data out of AI tools such as chatbots and virtual assistants. Some tools train on what you type, and adversaries could pull it back out 1.4.B.3.
  • Evaluate AI output carefully. Verify it with reputable, stable sources that are not AI-based 1.4.B.4.

Prevent: agree on your shared secret in person, never in a text or chat that could be read later.

Lock it in: memory hooks

One picture per idea. If you remember the hook, you can rebuild the details.

  • AI attacksAI gives adversaries a better mask (voice, face), a better pen (phishing), a better shovel (data from LLMs) and a better megaphone (false info in training data).
  • Shared secretA family password beats a perfect voice: a cloned voice cannot know a word only two of you share.
  • AI outputTrust, then verify: check AI answers against reputable sources that are not AI.

Common misconceptions

  • "Phishing is easy to spot because of bad grammar."

    Not quite. Unnatural language used to be a useful clue. With generative AI, adversaries can write phishing that reads like a native speaker wrote it, in any language.

  • "If I recognize the voice on the phone, it is really them."

    Not quite. AI tools can clone a voice from existing samples. Use a shared secret word to verify identity in high-stakes situations.

  • "What I type into a chatbot stays private."

    Not quite. Some AI tools use your input to keep training the model, and adversaries can craft prompts to pull data back out. Keep personal data out.

Exam traps

  • A stronger password does not stop voice cloning. A shared secret or a second factor does.

By the numbers

Say each list out loud, then tap to check.

6 AI attacks Say them, then tap
  • voice and video cloning
  • AI-written phishing
  • extracting data from LLMs
  • poisoning training data
  • AI reconnaissance
  • AI-assisted coding
4 AI protections Say them, then tap
  • shared secret
  • MFA
  • no sensitive data in AI tools
  • verify AI output with non-AI sources

In 30 seconds

  1. Cloned voices and avatars let adversaries impersonate people on calls and video, a bigger threat as voice authentication spreads.
  2. Generative AI writes fluent phishing in any language, so poor grammar is no longer a reliable clue.
  3. Adversaries craft prompts to extract sensitive data from LLMs, publish false information so LLMs learn it, scan the internet for reconnaissance, and use AI coding tools to write malware or find vulnerabilities.
  4. Protect: shared secrets, MFA, no personal data in AI tools, and verify AI output with reputable non-AI sources.

Cover this list, say the points out loud, then take the checkpoint without scrolling back up.

Checkpoint · 8 questions

Checkpoint: AI-Based Cybersecurity Attacks

Pick an answer to mark it. Every answer is explained, and you can try any question again. Your saved score counts first tries only, and stays in this browser.

  1. 1 Why is "look for spelling and grammar mistakes" a weaker phishing check than it used to be? 1.4.A.2
    Show answer

    Answer: C. Large language models (LLMs) let adversaries write phishing that reads as though a native speaker wrote it, so unnatural language is a less reliable clue.

  2. 2 Why does voice cloning become more dangerous as more organizations adopt voice-based authentication? 1.4.A.1
    Show answer

    Answer: C. With voice-based authentication, a convincing cloned voice can impersonate the user to the system, increasing the potential impact.

  3. 3 An adversary publishes many web pages with false claims so that a chatbot later repeats them. What is the adversary targeting? 1.4.A.4
    Show answer

    Answer: D. Adversaries can publish or modify websites so false information enters LLM training sets, causing the LLM to repeat it.

  4. 4 A parent receives a call in their child's voice asking for money urgently. Which control best verifies the caller? 1.4.B.1
    Show answer

    Answer: D. A shared secret known only to the two people verifies identity in high-stakes situations. A cloned voice cannot know it.

  5. 5 Why should a student avoid pasting their medical form into a free chatbot? 1.4.B.3, 1.4.A.3
    Show answer

    Answer: A. Some AI-powered tools feed user input back into training, and adversaries can craft prompts that extract sensitive information from LLMs.

  6. 6 What is the recommended way to treat output from an AI-powered tool? 1.4.B.4
    Show answer

    Answer: C. AI output should be carefully evaluated and verified using reputable, stable, non-AI sources.

  7. 7 An adversary uses an AI tool that scans social media and public websites to build a profile of a target in minutes. Which attack activity is this? 1.4.A.5
    Show answer

    Answer: A. AI-powered tools can scan the internet for information posted on social media and public websites, speeding up reconnaissance on a target.

  8. 8 How can adversaries misuse AI-enhanced coding tools? 1.4.A.6
    Show answer

    Answer: B. AI-enhanced coding tools can help adversaries write new malware, modify existing code to act maliciously, or find vulnerabilities in large code bases.

Key terms: flip the cards

Read the term, say the definition out loud, then flip to check. Struggling a little to remember is what makes it stick.

  • Artificial intelligence (AI)tap to flip

    Computer systems that perform tasks such as generating text, recognizing patterns or sorting events, used by both adversaries and defenders.

    Example and CED codes
  • Large language model (LLM)tap to flip

    A generative AI model trained on huge text data sets that writes human-like text, and can leak or repeat what it learned.

    Example and CED codes
  • Phishingtap to flip

    A social engineering message, usually an email or text, that pretends to be trustworthy to get the target to reveal information, click a link or open a file.

    Example and CED codes
  • Shared secrettap to flip

    A word or phrase known only to two people, used to verify each other's identity in high-stakes situations.

    Example and CED codes
  • Reconnaissancetap to flip

    The first phase of a cyberattack, gathering as much information about a target as possible, often from open source intelligence.

    Example and CED codes
  • Multifactor authentication (MFA)tap to flip

    A login that requires extra proof of identity, such as a one-time code, in addition to a password.

    Example and CED codes
  • Malwaretap to flip

    Software designed to harm a device or its user, such as by stealing data, spying or giving an adversary control.

    Example and CED codes

Questions students ask

How do adversaries use AI in cyberattacks?

They clone voices and faces to impersonate people, write convincing phishing in any language, extract sensitive data from large language models, plant false information in training data, gather information about targets, and write or modify malware.

How can a shared secret stop a voice cloning scam?

A secret word or phrase known only to two people lets them verify each other in a high-stakes call. A cloned voice can sound right but cannot know the secret.

How does MFA help against voice cloning?

If a system uses voice authentication and an adversary clones the voice, requiring a second factor can still stop them from getting into the account.

↑ Top

Enlarged view of the selected figure