A Security Engineer’s Guide to AI Hallucination Risks

AI coding assistants are now a part of many development workflows, suggesting dependencies and drafting the security logic around them. The output arrives fast and reads well, which is exactly what makes it worth scrutinizing.

An AI hallucination happens when a model produces content that looks authoritative and turns out to be fabricated. In a build pipeline, it can mean a malicious package installed with full trust or an authentication pattern that leaves a service exposed.

How Do AI Hallucinations Happen?

Language models predict likely tokens based on patterns in their training data. When a model reaches a gap in what it learned, it often fills that gap with something statistically plausible. The result reads with the same confidence as a correct answer, which removes the usual signal that engineers rely on to spot bad information.

Code makes this harder to catch than prose. Fabricated library names follow the same conventions as real ones. Invented function signatures match the style of the surrounding SDK. Your reviewer sees syntactically valid code that compiles and moves on.

The Supply Chain Risk Hiding in Your Autocomplete

Package hallucination has become a well-studied security consequence of this behavior. Researchers analyzing output from 16 code-generating models found that roughly 1 in 5 recommended dependencies pointed to packages that never existed, producing more than 205,000 unique fabricated package names across their test set.

The attack follows naturally. Threat actors monitor which package names models tend to invent, register those names on PyPI or npm, then wait. Developers accept the suggestion, run the install command and execute attacker-controlled code with whatever privileges the build environment grants. Researchers named this pattern slopsquatting, a variation of typosquatting that skips the typo entirely.

What makes it durable is repeatability. Hallucinated names recur across sessions rather than appearing randomly, so an attacker who registers the right name catches traffic for as long as the model keeps suggesting it.

The Significant Risk of Fabricated Security Advice

Ask a model to implement authentication and it may confidently produce a pattern that was standard practice a decade ago, complete with reasoning that sounds current.

For example, SMS-based one-time codes could still appear regularly in generated auth flows. This is at odds with modern guidance, which typically encourages teams to adopt phishing-resistant passkeys for accounts that matter, since they aren’t at risk of being forgotten or reused across sites. Considering that phishing attacks are becoming increasingly sophisticated through AI integration, this is especially concerning.

The same pattern shows up in cryptography, where models cite deprecated cipher suites, and in cloud configuration, where they generate IAM policies carrying permissions nobody asked for. Each case looks like a reasonable answer until someone with domain knowledge reads it closely.

Building Guardrails Into the Pipeline

Treat model output as untrusted input because that is, functionally, what it is. Dependency allowlists and lockfile review catch fabricated packages before they reach a build. Automated checks that verify every suggested import against a known registry catch them earlier still.

The OWASP security community now formally tracks this category, and its guidance on misinformation recommends that teams cross-check outputs against verified sources while maintaining human review for anything that touches critical paths. Retrieval-augmented generation helps by grounding answers in documentation you control rather than training data you cannot inspect.

Additionally, measurement is highly important. Published work on library hallucinations establishes benchmarks that teams can adapt to track how often their own tooling fabricates dependencies, turning a vague worry into something you can monitor over time.

Building Long-Term Resilience With the Right Security Frameworks

Hallucination belongs in your threat model alongside injection and misconfiguration. The mitigations are familiar ones, since verifying dependencies and grounding automated suggestions in trusted sources are practices most teams already understand. What changes is the volume. AI assistants generate suggestions faster than any human reviewer can absorb them, so the checks have to live in automation rather than in attention. Build those checks once, and the productivity gains stay while the attack surface stops growing beneath them.

April Miller is a Senior Writer at ReHack. She has more than 5 years of experience writing on AI-powered cybersecurity from threat prediction and automated defenses to measuring cyber risk for non-technical executives. You can connect with her on LinkedIn.