The security risks of LLM-generated code

Between 2022 and 2026, the models got extraordinarily good at writing code. They went from completing a function to completing a feature, from a plausible sketch to something that compiles, passes its tests, and handles the edge case you forgot to mention.

They did not get correspondingly better at writing secure code. That gap — capability racing ahead, security roughly flat — is the whole story, and it is worth being precise about why, because the usual framing ("AI writes insecure code, be careful") is both too vague to act on and not quite right.

What the evidence actually says

The first serious measurement was Pearce et al.'s Asleep at the Keyboard, presented at IEEE S&P in 2022. The researchers built 89 scenarios targeting MITRE's Top 25 weaknesses, generated 1,689 programs with GitHub Copilot, and found roughly 40% of them vulnerable — about 50% for C, about 39% for Python.

That number is quoted constantly, and it is usually quoted wrong. The scenarios were constructed to invite specific weaknesses; the study measures how often the model takes the bait when a task is shaped like a trap, not the vulnerability rate of a normal working day. It was a controlled experiment, and it deserves to be read as one.

The more uncomfortable result came a year later. Perry, Srivastava, Kumar and Boneh at Stanford put 47 participants through five security-relevant programming tasks across Python, JavaScript and C, half with an AI assistant and half without. The group with the assistant wrote significantly less secure code — and was significantly more likely to believe their code was secure.

That second half is the finding that matters. A tool that makes you worse at something while making you feel better about it has removed the signal you would otherwise have used to catch yourself. Notably, the participants who did best were the ones who trusted the assistant least and iterated on their prompts rather than accepting the first completion.

Why the gap exists

Two reasons, and neither is going away on its own.

The training data is the accumulated public record of how software is actually written. That corpus contains Stack Overflow answers optimised for making the error message go away, tutorial code with authentication stripped out for clarity, and a very large volume of repositories no one ever reviewed. It also contains the vulnerable half of every "here is the bug / here is the fix" pair, with nothing in the tokens themselves marking which half is which. The model learns the median, and the median is not secure.

Nothing in the loop punishes an insecure completion. Consider how every other class of defect gets caught. A syntax error fails to parse. A logic error produces the wrong output. A performance problem shows up as a slow page. Each has a fast, automatic feedback signal that reaches the developer within seconds.

A security defect has none. The code compiles, the tests pass, the feature works, the reviewer sees a diff that does the thing the ticket asked for. Nothing anywhere in that loop is measuring the property that failed. So of every defect class, security defects are the ones that survive AI-assisted development at the highest rate — not because models are especially bad at security, but because it is the only category with no error signal attached.

Which vulnerabilities assistants actually write

Not a random spread. The failures cluster, and once you know the shapes you start seeing them in review.

Injection at the edges of parameterisation

Models have thoroughly learned parameterised queries. Ask for a simple lookup and you will almost always get the correct thing:

cur.execute("SELECT * FROM orders WHERE user_id = %s", (user_id,))

The failure is at the boundary of what parameterisation can express. A placeholder can stand in for a value; it cannot stand in for an identifier — a column name, a table, a sort direction. So the moment the query shape becomes dynamic, the model falls back to the only thing that works:

sort = request.args.get("sort", "created_at")
direction = request.args.get("dir", "ASC")
cur.execute(
    f"SELECT * FROM orders WHERE user_id = %s ORDER BY {sort} {direction}",
    (user_id,),
)

This is the dangerous case, and not only because it is injectable. It is dangerous because it looks reviewed. There is a %s on the line. A reviewer skimming a forty-file diff sees the placeholder, registers "parameterised", and moves on — which is exactly the outcome the shape of the code invites.

The same pattern recurs everywhere parameterisation runs out: IN clauses built by joining a list, LIKE patterns assembled with f-strings, ORM .filter() calls that drop into raw SQL for one condition.

Authorisation on the route, not on the object

This is the most common serious flaw in AI-assisted code, and the reason is worth understanding.

@app.get("/api/invoices/{invoice_id}")
@login_required
def get_invoice(invoice_id: int):
    return db.query(Invoice).filter(Invoice.id == invoice_id).one()

The @login_required is present. Every other route in the file has one, it is visible in the model's context, and imitating the surrounding code is the single thing these models do most reliably.

What is missing is the check that this particular user may read this particular invoice — and that check is not a pattern visible in the file. It is a domain fact: that invoices belong to organisations, that users belong to one organisation, that the relationship is enforced at read time. The model cannot infer it from the tokens in front of it, so it does not write it, and the result authenticates carefully and authorises not at all.

Any authenticated user can read every invoice by incrementing an integer.

Path handling that looks defensive

path = os.path.join(UPLOAD_DIR, filename)
with open(path, "rb") as fh:
    return fh.read()

os.path.join is not a sanitiser, though it reads like one. It does not normalise .., so ../../etc/passwd walks straight out of the upload directory. And it has a second behaviour that surprises people who have used it for years: if the later argument is absolute, everything before it is discarded.

>>> os.path.join("/var/uploads", "/etc/passwd")
'/etc/passwd'

The base directory is not a prefix. It is a suggestion, silently abandoned the moment the attacker supplies a leading slash.

The steady background set

These recur often enough to be worth a checklist:

• Ageing cryptography. MD5 or SHA-1 for password storage, AES in ECB mode, a hardcoded or reused IV, random where secrets is required. Training-data age showing through: this was ordinary advice for a long stretch of the corpus.
• Verification disabled. verify=False on requests, rejectUnauthorized: false in Node, custom trust managers that accept everything. These usually enter as a fix for a certificate error during development and are never removed.
• Unsafe deserialisation. pickle.loads on anything that crossed a network, yaml.load without SafeLoader, Java readObject on untrusted bytes.
• CORS with a wildcard and credentials. Access-Control-Allow-Origin: * alongside Allow-Credentials: true — a combination browsers reject, which is usually then "fixed" by reflecting the Origin header, which is worse.
• Shell interpolation. subprocess.run(..., shell=True) with an f-string in it.
• Placeholder secrets that ship. SECRET_KEY = "changeme", DEBUG = True, default credentials in a config file that was only ever meant for local use.

The dependency problem nobody predicted

Then there is a failure mode with no pre-AI equivalent at all.

Models invent packages. Asked to parse a date or sign a token, an assistant will confidently import a library with an entirely plausible name that does not exist on any registry. Spracklen et al., at USENIX Security 2025, generated 576,000 code samples across 16 models and found 19.7% of suggested packages were hallucinated — 205,474 unique names for libraries that were never published. Around 21.7% for open-weight models, about 5.2% for the strongest commercial ones. Roughly 38% of the invented names closely resembled real packages.

The attack writes itself, and it has a name: slopsquatting. Hallucinations are not random noise — the same prompt tends to produce the same invented name across runs and across users. An attacker collects those names, registers them on PyPI or npm, and publishes something malicious. Then they wait, because they do not need to compromise anything. They need a developer to paste an import that an assistant produced, hit install, and get a package that now exists.

Ordinary typosquatting requires you to make a typo. This requires you to trust a completion.

So — is vibe coding safe?

Worth defining before answering, because the term has drifted. Andrej Karpathy coined it for a specific practice: describing what you want, accepting what the model produces, and not reading the code. Not "using AI to write code" — everyone does that now. Specifically: shipping code no human has read.

Asked that way, the answer is not about AI at all.

Untrusted input is what makes code security-relevant. A script that reformats your own CSVs, a prototype you will demo once and delete, an internal tool that only ever sees data you produced yourself — none of these have an attacker in the picture, and reading every line of them buys you nothing. Vibe code them. That is a completely reasonable use of the technology and the productivity gain is real.

The moment anything crosses a boundary you do not control — a form submission, an uploaded file, a webhook, a query parameter, a message off a queue, a header — the code is now a security boundary, and shipping a security boundary that no human has read is the definition of the problem, whoever or whatever wrote it.

The dividing line is not AI or not. It is is there an untrusted input boundary, and did anything competent look at it.

And here is the part that makes this harder than it sounds: the ratio of code written to code understood has moved sharply, in one direction, in about three years. Teams are producing several times the diff. Review capacity did not multiply to match — the same two senior engineers are reading it, in the same working day. A reviewer facing three times the volume is not reviewing three times as fast. They are reviewing less carefully, and the defect class they will drop first is the one with no error signal attached.

That is the actual risk. Not that a model writes a bad line — humans have always written bad lines. It is that the bad line now arrives inside a volume of plausible, well-structured, test-passing code that nobody has the hours to read properly.

What actually helps

Ordered by how much difference it makes.

Constrain at the architecture, not at the prompt. Prompt instructions are advisory; the model imitates the code around it far more reliably than it follows a system message. If every database call in your codebase goes through a query builder, the model writes query-builder calls. If raw SQL exists anywhere in the repository, it will eventually write raw SQL. Make the secure path the only path that is visible, and the imitation works for you instead of against you.

Keep the review, move what it covers. Reading every line is no longer realistic and pretending otherwise just means the review becomes ceremonial. Read the boundaries instead — anything touching authentication, authorisation, input parsing, deserialisation, subprocess, file paths, or crypto. Everything else can have a lighter touch. A short, honest review of the 5% that matters beats a nominal review of 100%.

Automate the part that scaled. Volume is the thing that changed, so the answer has to be something whose cost does not rise with volume. Machine analysis on every diff is the only lever with that property.

Pin and verify dependencies. Lockfiles, hash pinning, and a check that a package existed before your assistant mentioned it.

Ask for the attack, not the review. "Is this secure?" reliably gets you "yes" — models are agreeable, and the Stanford result suggests that agreeableness transfers straight into the developer's own confidence. "Write me a request that exploits this endpoint" produces something you can actually run, and either it works or it does not.