Why I started using AI in security work
The first time I used a language model for security analysis, I asked it to audit a function and waited for a perfect answer. The answer looked good, but when I checked the code, some of the lines it quoted did not exist. I learned an important lesson: the model is a tool, not an oracle. It can help me find candidates, but it cannot replace the verification.
This post is my personal study note on AI-assisted vulnerability hunting. I am writing it because the technology is useful, but only when I understand its limits. The notes cover where AI adds value, how I write prompts, how I validate the output, and where I draw the line.
Where AI actually helps
Language models are good at pattern recognition. They can read a large amount of code and point to places that look unusual. They can explain an unfamiliar API or a framework feature. They can summarize logs and compare configuration files. These tasks are repetitive and context-rich, which makes them a good fit for the model.
The model is also useful for generating test cases. I can give it a function and ask for the edge cases that I should test. The output is a list of candidates, not a list of confirmed bugs. I still need to run the tests and analyze the results.
The biggest value is speed. A model can scan a codebase in minutes and give me a starting point. The starting point saves me hours of reading code that is unlikely to contain a vulnerability.
What AI cannot do
The model cannot understand the full context of a real system. It does not know the business rules, the deployment environment, or the data flow. A code path that looks vulnerable in isolation may be safe because of a control that is not visible in the function.
The model can also be confidently wrong. It may invent a function, a library, or a behavior that does not exist. It may describe a fix that introduces a new vulnerability. The confidence is not evidence.
I never let the model make a final decision about a finding. The decision requires reproduction, impact assessment, and context. Those are human tasks.
How I structure a prompt
The prompt is the most important part of using a model. A vague prompt like "find the vulnerabilities in this code" produces a vague answer. I use a specific prompt with the function, the language, the framework, and the question.
The prompt should ask for a vulnerability class, a trigger condition, and a prerequisite. It should not ask for a complete exploit chain. The output is easier to verify when it is split into small claims.
I include the relevant data flow. If the input comes from a user, I say so. If the output goes to a database, I say so. The context helps the model connect the code to the risk.
The prompt should also ask for the evidence. I want the model to quote the exact lines that support its claim. The quote is the first thing I verify.
Using the model for code review
My code review workflow starts with a static analysis tool. The tool produces a list of findings, and the model helps me triage the list. I give the model a finding, the relevant function, and the surrounding context. The model explains whether the finding is likely to be exploitable and why.
The model can also find issues that the tool missed. I ask it to look for specific patterns, such as missing authorization checks, unsafe deserialization, or path traversal. The model's suggestions become new candidates for the static analysis tool.
The review is iterative. I send the code, get the candidates, verify the candidates, and then send a new prompt with the verified results. The model learns from the feedback in the conversation, but I do not trust the conversation history as a source of truth.
Using the model for log analysis
Log analysis is another good use. The model can summarize a large log file, group the events by pattern, and highlight the anomalies. The summary saves me time when the log is too large to read manually.
The model can also explain the meaning of an event. A strange error message, a redirect, or a status code can be translated into a hypothesis. The hypothesis is a starting point, not a conclusion.
The important part is that the model does not have the same access to the system as I do. It cannot check the process list, the filesystem, or the network. I use the model to generate questions and then answer the questions with real evidence.
Validating the output
Every claim from the model needs validation. The first validation is the quote. I find the exact lines in the code and check whether they exist and whether they support the claim.
The second validation is the data flow. I trace the input from the entry point to the output. The model may describe a path that does not exist or skip a validation step that protects the code.
The third validation is the test. I write a minimal test that reproduces the claim. If the test passes, the finding is a candidate. If the test fails, I either adjust the test or discard the claim.
The final validation is the impact. A finding is only a vulnerability if it can be exploited in a real context. I assess the reachability, the prerequisites, and the effect.
Hallucinations and bias
Hallucinations are the biggest risk. The model can generate plausible code, function names, and library references that do not exist. I check every identifier that the model uses.
The training data also creates a bias. The model knows popular frameworks better than rare internal systems. A suggestion that works for a popular library may not work for a custom implementation.
The bias can also hide vulnerabilities. If a framework is rarely seen in the training data, the model may not recognize its security patterns. I use the model as one source, not the only source.
Writing the report
When I include AI analysis in a report, I mark it as AI-assisted. The report should separate the model's suggestion from the human verification. The reader needs to know that the finding was confirmed.
The report should include the prompt, the model output, and the validation. The transparency makes the finding reproducible and honest.
I do not include sensitive data in the prompt. The code, the logs, and the configuration that I send to the model may be stored by the service. I sanitize the data before I use an external model.
The ethical boundary
AI does not change the ethics of security testing. I still need authorization before I test a system. I still need to report the findings responsibly. I still need to avoid causing harm.
The model can make the work faster, but it does not make the work safer. A faster attack is not a better attack if it is outside the scope. The authorization and the impact assessment remain human decisions.
The model also does not change the legal responsibility. If I use AI to generate a payload and it causes damage, I am still responsible. The tool does not accept the risk.
A prompt that works
The difference between a useful prompt and a useless prompt is the amount of context. I start with the language, the framework, the entry point, and the question. Here is the shape I use:
This is a Python FastAPI endpoint. The user_id parameter comes from the authenticated session. The query parameter comes from the request. Please list the vulnerability classes that apply, the trigger condition for each, and the exact lines that support the claim.The prompt does not ask for a complete exploit. It asks for a list and the evidence. The output is easy to triage and verify.
I also ask the model to separate its answer into facts and assumptions. The facts are the lines that it quotes. The assumptions are the parts that depend on the surrounding code. The separation makes the verification easier.
When the model returns a finding, I send a follow-up prompt with the result of my verification. The follow-up asks the model to explain how the finding could be triggered with the real data flow. The explanation is the final test.
Limitations I keep hitting
The model often assumes that a function is called in a certain way. The assumption may be wrong. I have seen models describe an endpoint that exists only in an older version of the framework, or a parameter that is never passed.
The model also struggles with large codebases. It can analyze a single function well, but it cannot keep the whole application in context. The connections between the functions are where the vulnerabilities hide.
The model's knowledge has a cutoff date. A framework that changed after the cutoff may behave differently. I check the current documentation instead of relying on the model's description.
The model can also be influenced by the prompt. If I ask whether a function is vulnerable, it tends to say yes. If I ask whether it is secure, it tends to say yes. I try to ask neutral questions and compare the answers.
Guardrails in the workflow
The workflow needs guardrails so that the model does not become the decision maker. The first guardrail is a rule that every finding must be reproduced. The reproduction can be a test, a manual request, or a code trace.
The second guardrail is a data policy. I do not send customer data, production credentials, or personal information to an external model. I sanitize the input and use fake data when possible.
The third guardrail is a review step. A human reviews the model output before it enters the report. The review checks the evidence, the impact, and the context.
The fourth guardrail is a logging step. The prompt, the output, and the verification are saved. The log makes the process auditable and helps me improve the prompts.
The tool is not the analyst
The model is a fast assistant, but I am the analyst. It can find a suspicious line faster than I can, but it cannot decide whether the line matters. The decision depends on the business context, the deployment, and the authorization. I keep this boundary clear in every prompt and in every report. When I present a finding, I present the evidence and the reasoning that I verified, not a model's confident summary. The model helps me get to the evidence faster, but the evidence is what the report is built on.
Common mistakes I make
My biggest mistake is treating the model's answer as a final report. The answer is a hypothesis. My second mistake is sending too much code without context. The model works better with a focused question. My third mistake is not checking the quoted lines.
The fourth mistake is using an external model with sensitive data. The data may be retained. The fifth mistake is skipping the manual test because the answer looked confident.
My quick checklist
- Start with a specific prompt and a clear question.
- Ask for the vulnerability class, trigger, and prerequisite.
- Ask the model to quote the relevant lines.
- Verify every quote and every identifier.
- Trace the data flow and run a minimal test.
- Assess the impact in the real context.
- Sanitize the data before sending it to an external model.
- Mark AI-assisted findings and include the validation in the report.
What I would do next time
Next time I want to build a better feedback loop between the model and the static analysis tools. The model is most useful when it is combined with deterministic checks. I also want to spend more time on the validation, because that is where the difference between a candidate and a finding is decided. AI is not a replacement for security testing. It is a way to make the human part of the work more efficient.