Why I started reading access logs
Access logs used to look like a boring list of requests. I did not spend much time on them until I had to answer a difficult question: how did the attacker get in? The logs were the only record of what happened, and they contained the answer. I just had to know how to read them.
This post is my personal study note on access log forensics. I am writing it because the process is more than searching for a suspicious IP address. It is about building a timeline, following the request path, and connecting the web logs to the system artifacts. The result is a story that explains the attack.
Start with the incident
The investigation starts with the incident, not with the logs. I need to know what was detected, when it was detected, and what the alert said. The alert could be a failed login, a file change, an outbound connection, or a customer complaint. Each alert gives me a starting point.
From the alert, I build an initial timeline. I record the time of the alert and the time of the first known malicious action. The window between the two is where the attack happened. I work backward from the alert to find the requests that preceded it.
The timeline should use a single time zone. Logs from different systems often use different time zones, and a one-hour error can change the entire reconstruction. I normalize every timestamp to UTC before I compare the events.
Find the first request
The first request is the hardest to find. An attacker usually scans the application, sends a few harmless requests, and then tries a vulnerability. The first request may look normal. I look for the beginning of a pattern rather than a single suspicious request.
The pattern often starts with directory scanning. The attacker requests common paths like /admin, /wp-admin, /backup, and /.git. The requests are spread over a short period and come from the same source. The pattern is easier to see when I group the logs by source IP and time.
The first request may also be a login attempt. A brute force attack starts with many failed login requests. I look for the first failed login and compare it with the successful login that followed.
Reconstruct the request path
The request path is the sequence of requests that led to the compromise. I follow the timeline and record each step: the scan, the probe, the exploit, the upload, and the access to the backdoor.
The scan phase produces a large number of requests. The exploit phase is usually short and targeted. The upload phase leaves a file on the server. The backdoor phase creates a new pattern of requests that the attacker uses to control the server.
I compare the request path with the application's normal behavior. A request to an upload endpoint followed by a request to a file that does not exist in the normal flow is a strong indicator. The combination of events is more reliable than any single request.
Look for tool signatures
Attackers use tools, and tools leave signatures. The user agent is the most obvious signature, but it is also the easiest to fake. I look for other indicators, such as the request order, the HTTP headers, and the timing.
A scanner might send requests in a fixed order with a fixed delay. An attacker using a browser might send requests in a more natural order with variable timing. The difference is not always clear, but it helps me separate automated activity from manual activity.
The payload is another signature. A request that contains an exploit string, a base64 value, or an unusual parameter name is a candidate. I search the logs for known exploit patterns and compare them with the timeline.
Analyze the successful request
The most important request is the one that succeeded. A failed exploit does not compromise the server. A successful exploit changes the state of the application. I focus on the requests that returned a different status code, a different response size, or a different response time.
The status code alone is not enough. A request that returns 200 may be the one that uploaded the backdoor. A request that returns 500 may be the one that caused the database error that led to the compromise. I compare the request with the normal traffic and look for anomalies.
The successful request may also be invisible in the web logs. If the attacker used a direct database connection or a network-level vulnerability, the web logs will not show it. I use the web logs as one piece of the puzzle, not the whole puzzle.
Follow the outbound connection
After the compromise, the attacker usually needs a way to control the server. The control channel creates an outbound connection to an external host. I look for the first outbound connection that does not fit the normal traffic pattern.
The outbound connection can be found in the firewall logs, the DNS logs, the proxy logs, and the process network state. I search for connections to unusual ports, unusual hosts, and unusual protocols. The destination often belongs to the attacker's infrastructure.
The callback host can reveal the attacker's tooling. A connection to a common port with a known protocol may indicate a reverse shell. A connection to a random domain with a custom protocol may indicate a more sophisticated implant.
Correlate with system artifacts
The web logs are only one layer. The full reconstruction requires the filesystem, the process list, the user accounts, and the startup configuration. I correlate the web request with the system changes that happened at the same time.
If the web log shows an upload request, I look for the uploaded file on the filesystem. If the process list shows a new process, I look for the command line that started it. If a new user account appears, I look for the request that created it.
The correlation turns the timeline into a chain. Each web request should map to a system artifact. A request without an artifact is a gap. An artifact without a request is also a gap. The gaps tell me where I need to look next.
Build the final timeline
The final timeline is the deliverable. It should show the time, the source, the action, and the evidence for every step. The timeline starts with the first reconnaissance request and ends with the containment action.
The timeline should be readable by someone who was not part of the investigation. I use a table or a list with clear labels. The source IP, the timestamp, the request, and the response are the minimum fields.
The timeline should also include the confidence for each step. A request that I verified with a system artifact has high confidence. A request that I inferred from the pattern has lower confidence. The reader needs to know which parts of the story are proven.
Improve the logging
The investigation is easier when the logs are good. I use the gaps in the reconstruction to recommend logging improvements. The logs should include the client IP, the timestamp, the method, the path, the status, the user agent, and the response size.
The logs should be stored outside the web server. If the attacker compromises the server, they may delete the local logs. A central log collector with restricted access preserves the evidence.
The logs should be structured and searchable. A structured format makes it easier to query by time, source, and path. The searchability is what turns a pile of lines into an investigation tool.
HTTP methods and status codes
The HTTP method can tell me what the attacker was trying to do. A GET request is usually a read. A POST request can create or modify a resource. A PUT or PATCH request can change an existing resource. A DELETE request can remove it.
I group the requests by method and look for a sequence. The sequence often follows a pattern: a POST to an upload endpoint, a GET to the uploaded file, and then a POST to a command execution endpoint. The pattern is easier to see when the methods are separated.
The status code is another clue. A large number of 404 responses suggests directory scanning. A single 200 response after many 404s may be the successful access to a hidden path. A 500 response after a suspicious request may be the moment the exploit failed.
The status codes are not perfect, but they add structure to the timeline. I record the code for every request that I consider important.
Log sources to combine
The web access log is only one source. I also look at the application log, the database log, the system log, and the network log. Each source has a different view of the same event.
The application log shows the business action that the web request triggered. A request to /api/login may succeed in the web log and fail in the application log because of a missing permission. The difference is valuable.
The database log shows the queries that the application ran. A request that triggers an unexpected query can reveal a SQL injection. The query pattern is also useful for finding data exfiltration.
The network log shows the connections between the hosts. A web request that is followed by an outbound connection to a new host may be the first step of a command-and-control channel. The network log connects the web layer to the system layer.
Verifying the story
The reconstructed story should be verified before it is reported. I test the key steps by reproducing the request in a safe environment. The reproduction shows whether the path is real.
The verification starts with the entry point. I send the same request to a test instance and compare the response. If the response matches the logs, the step is confirmed.
The verification continues with the artifacts. I check the filesystem for the files that the logs say were uploaded. I check the process list for the processes that the logs say were started.
The final verification is the impact. I confirm that the attacker had access to the data or the actions described in the story. The confirmation makes the report defensible.
Handling missing logs
The logs are often missing a piece of the story. The web server may rotate the logs, the application may not log a certain action, or the attacker may have deleted the evidence. I do not stop at the gap. I use the surrounding events to narrow down what the missing log would have contained. The gap itself becomes a finding if it shows that the logging was not sufficient. The recommendation is to store the logs outside the server, to keep them for a longer period, and to add the fields that are missing.
Common mistakes I make
My biggest mistake is starting with a suspicious IP address and searching for that address only. The attacker may use many addresses, or the real source may be a proxy. I start with the timeline instead. My second mistake is ignoring the time zone differences.
The third mistake is treating a single request as proof. The same request can appear in normal traffic. I need the pattern and the correlation. The fourth mistake is stopping at the web logs and not looking at the system artifacts.
My quick checklist
- Start with the alert and build an initial timeline.
- Normalize timestamps to a single time zone.
- Find the first request in the attack pattern.
- Reconstruct the request path from scan to backdoor.
- Look for tool signatures and payloads.
- Analyze the successful request and its response.
- Follow the outbound connection to the attacker infrastructure.
- Correlate the web logs with system artifacts and build the final timeline.
What I would do next time
Next time I want to collect the system artifacts before I read the web logs. The artifacts give me the endpoints of the story, and the web logs fill in the middle. I also want to spend more time on the time zone normalization, because that is where I make the most mistakes. Access log forensics is not about finding one bad request. It is about rebuilding the whole sequence of events from the evidence.