Why I keep coming back to this topic
I have lost track of how many times I started a web test by opening a scanner and running it against a root domain. The scan would return a few thousand lines, most of them useless, and I would spend the rest of the day trying to decide what mattered. After doing that enough times, I stopped pretending that a big scan was the same thing as good reconnaissance. This post is my attempt to write down the process I actually use now. It is not a list of the most advanced tricks in the world. It is the workflow that I trust when I need to find a real vulnerability before I run out of time.
The main idea is simple: reconnaissance is about building a map of the target, not about collecting the largest possible pile of DNS names. A map needs to tell me what the organization owns, which services are reachable, what technology is running, and which paths are worth attacking. Everything else is noise. When I look at the results of a good recon phase, I should be able to explain why each target is in scope, what it runs, and why it is worth testing. If I cannot explain that, I probably copied the asset from a scan output without thinking about it.
Scope comes before everything
The first thing I write down is the authorized scope. It sounds obvious, but I have seen people skip this step because they want to get straight to the fun part. A scope table does not need to be complicated. I usually create a text file with four columns: asset, ownership, access level, and notes. The asset column contains domains, subdomains, IP ranges, cloud accounts, or third-party services. The ownership column records who is responsible for that asset or why we believe the organization controls it. The access level column records whether we are allowed to test the asset directly or whether it belongs to a vendor. The notes column is for anything unusual, such as a staging environment that should not be touched during business hours.
This table saves me in two ways. First, it prevents me from attacking something that is not in scope. A CDN node, a parking page, or a service hosted by a customer can look like a target but should not be tested without permission. Second, it gives me a way to explain my decisions later. If someone asks why I did not test a particular subdomain, I can point to the table and show that the ownership was unclear. If someone asks why I spent time on a strange API, I can show the note that led me there.
I also record the assumptions I make before the work starts. For example, I might assume that app.example.com is owned by the target because it returns a login page with the company logo. That is a reasonable assumption, but it is still an assumption. I write it down so that I can revisit it if the evidence changes. If the same subdomain later resolves to a different IP range owned by a cloud provider, I know exactly which assumption needs to be checked.
Collecting subdomains without losing my mind
For most engagements, subdomain enumeration gives me the largest expansion of attack surface. I combine passive sources first, because they do not touch the target and they usually provide a solid baseline. Certificate transparency logs are the source I check first. Every time an organization issues a certificate for a hostname, the certificate is logged, and the hostname becomes part of a public record. Tools that query these logs can return names that are not linked from the main website and may not be indexed by search engines.
The second passive source is DNS data from public resolvers and historical databases. These records can reveal names that were used in the past, even if they no longer resolve from a normal recursive DNS query. A decommissioned subdomain can still point to an internal IP range, a forgotten development server, or a service that was handed over to a third party. Historical records do not prove that a host is alive, but they give me a list of candidates that I can check with a direct query.
I also look at search engine results, but I treat them as a hint rather than a source of truth. Search indexes contain old pages, cached content, and snippets that may mention internal hostnames. Some of those hostnames are not meant to be public. The search results are useful for finding things like a staging login page that was accidentally indexed or an internal documentation link that was shared publicly.
Once I have the passive list, I validate every name with a direct DNS query. This step is important because passive data can be stale. A name that appears in a certificate log might have been removed from DNS months ago. If I keep that name in my list, I will waste time trying to resolve it later. I also check for wildcard DNS records. Some organizations configure *.example.com to resolve to the same IP, which makes every random subdomain appear to be alive. If I do not detect the wildcard early, my final list will be full of names that never existed.
Certificate transparency is still my favorite trick
Certificate transparency deserves its own section because it is the most consistent source of new subdomains in my experience. The idea behind it is simple: a certificate authority publishes every certificate it issues, and anyone can query the public logs. When a company buys a wildcard certificate for *.example.com, the certificate log contains the wildcard name but not every individual subdomain. More importantly, when a company buys separate certificates for individual hosts, the logs contain those exact hostnames.
I have found development servers, old admin panels, and internal tools through certificate logs when the target had no public links to them. In one engagement, the main website had only a handful of subdomains, but the certificate logs revealed dozens of hostnames used by different teams. Most of them were dead, but one was a legacy application with default credentials. That single name would have been nearly impossible to find with a wordlist because it was a project-specific internal name.
The practical workflow is simple. I collect the certificate transparency results for the root domain and its known subdomains, then I add the results to my DNS validation list. I do not assume that every name in the log is still active. I also do not ignore names that fail to resolve immediately, because some of them only resolve inside the corporate network or on specific DNS servers. I keep a separate list for unresolved names and revisit it if I find a way into the internal network.
DNS validation and the wildcard trap
After passive collection, I run a batch DNS query against every candidate. The output should include the resolved IP address, the CNAME target, and whether the name returned an NXDOMAIN response. I compare the results with the wildcard record before making any decision. If the root domain resolves to 10.0.0.1 for a random name, then a wildcard is probably configured, and every random subdomain I generate will also resolve to 10.0.0.1.
There are a few ways to handle a wildcard. The simplest is to test several random names that are unlikely to exist. If they all resolve to the same IP, I mark the zone as wildcard-enabled and filter my list accordingly. The filtering is not perfect, because a legitimate subdomain might share the same IP. I use additional evidence, such as certificate names, HTTP response differences, and DNS history, to decide which names are real.
CNAME records are another important part of DNS validation. A subdomain might resolve to a third-party service through a CNAME. For example, blog.example.com might point to a GitHub Pages URL or an S3 bucket. These records matter because they can lead to subdomain takeover, but they also matter for scope. If the CNAME points to a vendor-controlled host, I need to know whether the organization still controls that resource or whether the vendor is responsible for it.
Grouping IPs and thinking about ownership
Once I have a list of live subdomains and their IP addresses, I group them by IP ownership. The goal is to find clusters of services that share the same infrastructure. If ten subdomains all resolve to the same cloud load balancer, I can treat them as one deployment instead of ten independent applications. If two unrelated subdomains resolve to the same server, that server becomes a more interesting target because compromising it might affect multiple applications.
I use IP ownership data to separate cloud providers, hosting companies, and internal ranges. The organization may use a CDN for the public website and a separate cloud account for the API. The CDN IPs are not the origin server, so testing them directly will not give me the same results as testing the origin. I look for origin IPs through DNS history, email headers, mobile app endpoints, and other services that may bypass the CDN.
Port scanning comes after I have a manageable list of hosts. I do not scan every port on every host at the start. I focus on the ports that are most likely to expose useful services, such as web servers, SSH, databases, remote management interfaces, and internal application ports. If a host is only supposed to be a web server, I check whether it also listens on a database port or a debug port. Those unexpected services are often where the interesting findings hide.
Fingerprinting the technology
Technology fingerprinting tells me what I am actually dealing with. The easiest place to start is the HTTP response. I look at the Server header, the X-Powered-By header, the cookies, and the content of the page itself. A specific framework often leaves traces in the way it generates URLs, forms, error messages, and session cookies. For example, a Java application might use a JSESSIONID cookie, while a PHP application might use a PHPSESSID cookie. Those small differences matter because they tell me which exploit techniques are relevant.
I also look at the page structure. Some frameworks add a specific set of static files, such as /assets/, /static/, or /wp-content/. The presence of WordPress, Drupal, or a custom application changes the wordlist I use for content discovery. It also changes the way I think about authentication, plugins, and configuration files.
Version numbers are the part I care about most. A version number is not interesting by itself, but it connects directly to vulnerability databases. If I can identify that an application runs an old version of a framework, I can search for known vulnerabilities and build a much more targeted test. If I cannot determine the version from the page, I look at the source map, the JavaScript bundle, the API response, or the release notes that are sometimes left in the application.
The important rule I follow is to confirm a fingerprint with at least two independent signals. If the cookie says PHP but the page structure looks like a static site, I do not assume PHP. I check the headers again, look at the error page, and examine the server behavior. A single signal can be misleading. Two signals that agree are much more reliable.
Content and directory discovery
Content discovery is where I find the endpoints that the application was not trying to show me. The most efficient way to start is not a dictionary brute force. I first check the files that the application itself exposes: robots.txt, sitemap.xml, the main JavaScript files, and any API documentation. These files can contain paths that are not linked anywhere in the UI.
JavaScript files are especially useful. Modern applications bundle a lot of their routing, API calls, and configuration into JavaScript. I search the bundles for URL patterns, API paths, and strings that look like internal endpoints. A single bundle can reveal the entire client-side route map, including admin pages that are hidden from the navigation menu.
When I do use a dictionary, I choose it based on the technology I identified. A WordPress site needs a WordPress-specific list that includes files like wp-config.php.bak and plugin paths. A custom Java application needs a list that includes common Spring and Tomcat paths. The generic lists still have a place, but they are not my first choice.
Every path I find gets scored. The score is based on how accessible the path is, whether it requires authentication, and how sensitive the data is. A public Swagger page is more interesting than a random static file. An admin login that returns a different error for valid usernames is more interesting than a generic 404 page. I keep the scored list in my notes so I can prioritize the active testing phase.
Making sense of the results
The final step is not a final step at all. I review the whole list, remove duplicates, and re-check the ownership of anything that looks unusual. I then assign a priority to each target. The priority should reflect the value of the asset, the ease of reaching it, and the amount of functionality it exposes. A staging server with an admin panel and weak authentication usually ranks higher than a static marketing page.
I also write down the things I could not confirm. If a subdomain resolves to an IP that I cannot identify, I note that it is unresolved. If a technology fingerprint is uncertain, I note that too. These open questions are not failures. They are reminders that the reconnaissance phase is complete only when I can explain what I know and what I still do not know.
Things I keep getting wrong
I still make a few mistakes on almost every engagement. The first is spending too much time on a single source. Certificate transparency is useful, but if I run twenty variations of the same query, I am not learning anything new. The second mistake is treating every response as a target. A shared hosting page, a parked domain, or a CDN error page is not a vulnerability. The third mistake is forgetting to record the exact commands I ran. Without that record, I cannot reproduce my own findings a week later.
The fourth mistake is skipping the wildcard check. I have wasted hours resolving names that were generated by a wildcard record. The fifth mistake is assuming that a subdomain is owned by the organization just because it shares the root domain. Subdomains can point to third parties, and that distinction changes the scope. I now write these reminders at the top of every recon note.
My quick checklist
- Write the authorized scope and assumptions before running anything.
- Collect passive subdomains from certificate transparency and DNS history.
- Validate every name with a direct DNS query and check for wildcards.
- Group IPs by ownership and identify CDN versus origin infrastructure.
- Fingerprint technology with multiple independent signals.
- Read robots, sitemap, JavaScript, and documentation before brute force.
- Score each path by accessibility, authentication, and data sensitivity.
- Record every command and mark unresolved questions.
What I would do next time
Next time I want to spend less time on raw collection and more time on validation. The collection part is easy to automate, but the validation is where I actually learn something. I also want to keep a tighter list of targets, because a small list that I understand is worth more than a large list that I do not. Reconnaissance is not finished when the scan finishes. It is finished when I can explain the map well enough to start testing.