How does archived endpoint reconnaissance uncover hidden attack surface?
Max Intel's Wayback & CommonCrawl Recon queries the Internet Archive's Wayback Machine CDX API and the CommonCrawl Index API to retrieve every URL ever crawled for a target domain. Web crawlers capture publicly accessible paths — including configuration files, admin panels, API documentation, and database backups that were briefly exposed before being removed. According to the OWASP Testing Guide v4.2, passive reconnaissance using web archives is a foundational step in application security testing because it reveals the historical attack surface that active scanning misses.
Why do archived URLs matter for security assessments?
A file removed from a live server may still be accessible at a different path, cached by a CDN, or retrievable from the archive itself. The SANS Institute penetration testing methodology recommends web archive enumeration as a standard reconnaissance technique because it reveals: endpoints that existed during development but were removed before production, configuration files that leaked credentials or internal hostnames, API versions that were deprecated but never fully decommissioned, and backup files that contain source code or database schemas. A 2024 HackerOne report found that approximately 18% of valid bug bounty submissions involved endpoints discovered through passive reconnaissance rather than active scanning — many of which were found in web archives.
What is the difference between Wayback Machine and CommonCrawl?
The Wayback Machine (operated by the Internet Archive since 2001) provides the largest single web archive, with over 890 billion archived pages. Its CDX API supports time-range filtering and returns HTTP status codes alongside URLs. CommonCrawl is a separate non-profit that publishes monthly web crawls as open datasets. Querying both sources provides broader coverage — CommonCrawl sometimes captures URLs that the Wayback Machine missed, and vice versa. Max Intel deduplicates results across both sources and marks URLs found in both as higher-confidence findings.
- Wayback Machine CDX API
- The programmatic interface to the Internet Archive's URL index, returning archived URLs with timestamps, HTTP status codes, and MIME types for any domain. Supports wildcard subdomains, date range filtering, and result collapsing by URL key.
- CommonCrawl Index
- A publicly accessible index of URLs captured during CommonCrawl's monthly web crawls. Each crawl index covers approximately 3–4 billion pages and can be queried via a REST API that returns URL, status, MIME type, and crawl metadata.
- Attack Surface Mapping
- The process of identifying all externally accessible endpoints, services, and data stores associated with an organization. Archived endpoint discovery contributes to attack surface mapping by revealing paths that no longer appear in DNS records, sitemaps, or live crawls.
- High-Value Target
- An archived URL classified as containing potentially sensitive information — such as
.envfiles, database credentials, API documentation (Swagger/OpenAPI), version control metadata (.git/config), or server configuration files that may reveal internal architecture.